System

Abstract

A system is provided.SOLUTION: A system comprising: means for collecting speech data of a conversation; means for transmitting the collected speech data in real time; means for analyzing the transmitted speech data; means for extracting an intention of an utterance from the analyzed speech data; means for summarizing information obtained from the means for extracting an intention of an utterance; means for generating information effective when the next utterance is made based on the summarized information; and means for presenting the generated information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Reducing unnecessary stress and tension in interpersonal relationships and ensuring smooth communication are common challenges for many people. In particular, misunderstandings and miscommunication, lack of social skills, and emotional issues have a significant impact on daily life and the work environment. Effectively resolving these issues using traditional methods has proven difficult, creating a demand for real-time conversation support. [Means for solving the problem]

[0005] This invention provides a means for collecting speech data from a conversation, preprocessing it, and transmitting it to a server in real time. The server analyzes the transmitted speech data, extracts the intent of the speech from the results, and summarizes it in an easy-to-understand format. Furthermore, based on this summarized information, it generates effective next utterances. The generated information is customized according to the user's past speech patterns and set preferences, and is transmitted to and presented on the terminal. This invention makes it possible to understand the other person's intent in real time and respond appropriately and effectively, thereby achieving smooth communication.

[0006] "Conversational audio data" refers to data in which the contents of a conversation between a user and another person are recorded in audio format.

[0007] "Means for collection" refers to devices or software used to collect voice data of users speaking.

[0008] "Means of transmission" refers to the functions and technologies for transmitting collected voice data to a server in real time.

[0009] "Means for analyzing" refers to a system that converts transmitted voice data into text data using voice recognition technology or natural language processing technology and understands its content.

[0010] "Means of understanding the intention of a statement" refers to the function of extracting and understanding the speaker's intentions and emotions from analyzed text data.

[0011] "Means of summarizing" refers to the function of summarizing and concisely summarizing the intent and information that has been extracted.

[0012] "Means of generation" refers to techniques for generating effective answers or responses to be spoken next based on the summarized information.

[0013] "Presentation means" refers to a device or interface for providing generated information and candidate responses to a user.

[0014] "Noise filtering" refers to the process of removing unnecessary noise and background sounds from collected audio data to obtain clear audio.

[0015] "Volume normalization means" refers to technology that adjusts the volume of audio data to an appropriate level to make it easier to analyze.

[0016] "Real-time" means that processing occurs without delay at the exact moment the conversation takes place.

[0017] "Past speech patterns" refers to data that indicates the tendencies and characteristics of speech made by a user up to now.

[0018] "Means for customization" refers to a function for individually adjusting the responses generated and the information presented according to the user's preferences and past speech patterns. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] This invention relates to an "AI Conversation Advisor" that utilizes generative AI to provide conversation support and advice on what to say. This system collects and analyzes voice data in real time and provides appropriate responses to users, thereby supporting smooth communication.

[0041] System Overview

[0042] 1. User device: The user uses a smart glasses-type device or a smartphone. These devices are equipped with a microphone to collect conversations between the user and others in real time, and a display or audio output to show the results.

[0043] 2. Server: The collected voice data is sent to the server via the network, where it is analyzed using high-performance voice recognition and natural language processing technologies.

[0044] Program processing

[0045] Collection and transmission of voice data

[0046] First, when a user starts a conversation, the smart glasses or smartphone collects audio data. For example, when collecting audio during a meeting, collection begins automatically as soon as the meeting begins. The collected audio data undergoes pre-processing such as noise filtering to make it clear audio data. This audio data is then sent to a server in real time.

[0047] Analyzing voice data and understanding intent

[0048] The server uses speech recognition technology to convert the received voice data into text. The text data is then subjected to syntactic analysis, which identifies the subject, predicate, and object of the sentence and associates them with the context. Contextual and sentiment analysis are then performed to understand the intention and emotion behind the other person's remarks. For example, if the question includes "How is the project progressing?", it is interpreted as "You are asking about the progress of the project."

[0049] Summarization and response generation

[0050] The intent that is captured is summarized on the server and put into a form that is easy for the user to understand. For example, it may be summarized as "Your boss is asking about the progress of the project." The generative AI also generates multiple effective responses to utter next. Response candidates are customized based on the user's past speech patterns and set preferences.

[0051] Presentation of results

[0052] The summarized information and possible responses are sent to the device and displayed on the smartglasses display or smartphone screen. Voice guidance is also provided as needed. For example, the message might read, "Your boss is asking about the progress of a project. What do you think of the following response?"

[0053] Specific examples

[0054] Business Meeting Scenario

[0055] 1. User: Puts on smart glasses as soon as a business meeting begins.

[0056] 2. Terminal: Collects voice data, filters it, and then sends it to the server.

[0057] 3. Server: Converts the speech data into text and performs syntax and sentiment analysis. Example: "Your boss is asking about the progress of the project."

[0058] 4. Server: Generates candidate responses and presents options such as:

[0059] "The project is progressing as scheduled and is currently 70% complete."

[0060] "There are delays to some tasks, but we expect them to be completed by next week."

[0061] "Would you like a more detailed progress report?"

[0062] 5. Device: The generated summary and response suggestions are displayed on the smart glasses display. For example, it shows, "Your boss is asking about the progress of the project. What do you think of the following response?"

[0063] 6. User: Select the appropriate response from the suggested options and execute it in the conversation.

[0064] In this way, the "AI Conversation Advisor" will be able to assist users in their conversations in real time, allowing for smoother communication.

[0065] The processing flow will be explained below.

[0066] Step 1:

[0067] User: Turn on the smart glasses or smartphone and start collecting voice data of the conversation. For example, the moment the user starts speaking in a business meeting, voice data collection starts automatically.

[0068] Step 2:

[0069] Terminal: Noise filtering is performed on the collected voice data to remove background noise, etc. This allows for clear voice data to be obtained.

[0070] Step 3:

[0071] Terminal: Sends noise-filtered audio data to the server in real time. The data is transferred using a secure communication protocol.

[0072] Step 4:

[0073] Server: The received voice data is converted into text data using voice recognition technology. A highly accurate voice recognition engine is used in this process.

[0074] Step 5:

[0075] Server: Performs syntactic analysis on the text data to identify the components of a sentence, such as the subject, predicate, and object.

[0076] Step 6:

[0077] Server: Further contextual and sentiment analysis is performed to understand the intent and sentiment of the statement. For example, if the question "How is the project progressing?" is included, it will be interpreted as "Ask about the progress of the project."

[0078] Step 7:

[0079] Server: Summarizes the intent and generates information to present to the user in an easy-to-understand format. For example, it might summarize the intent as "The boss is asking about the progress of the project."

[0080] Step 8:

[0081] Server: Based on the generated summary, the server generates multiple effective responses for the next utterance. The responses are customized based on the user's past utterance patterns and preferences.

[0082] Step 9:

[0083] Server: Formats the generated summary and response candidates and sends them to the user's terminal.

[0084] Step 10:

[0085] Device: The summary and response candidates are displayed on the smartglasses display or smartphone screen. Voice guidance is also possible. For example, the message may say, "Your boss is asking about the progress of the project. What do you think of the following response?"

[0086] Step 11:

[0087] User: Choose the appropriate response from the suggested responses and use it in the actual conversation.

[0088] Through this series of steps, the "AI Conversation Advisor" supports users' conversations in real time, avoiding misunderstandings and miscommunication and ensuring smooth communication.

[0089] Example 1

[0090] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0091] During a conversation, it can be difficult for users to instantly grasp and respond appropriately. This is especially true in business meetings and important discussions, where it is necessary to respond appropriately on the spot. However, conventional technology lacks the means to analyze the content of a conversation in real time and provide appropriate responses. To solve this problem, a system that allows users to respond smoothly is needed.

[0092] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0093] In this invention, the server includes means for using a device to collect voice data, means for noise filtering the collected voice data in real time and transmitting the data to the server, means for converting the transmitted voice data into text using voice recognition technology, means for syntactically analyzing and sentiment analyzing the converted text data, means for extracting the intention of the utterance from the analyzed data, means for summarizing the intention of the utterance, means for generating information on effective next utterances using a generative AI model, and means for presenting the generated information on a user device, thereby enabling the user to grasp appropriate responses in real time and to communicate smoothly and effectively.

[0094] "Audio data" refers to audio signals of conversations or utterances made by a user.

[0095] "Device" refers to a hardware device for collecting audio data, including, for example, smart glasses and smartphones.

[0096] A "server" refers to a computer system that is connected via a network and processes and analyzes data.

[0097] "Noise filtering" refers to the process of removing unwanted background sounds and noise from collected audio data.

[0098] "Speech recognition technology" refers to technology for analyzing collected voice data and converting voice into text.

[0099] "Syntax analysis" refers to the process of analyzing the grammatical structure of text data and clarifying the relationships between the subject, predicate, object, and other parts of a sentence.

[0100] "Sentiment analysis" refers to the process of inferring the emotional state of a speaker from text data.

[0101] "Intention of speech" refers to understanding what the speaker intended based on the analyzed data.

[0102] "Summarization" means summarizing analyzed information into a concise and easy-to-understand format.

[0103] "Generative AI model" refers to an artificial intelligence model that performs natural language processing and is used to generate new text and responses.

[0104] The term "user device" refers to a device used by a user for displaying information and executing voice guidance.

[0105] This invention relates to an "AI Conversation Advisor" that utilizes generative AI to provide conversation support and advice on what to say. This system collects and analyzes voice data in real time and presents appropriate responses to users to support smooth communication.

[0106] System Overview

[0107] 1. User Devices

[0108] Users use smart glasses or smartphones, which are equipped with a microphone to collect conversations between the user and others in real time and a display or audio output to display the analysis results.

[0109] 2. Server

[0110] The collected voice data is sent to a server via a network. The server then uses high-performance voice recognition and natural language processing technologies to analyze the data. Specifically, it converts the voice data into text using voice recognition technology (e.g., Google Speech-to-Text API) and generates an optimal response using a generative AI model (e.g., OpenAI's GPT-4).

[0111] Program processing

[0112] First, when a user starts a conversation, the smart glasses or smartphone collects audio data. For example, when collecting audio during a meeting, collection begins automatically as soon as the meeting begins. The collected audio data undergoes pre-processing such as noise filtering on the device to create clear audio data. This audio data is then sent to the server in real time.

[0113] The server converts the received voice data into text using speech recognition technology. It then uses a generative AI model to perform syntax analysis and sentiment analysis on the text data to understand the intent and emotion of the other person's speech. For example, if the question "How is the project progressing?" is included, it can be interpreted as "A question about the progress of the project."

[0114] The intent that is captured is summarized on the server and put into a form that is easy for the user to understand. For example, it might summarize it as "your boss is asking about the progress of the project." It also uses a generative AI model to generate multiple effective next utterances. Response candidates are customized based on the user's past utterance patterns and set preferences.

[0115] The summarized information and possible responses are sent to the device and displayed on the smartglasses display or smartphone screen. Voice guidance is also provided as needed. For example, the message might read, "Your boss is asking about the progress of a project. What do you think of the following response?"

[0116] Specific examples

[0117] Business Meeting Scenario

[0118] 1. User: Puts on smart glasses as soon as a business meeting begins.

[0119] 2. Terminal: Collects voice data, filters it, and then sends it to the server.

[0120] 3. Server: Converts the speech data into text and performs syntax and sentiment analysis. Example: "Your boss is asking about the progress of the project."

[0121] 4. Server: Generates candidate responses and presents options such as:

[0122] "The project is progressing as scheduled and is currently 70% complete."

[0123] "There are delays to some tasks, but we expect them to be completed by next week."

[0124] "Would you like a more detailed progress report?"

[0125] 5. Device: The generated summary and response suggestions are displayed on the smart glasses display. For example, it shows, "Your boss is asking about the progress of the project. What do you think of the following response?"

[0126] 6. User: Select the appropriate response from the suggested options and execute it in the conversation.

[0127] Prompt Sentence Examples

[0128] Below are some example prompts to input to a generative AI model (e.g., GPT-4):

[0129] "If your boss is asking about the progress of a project, generate an appropriate response."

[0130] This will cause the AI ​​to generate a response like this:

[0131] 1. "The project is on schedule and is currently 70% complete."

[0132] 2. "We're running behind on some tasks, but we expect to complete them next week."

[0133] 3. "Would you like more detailed progress reports?"

[0134] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0135] Step 1: Collect and preprocess audio data

[0136] The user puts on the smart glasses or smartphone and launches the application.

[0137] Input: Real-time conversational audio produced by the user.

[0138] The device uses the device's microphone to capture audio data in real time. For example, if you're in a meeting, audio collection begins as soon as it starts.

[0139] The device performs noise filtering on the acquired voice data to remove background noise (for example, noise reduction processing using FFT).

[0140] Output: Clear audio data after filtering.

[0141] Step 2: Sending audio data

[0142] The device transmits the pre-processed voice data to the server in real time.

[0143] Input: Clear audio data after filtering.

[0144] Terminals may compress data to improve communication efficiency.

[0145] Output: The compressed audio data is sent over the network to the server.

[0146] Step 3: Convert audio data to text

[0147] The server converts the received voice data into text data using voice recognition technology (e.g., Google Speech-to-Text API).

[0148] Input: Compressed and transmitted audio data.

[0149] The server extracts linguistic features from the audio and converts it into text format.

[0150] Output: The converted text data.

[0151] Step 4: Parsing and sentiment analysis of text data

[0152] The server performs syntactic analysis on the generated text data, thereby identifying the subject, predicate, and object of the sentence.

[0153] Input: The converted text data.

[0154] The server then performs a sentiment analysis to assess the emotional state of the speaker.

[0155] Output: Parsed data and sentiment analysis results.

[0156] Step 5: Understand and summarize the intention of the statement

[0157] The server uses the analyzed data to understand the intent behind the statement.

[0158] Input: Syntactic and sentiment analysis results.

[0159] The server summarizes the intent of the conversation and condenses it into a form that is easy for the user to understand, such as "your boss is asking about the progress of a project."

[0160] Output: Summarized intent data.

[0161] Step 6: Generate candidate responses

[0162] The server uses a generative AI model (e.g., OpenAI's GPT-4) to generate multiple effective next utterances.

[0163] Input: Summarized intent data.

[0164] The server feeds example prompts to the model, which generates appropriate response patterns, taking into account the user's past speech patterns and preferences.

[0165] Output: The generated list of possible responses.

[0166] Step 7: Presenting results and user response

[0167] The server transmits the summarized information and candidate responses to the user terminal.

[0168] Input: The generated response candidate list.

[0169] The device will then display this information on the smartglasses display or smartphone screen, for example, in the form of, "Your boss is asking you about the progress of a project. What do you think of the following response?"

[0170] The user selects the appropriate response from the presented options and executes it in the actual conversation. Selection can be done by voice input or touch operation.

[0171] Output: The response selected by the user.

[0172] (Application example 1)

[0173] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0174] There is a need to improve the quality of customer interactions in brick-and-mortar stores and simultaneously increase sales staff efficiency and customer satisfaction. In particular, there is a lack of support tools to enable appropriate and prompt responses during customer service, and technology to ensure smooth communication with customers is needed. There is also a need for a system that can provide consistent, high-quality customer service that is not dependent on the experience or skills of staff.

[0175] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0176] In this invention, the server includes a preprocessing means for performing noise filtering on voice data collected using a smart device, a means for converting the voice data sent to the server into text and performing syntax analysis and sentiment analysis, and a means for displaying the generated response candidates and summaries on the display of the smart device. This makes it possible to support conversations with customers in real time using the smart device, allowing sales staff to respond to customers more quickly and accurately.

[0177] "Conversational voice data" refers to data in which the voice of a user or customer is recorded in digital format.

[0178] "Means for transmitting in real time" refers to technology for transmitting collected voice data to a server immediately without delay.

[0179] "Means for analyzing" refers to the technology or system that converts transmitted voice data into text data and understands grammar and context.

[0180] "Means for understanding the intention of a statement" refers to technology for extracting the speaker's intentions and emotions from analyzed text data.

[0181] "Means of summarizing" refers to the technique of summarizing the intent and information that has been extracted into a concise, to-the-point format.

[0182] The "means for generating effective information" is a technology for automatically generating information that is determined to be effective when spoken next.

[0183] "Means for presenting" refers to a technique for visually or audibly notifying the user of the generated information.

[0184] "Smart device" refers to an electronic device that can be carried or worn by a user and has the function of collecting and displaying voice data.

[0185] "Noise filtering" is a technique for removing background noise from audio data to obtain clear audio data.

[0186] "Syntax analysis" is a technique for analyzing the grammatical structure of text data and identifying elements such as subjects, predicates, and objects.

[0187] "Sentiment analysis" is a technology that identifies emotions from a speaker's text data and classifies the type of emotion.

[0188] "Response candidates" refer to multiple appropriate responses to be made next.

[0189] A "summary" is a concise summary of the main points and important information of a conversation.

[0190] "Means for displaying on a display" refers to technology for displaying the generated information on the screen of a smart device.

[0191] System configuration

[0192] A system for implementing the present invention includes the following hardware and software.

[0193] Smart device: A device carried or worn by the user that has the ability to collect voice data and display the results. Examples include smart glasses and smartphones.

[0194] Server: Has high-performance computing resources to perform voice data analysis, syntax analysis, and sentiment analysis. It mainly uses speech recognition APIs (e.g., Google Speech-to-Text API) and natural language processing APIs (e.g., OpenAI GPT-3).

[0195] Program Description

[0196] Collection and transmission of voice data

[0197] Smart devices (smart glasses or smartphones) collect the user's voice data in real time using a microphone built into the device. The collected voice data is filtered to make it clear and then sent to a server.

[0198] Analyzing voice data and understanding intent

[0199] The server converts the transmitted voice data into text using a speech recognition API (Google Speech-to-Text API). A natural language processing API (OpenAI GPT-3) is used to perform syntactic analysis and sentiment analysis on the converted text data. Syntactic analysis identifies the subject, predicate, and object, while sentiment analysis extracts the speaker's intention and emotion from the context.

[0200] Summarization and response generation

[0201] Based on the intent, the server summarizes the key points. The AI ​​then generates multiple effective responses for the next utterance. These responses are customized based on the user's past speech patterns and preferences.

[0202] Presentation of results

[0203] The summary and candidate responses generated by the server are displayed on the smart device's display. Voice guidance is also provided as needed. For example, a message such as, "A customer has a question about a new smartphone. What do you think of the following response?" can be displayed.

[0204] Specific examples

[0205] Scene: Customer service in a brick-and-mortar store

[0206] 1. A user puts on the smart glasses and starts a conversation with a customer.

[0207] 2. The smart device collects the voice data and sends it to the server along with noise filtering.

[0208] 3. The server converts the voice data into text and performs syntax and sentiment analysis. Example: "A customer is asking about the features of their smartphone."

[0209] 4. The server generates a candidate response, for example:

[0210] "This smartphone has an excellent camera."

[0211] "Long battery life and convenient"

[0212] Want to know more about the features?

[0213] 5. The generated summary and response suggestions are displayed on the smart glasses.

[0214] Prompt Sentence Examples

[0215] A customer has asked a question about a new smartphone. Generate possible responses.

[0216] In this way, the system can assist sales staff in serving customers in real time and improve customer satisfaction.

[0217] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0218] Step 1:

[0219] Smart devices (smart glasses or smartphones) collect voice data of conversations between users and customers in real time. The input is the voice data of the conversation, and the output is the collected voice data. The voice data is acquired using a microphone, and clear voice data is prepared by noise filtering. This voice data is then passed to the next step.

[0220] Step 2:

[0221] The smart device sends noise-filtered audio data to the server. The input is clear audio data, and the output is the audio data sent to the server. Here, the data is sent in real time using the device's communication module.

[0222] Step 3:

[0223] The server converts the transmitted voice data into text data. The input is voice data and the output is text data. The server uses a speech recognition API (Google Speech-to-Text API) to analyze the voice waveform data and generate a corresponding string.

[0224] Step 4:

[0225] The server analyzes the converted text data and identifies grammatical structures. The input is the text data, and the output is the results of the analysis. This process uses a natural language processing API (OpenAI GPT-3) to identify sentence elements such as subject, predicate, and object.

[0226] Step 5:

[0227] The server performs sentiment analysis based on the parsing results. The input is the parsing results, and the output is the intention of the utterance and the sentiment analysis results. The sentiment analysis module is used to determine the speaker's emotional state and intention. This information is used for further processing.

[0228] Step 6:

[0229] The server summarizes the main points of the speech based on the results of sentiment analysis. The input is the intention of the speech and the results of sentiment analysis, and the output is a summary. A summarization algorithm is used to extract the main points of the text and create a concise summary.

[0230] Step 7:

[0231] The server generates effective response candidates based on the summary. The input is the summary, and the output is multiple response candidates. Using a generative AI model (OpenAI GPT-3), multiple next-best utterances are generated.

[0232] Step 8:

[0233] The smart device receives the summary and candidate responses sent from the server and displays them on the display. The input is the summary and candidate responses, and the output is information provided to the user's visual sense. The display function of the device is used to visually present appropriate candidate responses to the user.

[0234] Step 9:

[0235] The user checks the response candidates presented by the smart device, selects the most appropriate one, and speaks to the customer. The input is the displayed response candidate, and the output is the response selected by the user. This enables smooth and effective customer service.

[0236] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0237] This invention relates to an "AI Conversation Advisor" that combines generative AI and an emotion engine to provide conversation support and advice on what to say. This system collects and analyzes voice data in real time, recognizes emotions, and then presents appropriate responses to the user, supporting smooth communication.

[0238] System Overview

[0239] 1. User device: The user uses a smart glass or smartphone, which is equipped with a microphone to collect real-time conversations between the user and others, a camera to capture the user's speech and facial expressions, and a display or audio output to show the results.

[0240] 2. Server: The collected audio and video data is sent to the server via the network. The server uses high-performance speech recognition technology, natural language processing technology, and an emotion engine to analyze the data.

[0241] Program processing

[0242] Collection and transmission of audio and video data

[0243] First, when a user starts a conversation, the smart glasses or smartphone collects audio and video data. For example, if collecting audio and facial expressions during a meeting, collection begins automatically as soon as the meeting begins. The collected data undergoes pre-processing such as noise filtering and video cleaning. This data is then sent to a server in real time.

[0244] Data analysis and understanding of intentions and emotions

[0245] The server converts the received voice data into text data using speech recognition technology. The text data is then subjected to syntactic analysis, which identifies the subject, predicate, and object of the sentence and associates them with the context. At the same time, the video data is analyzed by an emotion engine, which recognizes emotions from the user's facial expressions and tone of voice. For example, the question "How is the project progressing?" is analyzed as "You are asking about the progress of the project," and it is recognized that the user is feeling nervous.

[0246] Summarization and response generation

[0247] The intent and emotions that have been captured are summarized on the server, generating information to present to the user in an easy-to-understand format. For example, it may summarize the message as "Your boss is asking about the progress of the project." The generation AI also generates multiple effective responses to utter next. Response candidates are customized based on the user's past speech patterns and preferences. Additionally, the response content is adjusted based on the user's emotions as recognized by the emotion engine.

[0248] Presentation of results

[0249] The summarized information and possible responses are sent to the device and displayed on the smartglasses display or smartphone screen. Voice guidance is also provided as needed. For example, the message might read, "Your boss is asking about the progress of a project. What do you think of the following response?"

[0250] Specific examples

[0251] Business Meeting Scenario

[0252] 1. User: Puts on smart glasses as soon as a business meeting begins.

[0253] 2. Terminal: Collects audio and video data, performs noise filtering and video preprocessing, and then sends it to the server.

[0254] 3. Server: Converts the voice data into text and performs syntax analysis and sentiment analysis. For example, it analyzes "My boss is asking about the progress of the project" and recognizes that the user is nervous.

[0255] 4. Server: Generates response candidates and presents options such as:

[0256] "The project is on schedule and is currently 70% complete." (In a calm tone to ease tension)

[0257] "We're experiencing delays with some tasks, but we expect them to be completed by next week." (Providing reassurance with specific numbers)

[0258] "Would you like more detailed progress reports?" (Supportive attitude)

[0259] 5. Device: The generated summary and response suggestions are displayed on the smart glasses display. For example, it may say, "Your boss is asking about the progress of the project. What do you think of the following response?"

[0260] 6. User: Select the appropriate response from the presented options and execute it in the conversation.

[0261] In this way, the "AI Conversation Advisor" can assist users in conversations in real time, avoiding misunderstandings and miscommunication while providing appropriate responses that take emotions into account.

[0262] The processing flow will be explained below.

[0263] Step 1:

[0264] User: Turns on the smart glasses device or smartphone and collects audio and video data of the conversation. For example, the moment a user starts speaking in a business meeting, audio and video data collection begins automatically.

[0265] Step 2:

[0266] Terminal: Noise filtering is performed on the collected audio data to remove unwanted noise, and video cleaning is performed on the video data to generate clearer images.

[0267] Step 3:

[0268] Terminal: After noise filtering and image cleaning, the data is sent to the server in real time via a secure communication protocol.

[0269] Step 4:

[0270] Server: Converts the received voice data into text data using voice recognition technology. Stores the converted text data.

[0271] Step 5:

[0272] Server: Analyzes the received video data and combines facial expression recognition algorithms and tone analysis of audio data to identify the user's emotions. For example, it recognizes whether the user is nervous or angry.

[0273] Step 6:

[0274] Server: Performs syntactic analysis on text data to extract components such as the subject, predicate, and object of a sentence. It also performs contextual analysis to understand the intention and topic of what is being said.

[0275] Step 7:

[0276] Server: Based on the results of syntactic and contextual analysis, summarizes what the other person is saying. For example, it summarizes it as "your boss is asking about the progress of the project."

[0277] Step 8:

[0278] Server: Based on the summarized information and the user's emotions recognized by the emotion engine, the server generates effective response candidates for the next utterance. For example, if the user is nervous, the server generates a response in a calm tone.

[0279] Step 9:

[0280] Server: Formats the generated summary and candidate responses and sends them to the user terminal. Data is transmitted end-to-end encrypted.

[0281] Step 10:

[0282] Device: A formatted summary and suggested responses are displayed on the smartglasses display or smartphone screen. Voice guidance is also possible if necessary. For example, the display might say, "Your boss is asking about the progress of the project. What do you think of the following response?"

[0283] Step 11:

[0284] User: Choose the appropriate response from the suggested responses and use it in the actual conversation.

[0285] Through this series of steps, the AI ​​conversation advisor can assist users in real-time conversations, avoiding misunderstandings and miscommunications, and providing appropriate responses that take emotions into account.

[0286] Example 2

[0287] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0288] Conventional conversation support systems were unable to fully grasp the intentions and emotions of users' remarks, resulting in misunderstandings and miscommunication. Furthermore, due to a lack of technology to recognize emotions from video data in addition to analyzing audio data, it was difficult to provide accurate responses in real time. This meant that users had to perform many manual operations to obtain the information they needed, making it difficult to support smooth communication.

[0289] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting audio data and video data of a conversation, means for transmitting the collected audio data and video data in real time, means for converting the transmitted audio data into text data, means for analyzing the syntax of the text data and recognizing the user's emotions, means for summarizing the analyzed data and information obtained from the emotion recognition, means for generating information that would be effective to utter next based on the summarized information, and means for presenting the generated information to the user. This makes it possible to accurately analyze the user's intentions and emotions based on the collected data and generate and present an appropriate response in real time.

[0290] "Means for collecting audio data and video data of a conversation" is a general term for devices equipped with a microphone and a camera for capturing the user's voice and video in real time.

[0291] The "means for transmitting the collected audio data and video data in real time" refers to a communication function for instantly transmitting the collected data to a server.

[0292] The "means for converting the transmitted voice data into text data" refers to a method for converting voice data into text format using voice recognition technology.

[0293] The "means for analyzing the syntax of the text data and recognizing the user's emotions" refers to a function for analyzing the grammatical structure of the text data using natural language processing technology and recognizing the user's emotions.

[0294] The "means for summarizing the analyzed data and information obtained from emotion recognition" refers to a function for summarizing information in order to display the analysis results in a concise manner.

[0295] "Means for generating information that would be effective to utter next based on the summarized information" refers to a technology that generates a response that would be effective for the user to utter next based on summarized information.

[0296] "Means for presenting the generated information to the user" refers to an interface including a display and a speaker for visually or audibly presenting the generated response candidates to the user.

[0297] MODE FOR CARRYING OUT THE INVENTION

[0298] This invention relates to a system that supports user conversations in real time by combining audio data and video data. The system consists of a terminal worn by the user and a server that processes the data.

[0299] Device configuration

[0300] Users use devices such as smart glasses and smartphones. These devices include the following hardware:

[0301] 1. Microphone: To collect audio data of conversation.

[0302] 2. Camera: To capture the user's facial expressions and collect video data.

[0303] 3. Display: For showing summarized information and possible responses.

[0304] 4. Communication module: To transmit collected data to the server in real time.

[0305] Server Configuration

[0306] The server includes the following software modules and functions:

[0307] 1. Speech recognition technology: Used to convert voice data into text data. Specifically, speech recognition engines include Google Cloud Speech-to-Text API and IBM Watson Speech to Text.

[0308] 2. Natural language processing technology: Used to analyze the syntax of text data and understand the user's intent. For example, libraries such as Spacy and NLTK are used.

[0309] 3. Emotion Engine: Recognizes user emotions from video data and analyzes voice tone.

[0310] 4. Generative AI models: Used to generate effective next utterances. For example, OpenAI's GPT-4.

[0311] 5. Summarization module: Summarizes the acquired information and presents it to the user.

[0312] Specific examples

[0313] Business Meeting Scenario

[0314] 1. User: The user puts on the smart glasses as soon as the business meeting begins.

[0315] 2. Terminal: Collects audio and video data, performs noise filtering and video preprocessing, and then sends it to the server.

[0316] 3. Server: Converts the voice data into text and performs syntax analysis and sentiment analysis. For example, it analyzes the speech data as "your boss is asking about the progress of the project" and recognizes that the user is nervous.

[0317] 4. Server: Generates response candidates and presents options such as:

[0318] "The project is on schedule and is currently 70% complete." (In a calm tone to ease tension)

[0319] "We're experiencing delays with some tasks, but we expect them to be completed by next week." (Providing reassurance with specific numbers)

[0320] "Would you like more detailed progress reports?" (Supportive attitude)

[0321] 5. Terminal: The generated summary and response suggestions are displayed on the screen. For example, it may be displayed in the form of "Your boss is asking about the progress of the project. What do you think of the following response?"

[0322] 6. User: Select the appropriate response from the presented options and execute it in the conversation.

[0323] Prompt Sentence Examples

[0324] "Please advise me on how to respond in a tense situation when someone asks me how the project is going."

[0325] "Generate phrases to speak in a calm tone when reporting an ongoing project to your boss."

[0326] The above is a specific embodiment for carrying out the present invention. This system effectively combines audio and video data to support user conversations in real time, thereby facilitating user communication and preventing misunderstandings and miscommunication.

[0327] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0328] Step 1: User initiates conversation

[0329] A user puts on smart glasses or a smartphone and starts a conversation. Specifically, the device automatically detects the start of a conversation, or the user manually instructs the device to start a conversation. The input is the user's voice and video, and the output is the device collecting these data.

[0330] Step 2: Collect and preprocess audio and video data

[0331] The device collects audio and video data of the conversation. It uses a microphone to pick up the audio and a camera to capture the user's face. The input is the collected audio and video data, and the output is preprocessed data. Specific preprocessing steps include reducing background noise from the audio data and adjusting the brightness and contrast of the video data.

[0332] Step 3: Send data from the device to the server

[0333] The terminal transmits the preprocessed data to the server in real time. The input is the preprocessed audio and video data, and the output is the data transmitted to the server. The data is converted into a packet format and transmitted using a secure communication protocol.

[0334] Step 4: Convert audio data to text

[0335] The server converts the received voice data into text data using voice recognition technology. The input is voice data and the output is text data. Specifically, it uses a voice recognition engine (e.g., Google Cloud Speech-to-Text API) to convert voice into text in real time.

[0336] Step 5: Parsing and Emotion Recognition

[0337] The server performs syntactic analysis of the text data and simultaneously performs emotion recognition using video data. The input is text data and video data, and the output is an analysis result that includes intent and emotion. A natural language processing library (e.g., Spacy) is used to identify the subject, predicate, and object of the sentence. At the same time, the emotion engine analyzes facial expressions and voice tone to recognize the user's emotions.

[0338] Step 6: Summarize and generate a response

[0339] The server summarizes the analysis results and generates effective information for the next user utterance. The input is the analysis results, and the output is summarized information and response candidates. A generative AI model (e.g., GPT-4) is used to generate multiple next utterances. Responses are customized based on past utterance patterns and the user's preferences.

[0340] Step 7: Summarize and present response candidates to the device

[0341] The server sends the generated summary and candidate responses to the terminal. The input is the summarized information and candidate responses, and the output is the information displayed on the terminal. The terminal receives this and displays on its display, "Your boss is asking about the progress of the project. What do you think of the following response?"

[0342] Step 8: User selects and executes the best response

[0343] The user selects an appropriate response from the candidate responses displayed on the device and executes it in the conversation. The input is the displayed candidate response, and the output is the confirmed response selected by the user. The user selects a response using the touch interface or voice commands and executes the response in the actual conversation.

[0344] (Application example 2)

[0345] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0346] In autonomous vehicles, communication with passengers and drivers can be difficult, and a major issue is the lack of appropriate instructions and information provided in emergencies. Furthermore, there is a need for a flexible system that can respond to various situations that arise while driving.

[0347] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for transmitting audio data and video data of a conversation in real time, means for analyzing the transmitted audio data and video data, and means for extracting the intention and emotion of the utterance from the analyzed data. This makes it possible to identify the intention and emotion of the speaker through the analysis of the audio and video data in real time and provide appropriate responses and instructions.

[0348] "Conversational voice data" refers to voice signals uttered by a user, and includes linguistic information and emotional information.

[0349] "Video data" refers to video signals collected to capture a user's facial expressions and movements, and includes visual information.

[0350] "Means for transmitting in real time" refers to a function for transmitting collected audio and video data to a server immediately without delay.

[0351] "Means for analysis" refers to software or algorithms used to analyze transmitted audio and video data, thereby enabling understanding of intent and emotion.

[0352] "Means of understanding intent" refers to the function of understanding the speaker's purpose and intention from the analyzed data.

[0353] "Means of capturing emotions" refers to the function of identifying the speaker's emotional state from the analyzed data.

[0354] "Means of summarization" refers to the function of summarizing analyzed information in a concise and easy-to-understand form.

[0355] "Means for generating information that will be effective to utter next" refers to the function for generating appropriate content to be uttered next based on the intention and emotion that has been grasped.

[0356] "Presentation means" refers to a device or function for visually or audibly presenting the generated information to the user.

[0357] "Means for providing appropriate instructions in an emergency" refers to a function that provides users with prompt and accurate instructions based on the results of data analysis in an emergency situation.

[0358] "Noise filtering" refers to the process of removing unwanted background noise from collected audio data.

[0359] "Image processing" refers to the process of converting collected video data into a form that is easier to clean and analyze.

[0360] "Normalizing the volume" refers to the process of adjusting the volume level of audio data to keep it constant.

[0361] System program generation and hardware / software configuration

[0362] System Configuration

[0363] This system includes multiple devices and servers for collecting and analyzing audio and video data of conversations in real time. Details of each element are given below.

[0364] 1. User Device:

[0365] Smart glasses: Worn by passengers or drivers, these glasses contain microphones and cameras for real-time audio and video capture.

[0366] Smartphones: Can be used as auxiliary devices to collect audio and video data.

[0367] 2. Server:

[0368] Speech recognition system: Uses the speech_recognition library to convert voice data into text.

[0369] Emotion analysis system: Uses the emotion_recognition library to identify emotions from collected audio and video data.

[0370] Natural language processing engine: Uses generative AI models to generate responses based on the speaker's intent and sentiment.

[0371] Data Processing

[0372] 1. Audio and video data collection and transmission:

[0373] Smart glasses or smartphones collect real-time audio and video data of conversations, which undergo local noise filtering and image processing before being sent to a server.

[0374] 2. Audio and video data analysis:

[0375] The server converts the received audio data into text data using the speech_recognition library, then performs syntactic analysis on the text data and combines it with context.

[0376] Next, the collected video data is analyzed using the emotion_recognition library to identify the speaker's emotions.

[0377] 3. Understanding intentions and emotions and summarizing information:

[0378] The system identifies the speaker's intention and emotion from the analyzed data and summarizes this information. For example, if the question "Where is the next exit?" is analyzed and the speaker is found to be impatient, it can be summarized as "The speaker is in a hurry to find out where the next exit is."

[0379] 4. Response generation and presentation:

[0380] Based on the summarized information, a generative AI model is used to generate appropriate responses, which are customized based on the user's past speech patterns and set preferences.

[0381] The generated response will be displayed on the smartglasses display or smartphone screen, and will also be provided as an audio guide if necessary.

[0382] Examples and prompts

[0383] Example 1: Emergency response on expressways

[0384] Scenario: Providing emergency information when a driver brakes suddenly.

[0385] Voice input: A passenger asks, "Where is the next exit?"

[0386] Emotion analysis: Detect anxiety from the tone of voice.

[0387] AI response: Instruct: "The next exit is 3 kilometers away. Please remain calm and keep driving."

[0388] Example prompt sentence:

[0389] Consider the user's nervousness and create a guiding message like this: 'Where is the next exit?' Answer in a calm tone.

[0390] Thus, the embodiment of the present invention includes a series of processes that collects audio and video data, analyzes it to understand the user's intentions and emotions, and provides appropriate information. In particular, it enables quick and accurate instructions in emergencies while driving, thereby improving the safety and comfort of the user.

[0391] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0392] Step 1:

[0393] The user device collects audio and video data of the conversation. Specifically, the microphone and camera built into smart glasses or smartphones record and record the user's speech and facial expressions in real time. The input is the user's voice and facial expressions, and the output is the collected audio and video data.

[0394] Step 2:

[0395] The terminal transmits collected audio and video data to the server in real time. Specifically, after noise filtering and preprocessing, the data is transmitted to the server via the network. The input is the collected audio and video data, and the output is the transmission of the preprocessed data to the server.

[0396] Step 3:

[0397] The server converts the received voice data into text data using the speech_recognition library. Specifically, it analyzes the voice signal using a speech recognition algorithm and converts the results into text format. The input is preprocessed voice data, and the output is text data.

[0398] Step 4:

[0399] The server analyzes the received video data using the emotion_recognition library to identify the user's emotions. Specifically, it analyzes facial expressions and vocal tone to identify the speaker's emotional state. The input is preprocessed video data, and the output is identified emotional information.

[0400] Step 5:

[0401] The server uses data obtained from speech recognition and emotion analysis to understand the speaker's intent and summarize the information. Specifically, it uses natural language processing algorithms to understand the gist of what is being said and create a concise summary. The input is text data and emotion information, and the output is summarized information.

[0402] Step 6:

[0403] The server uses the generative AI model to generate information that will be effective next time. Specifically, it generates an appropriate response based on the summarized information and the speaker's emotional state. The input is the summarized information and emotional information, and the output is the generated response.

[0404] Step 7:

[0405] The device presents the generated response. Specifically, the response is displayed on the smartglasses display or smartphone screen, and audio guidance is also provided if necessary. The input is the generated response, and the output is the information presented to the user.

[0406] This series of processes allows users to receive accurate information and instructions in real time. In particular, in emergencies, safety can be improved by providing prompt and appropriate instructions according to the user's emotional state.

[0407] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0408] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0409] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0410] [Second embodiment]

[0411] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0412] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0413] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0414] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0415] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0416] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0417] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0418] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0419] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0420] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0421] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0422] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0423] This invention relates to an "AI Conversation Advisor" that utilizes generative AI to provide conversation support and advice on what to say. This system collects and analyzes voice data in real time and provides appropriate responses to users, thereby supporting smooth communication.

[0424] System Overview

[0425] 1. User device: The user uses a smart glasses-type device or a smartphone. These devices are equipped with a microphone to collect conversations between the user and others in real time, and a display or audio output to show the results.

[0426] 2. Server: The collected voice data is sent to the server via the network, where it is analyzed using high-performance voice recognition and natural language processing technologies.

[0427] Program processing

[0428] Collection and transmission of voice data

[0429] First, when a user starts a conversation, the smart glasses or smartphone collects audio data. For example, when collecting audio during a meeting, collection begins automatically as soon as the meeting begins. The collected audio data undergoes pre-processing such as noise filtering to make it clear audio data. This audio data is then sent to a server in real time.

[0430] Analyzing voice data and understanding intent

[0431] The server uses speech recognition technology to convert the received voice data into text. The text data is then subjected to syntactic analysis, which identifies the subject, predicate, and object of the sentence and associates them with the context. Contextual and sentiment analysis are then performed to understand the intention and emotion behind the other person's remarks. For example, if the question includes "How is the project progressing?", it is interpreted as "You are asking about the progress of the project."

[0432] Summarization and response generation

[0433] The intent that is captured is summarized on the server and put into a form that is easy for the user to understand. For example, it may be summarized as "Your boss is asking about the progress of the project." The generative AI also generates multiple effective responses to utter next. Response candidates are customized based on the user's past speech patterns and set preferences.

[0434] Presentation of results

[0435] The summarized information and possible responses are sent to the device and displayed on the smartglasses display or smartphone screen. Voice guidance is also provided as needed. For example, the message might read, "Your boss is asking about the progress of a project. What do you think of the following response?"

[0436] Specific examples

[0437] Business Meeting Scenario

[0438] 1. User: Puts on smart glasses as soon as a business meeting begins.

[0439] 2. Terminal: Collects voice data, filters it, and then sends it to the server.

[0440] 3. Server: Converts the speech data into text and performs syntax and sentiment analysis. Example: "Your boss is asking about the progress of the project."

[0441] 4. Server: Generates candidate responses and presents options such as:

[0442] "The project is progressing as scheduled and is currently 70% complete."

[0443] "There are delays to some tasks, but we expect them to be completed by next week."

[0444] "Would you like a more detailed progress report?"

[0445] 5. Device: The generated summary and response suggestions are displayed on the smart glasses display. For example, it shows, "Your boss is asking about the progress of the project. What do you think of the following response?"

[0446] 6. User: Select the appropriate response from the suggested options and execute it in the conversation.

[0447] In this way, the "AI Conversation Advisor" will be able to assist users in their conversations in real time, allowing for smoother communication.

[0448] The processing flow will be explained below.

[0449] Step 1:

[0450] User: Turn on the smart glasses or smartphone and start collecting voice data of the conversation. For example, the moment the user starts speaking in a business meeting, voice data collection starts automatically.

[0451] Step 2:

[0452] Terminal: Noise filtering is performed on the collected voice data to remove background noise, etc. This allows for clear voice data to be obtained.

[0453] Step 3:

[0454] Terminal: Sends noise-filtered audio data to the server in real time. The data is transferred using a secure communication protocol.

[0455] Step 4:

[0456] Server: The received voice data is converted into text data using voice recognition technology. A highly accurate voice recognition engine is used in this process.

[0457] Step 5:

[0458] Server: Performs syntactic analysis on the text data to identify the components of a sentence, such as the subject, predicate, and object.

[0459] Step 6:

[0460] Server: Further contextual and sentiment analysis is performed to understand the intent and sentiment of the statement. For example, if the question "How is the project progressing?" is included, it will be interpreted as "Ask about the progress of the project."

[0461] Step 7:

[0462] Server: Summarizes the intent and generates information to present to the user in an easy-to-understand format. For example, it might summarize the intent as "The boss is asking about the progress of the project."

[0463] Step 8:

[0464] Server: Based on the generated summary, the server generates multiple effective responses for the next utterance. The responses are customized based on the user's past utterance patterns and preferences.

[0465] Step 9:

[0466] Server: Formats the generated summary and response candidates and sends them to the user's terminal.

[0467] Step 10:

[0468] Device: The summary and response candidates are displayed on the smartglasses display or smartphone screen. Voice guidance is also possible. For example, the message may say, "Your boss is asking about the progress of the project. What do you think of the following response?"

[0469] Step 11:

[0470] User: Choose the appropriate response from the suggested responses and use it in the actual conversation.

[0471] Through this series of steps, the "AI Conversation Advisor" supports users' conversations in real time, avoiding misunderstandings and miscommunication and ensuring smooth communication.

[0472] Example 1

[0473] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0474] During a conversation, it can be difficult for users to instantly grasp and respond appropriately. This is especially true in business meetings and important discussions, where it is necessary to respond appropriately on the spot. However, conventional technology lacks the means to analyze the content of a conversation in real time and provide appropriate responses. To solve this problem, a system that allows users to respond smoothly is needed.

[0475] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0476] In this invention, the server includes means for using a device to collect voice data, means for noise filtering the collected voice data in real time and transmitting the data to the server, means for converting the transmitted voice data into text using voice recognition technology, means for syntactically analyzing and sentiment analyzing the converted text data, means for extracting the intention of the utterance from the analyzed data, means for summarizing the intention of the utterance, means for generating information on effective next utterances using a generative AI model, and means for presenting the generated information on a user device, thereby enabling the user to grasp appropriate responses in real time and to communicate smoothly and effectively.

[0477] "Audio data" refers to audio signals of conversations or utterances made by a user.

[0478] "Device" refers to a hardware device for collecting audio data, including, for example, smart glasses and smartphones.

[0479] A "server" refers to a computer system that is connected via a network and processes and analyzes data.

[0480] "Noise filtering" refers to the process of removing unwanted background sounds and noise from collected audio data.

[0481] "Speech recognition technology" refers to technology for analyzing collected voice data and converting voice into text.

[0482] "Syntax analysis" refers to the process of analyzing the grammatical structure of text data and clarifying the relationships between the subject, predicate, object, and other parts of a sentence.

[0483] "Sentiment analysis" refers to the process of inferring the emotional state of a speaker from text data.

[0484] "Intention of speech" refers to understanding what the speaker intended based on the analyzed data.

[0485] "Summarization" means summarizing analyzed information into a concise and easy-to-understand format.

[0486] "Generative AI model" refers to an artificial intelligence model that performs natural language processing and is used to generate new text and responses.

[0487] The term "user device" refers to a device used by a user for displaying information and executing voice guidance.

[0488] This invention relates to an "AI Conversation Advisor" that utilizes generative AI to provide conversation support and advice on what to say. This system collects and analyzes voice data in real time and presents appropriate responses to users to support smooth communication.

[0489] System Overview

[0490] 1. User Devices

[0491] Users use smart glasses or smartphones, which are equipped with a microphone to collect conversations between the user and others in real time and a display or audio output to display the analysis results.

[0492] 2. Server

[0493] The collected voice data is sent to a server via a network. The server then uses high-performance voice recognition and natural language processing technologies to analyze the data. Specifically, it converts the voice data into text using voice recognition technology (e.g., Google Speech-to-Text API) and generates an optimal response using a generative AI model (e.g., OpenAI's GPT-4).

[0494] Program processing

[0495] First, when a user starts a conversation, the smart glasses or smartphone collects audio data. For example, when collecting audio during a meeting, collection begins automatically as soon as the meeting begins. The collected audio data undergoes pre-processing such as noise filtering on the device to create clear audio data. This audio data is then sent to the server in real time.

[0496] The server converts the received voice data into text using speech recognition technology. It then uses a generative AI model to perform syntax analysis and sentiment analysis on the text data to understand the intent and emotion of the other person's speech. For example, if the question "How is the project progressing?" is included, it can be interpreted as "A question about the progress of the project."

[0497] The intent that is captured is summarized on the server and put into a form that is easy for the user to understand. For example, it might summarize it as "your boss is asking about the progress of the project." It also uses a generative AI model to generate multiple effective next utterances. Response candidates are customized based on the user's past utterance patterns and set preferences.

[0498] The summarized information and possible responses are sent to the device and displayed on the smartglasses display or smartphone screen. Voice guidance is also provided as needed. For example, the message might read, "Your boss is asking about the progress of a project. What do you think of the following response?"

[0499] Specific examples

[0500] Business Meeting Scenario

[0501] 1. User: Puts on smart glasses as soon as a business meeting begins.

[0502] 2. Terminal: Collects voice data, filters it, and then sends it to the server.

[0503] 3. Server: Converts the speech data into text and performs syntax and sentiment analysis. Example: "Your boss is asking about the progress of the project."

[0504] 4. Server: Generates candidate responses and presents options such as:

[0505] "The project is progressing as scheduled and is currently 70% complete."

[0506] "There are delays to some tasks, but we expect them to be completed by next week."

[0507] "Would you like a more detailed progress report?"

[0508] 5. Device: The generated summary and response suggestions are displayed on the smart glasses display. For example, it shows, "Your boss is asking about the progress of the project. What do you think of the following response?"

[0509] 6. User: Select the appropriate response from the suggested options and execute it in the conversation.

[0510] Prompt Sentence Examples

[0511] Below are some example prompts to input to a generative AI model (e.g., GPT-4):

[0512] "If your boss is asking about the progress of a project, generate an appropriate response."

[0513] This will cause the AI ​​to generate a response like this:

[0514] 1. "The project is on schedule and is currently 70% complete."

[0515] 2. "We're running behind on some tasks, but we expect to complete them next week."

[0516] 3. "Would you like more detailed progress reports?"

[0517] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0518] Step 1: Collect and preprocess audio data

[0519] The user puts on the smart glasses or smartphone and launches the application.

[0520] Input: Real-time conversational audio produced by the user.

[0521] The device uses the device's microphone to capture audio data in real time. For example, if you're in a meeting, audio collection begins as soon as it starts.

[0522] The device performs noise filtering on the acquired voice data to remove background noise (for example, noise reduction processing using FFT).

[0523] Output: Clear audio data after filtering.

[0524] Step 2: Sending audio data

[0525] The device transmits the pre-processed voice data to the server in real time.

[0526] Input: Clear audio data after filtering.

[0527] Terminals may compress data to improve communication efficiency.

[0528] Output: The compressed audio data is sent over the network to the server.

[0529] Step 3: Convert audio data to text

[0530] The server converts the received voice data into text data using voice recognition technology (e.g., Google Speech-to-Text API).

[0531] Input: Compressed and transmitted audio data.

[0532] The server extracts linguistic features from the audio and converts it into text format.

[0533] Output: The converted text data.

[0534] Step 4: Parsing and sentiment analysis of text data

[0535] The server performs syntactic analysis on the generated text data, thereby identifying the subject, predicate, and object of the sentence.

[0536] Input: The converted text data.

[0537] The server then performs a sentiment analysis to assess the emotional state of the speaker.

[0538] Output: Parsed data and sentiment analysis results.

[0539] Step 5: Understand and summarize the intention of the statement

[0540] The server uses the analyzed data to understand the intent behind the statement.

[0541] Input: Syntactic and sentiment analysis results.

[0542] The server summarizes the intent of the conversation and condenses it into a form that is easy for the user to understand, such as "your boss is asking about the progress of a project."

[0543] Output: Summarized intent data.

[0544] Step 6: Generate candidate responses

[0545] The server uses a generative AI model (e.g., OpenAI's GPT-4) to generate multiple effective next utterances.

[0546] Input: Summarized intent data.

[0547] The server feeds example prompts to the model, which generates appropriate response patterns, taking into account the user's past speech patterns and preferences.

[0548] Output: The generated list of possible responses.

[0549] Step 7: Presenting results and user response

[0550] The server transmits the summarized information and candidate responses to the user terminal.

[0551] Input: The generated response candidate list.

[0552] The device will then display this information on the smartglasses display or smartphone screen, for example, in the form of, "Your boss is asking you about the progress of a project. What do you think of the following response?"

[0553] The user selects the appropriate response from the presented options and executes it in the actual conversation. Selection can be done by voice input or touch operation.

[0554] Output: The response selected by the user.

[0555] (Application example 1)

[0556] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0557] There is a need to improve the quality of customer interactions in brick-and-mortar stores and simultaneously increase sales staff efficiency and customer satisfaction. In particular, there is a lack of support tools to enable appropriate and prompt responses during customer service, and technology to ensure smooth communication with customers is needed. There is also a need for a system that can provide consistent, high-quality customer service that is not dependent on the experience or skills of staff.

[0558] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0559] In this invention, the server includes a preprocessing means for performing noise filtering on voice data collected using a smart device, a means for converting the voice data sent to the server into text and performing syntax analysis and sentiment analysis, and a means for displaying the generated response candidates and summaries on the display of the smart device. This makes it possible to support conversations with customers in real time using the smart device, allowing sales staff to respond to customers more quickly and accurately.

[0560] "Conversational voice data" refers to data in which the voice of a user or customer is recorded in digital format.

[0561] "Means for transmitting in real time" refers to technology for transmitting collected voice data to a server immediately without delay.

[0562] "Means for analyzing" refers to the technology or system that converts transmitted voice data into text data and understands grammar and context.

[0563] "Means for understanding the intention of a statement" refers to technology for extracting the speaker's intentions and emotions from analyzed text data.

[0564] "Means of summarizing" refers to the technique of summarizing the intent and information that has been extracted into a concise, to-the-point format.

[0565] The "means for generating effective information" is a technology for automatically generating information that is determined to be effective when spoken next.

[0566] "Means for presenting" refers to a technique for visually or audibly notifying the user of the generated information.

[0567] "Smart device" refers to an electronic device that can be carried or worn by a user and has the function of collecting and displaying voice data.

[0568] "Noise filtering" is a technique for removing background noise from audio data to obtain clear audio data.

[0569] "Syntax analysis" is a technique for analyzing the grammatical structure of text data and identifying elements such as subjects, predicates, and objects.

[0570] "Sentiment analysis" is a technology that identifies emotions from a speaker's text data and classifies the type of emotion.

[0571] "Response candidates" refer to multiple appropriate responses to be made next.

[0572] A "summary" is a concise summary of the main points and important information of a conversation.

[0573] "Means for displaying on a display" refers to technology for displaying the generated information on the screen of a smart device.

[0574] System configuration

[0575] A system for implementing the present invention includes the following hardware and software.

[0576] Smart device: A device carried or worn by the user that has the ability to collect voice data and display the results. Examples include smart glasses and smartphones.

[0577] Server: Has high-performance computing resources to perform voice data analysis, syntax analysis, and sentiment analysis. It mainly uses speech recognition APIs (e.g., Google Speech-to-Text API) and natural language processing APIs (e.g., OpenAI GPT-3).

[0578] Program Description

[0579] Collection and transmission of voice data

[0580] Smart devices (smart glasses or smartphones) collect the user's voice data in real time using a microphone built into the device. The collected voice data is filtered to make it clear and then sent to a server.

[0581] Analyzing voice data and understanding intent

[0582] The server converts the transmitted voice data into text using a speech recognition API (Google Speech-to-Text API). A natural language processing API (OpenAI GPT-3) is used to perform syntactic analysis and sentiment analysis on the converted text data. Syntactic analysis identifies the subject, predicate, and object, while sentiment analysis extracts the speaker's intention and emotion from the context.

[0583] Summarization and response generation

[0584] Based on the intent, the server summarizes the key points. The AI ​​then generates multiple effective responses for the next utterance. These responses are customized based on the user's past speech patterns and preferences.

[0585] Presentation of results

[0586] The summary and candidate responses generated by the server are displayed on the smart device's display. Voice guidance is also provided as needed. For example, a message such as, "A customer has a question about a new smartphone. What do you think of the following response?" can be displayed.

[0587] Specific examples

[0588] Scene: Customer service in a brick-and-mortar store

[0589] 1. A user puts on the smart glasses and starts a conversation with a customer.

[0590] 2. The smart device collects the voice data and sends it to the server along with noise filtering.

[0591] 3. The server converts the voice data into text and performs syntax and sentiment analysis. Example: "A customer is asking about the features of their smartphone."

[0592] 4. The server generates a candidate response, for example:

[0593] "This smartphone has an excellent camera."

[0594] "Long battery life and convenient"

[0595] Want to know more about the features?

[0596] 5. The generated summary and response suggestions are displayed on the smart glasses.

[0597] Prompt Sentence Examples

[0598] A customer has asked a question about a new smartphone. Generate possible responses.

[0599] In this way, the system can assist sales staff in serving customers in real time and improve customer satisfaction.

[0600] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0601] Step 1:

[0602] Smart devices (smart glasses or smartphones) collect voice data of conversations between users and customers in real time. The input is the voice data of the conversation, and the output is the collected voice data. The voice data is acquired using a microphone, and clear voice data is prepared by noise filtering. This voice data is then passed to the next step.

[0603] Step 2:

[0604] The smart device sends noise-filtered audio data to the server. The input is clear audio data, and the output is the audio data sent to the server. Here, the data is sent in real time using the device's communication module.

[0605] Step 3:

[0606] The server converts the transmitted voice data into text data. The input is voice data and the output is text data. The server uses a speech recognition API (Google Speech-to-Text API) to analyze the voice waveform data and generate a corresponding string.

[0607] Step 4:

[0608] The server analyzes the converted text data and identifies grammatical structures. The input is the text data, and the output is the results of the analysis. This process uses a natural language processing API (OpenAI GPT-3) to identify sentence elements such as subject, predicate, and object.

[0609] Step 5:

[0610] The server performs sentiment analysis based on the parsing results. The input is the parsing results, and the output is the intention of the utterance and the sentiment analysis results. The sentiment analysis module is used to determine the speaker's emotional state and intention. This information is used for further processing.

[0611] Step 6:

[0612] The server summarizes the main points of the speech based on the results of sentiment analysis. The input is the intention of the speech and the results of sentiment analysis, and the output is a summary. A summarization algorithm is used to extract the main points of the text and create a concise summary.

[0613] Step 7:

[0614] The server generates effective response candidates based on the summary. The input is the summary, and the output is multiple response candidates. Using a generative AI model (OpenAI GPT-3), multiple next-best utterances are generated.

[0615] Step 8:

[0616] The smart device receives the summary and candidate responses sent from the server and displays them on the display. The input is the summary and candidate responses, and the output is information provided to the user's visual sense. The display function of the device is used to visually present appropriate candidate responses to the user.

[0617] Step 9:

[0618] The user checks the response candidates presented by the smart device, selects the most appropriate one, and speaks to the customer. The input is the displayed response candidate, and the output is the response selected by the user. This enables smooth and effective customer service.

[0619] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0620] This invention relates to an "AI Conversation Advisor" that combines generative AI and an emotion engine to provide conversation support and advice on what to say. This system collects and analyzes voice data in real time, recognizes emotions, and then presents appropriate responses to the user, supporting smooth communication.

[0621] System Overview

[0622] 1. User device: The user uses a smart glass or smartphone, which is equipped with a microphone to collect real-time conversations between the user and others, a camera to capture the user's speech and facial expressions, and a display or audio output to show the results.

[0623] 2. Server: The collected audio and video data is sent to the server via the network. The server uses high-performance speech recognition technology, natural language processing technology, and an emotion engine to analyze the data.

[0624] Program processing

[0625] Collection and transmission of audio and video data

[0626] First, when a user starts a conversation, the smart glasses or smartphone collects audio and video data. For example, if collecting audio and facial expressions during a meeting, collection begins automatically as soon as the meeting begins. The collected data undergoes pre-processing such as noise filtering and video cleaning. This data is then sent to a server in real time.

[0627] Data analysis and understanding of intentions and emotions

[0628] The server converts the received voice data into text data using speech recognition technology. The text data is then subjected to syntactic analysis, which identifies the subject, predicate, and object of the sentence and associates them with the context. At the same time, the video data is analyzed by an emotion engine, which recognizes emotions from the user's facial expressions and tone of voice. For example, the question "How is the project progressing?" is analyzed as "You are asking about the progress of the project," and it is recognized that the user is feeling nervous.

[0629] Summarization and response generation

[0630] The intent and emotions that have been captured are summarized on the server, generating information to present to the user in an easy-to-understand format. For example, it may summarize the message as "Your boss is asking about the progress of the project." The generation AI also generates multiple effective responses to utter next. Response candidates are customized based on the user's past speech patterns and preferences. Additionally, the response content is adjusted based on the user's emotions as recognized by the emotion engine.

[0631] Presentation of results

[0632] The summarized information and possible responses are sent to the device and displayed on the smartglasses display or smartphone screen. Voice guidance is also provided as needed. For example, the message might read, "Your boss is asking about the progress of a project. What do you think of the following response?"

[0633] Specific examples

[0634] Business Meeting Scenario

[0635] 1. User: Puts on smart glasses as soon as a business meeting begins.

[0636] 2. Terminal: Collects audio and video data, performs noise filtering and video preprocessing, and then sends it to the server.

[0637] 3. Server: Converts the voice data into text and performs syntax analysis and sentiment analysis. For example, it analyzes "My boss is asking about the progress of the project" and recognizes that the user is nervous.

[0638] 4. Server: Generates response candidates and presents options such as:

[0639] "The project is on schedule and is currently 70% complete." (In a calm tone to ease tension)

[0640] "We're experiencing delays with some tasks, but we expect them to be completed by next week." (Providing reassurance with specific numbers)

[0641] "Would you like more detailed progress reports?" (Supportive attitude)

[0642] 5. Device: The generated summary and response suggestions are displayed on the smart glasses display. For example, it may say, "Your boss is asking about the progress of the project. What do you think of the following response?"

[0643] 6. User: Select the appropriate response from the presented options and execute it in the conversation.

[0644] In this way, the "AI Conversation Advisor" can assist users in conversations in real time, avoiding misunderstandings and miscommunication while providing appropriate responses that take emotions into account.

[0645] The processing flow will be explained below.

[0646] Step 1:

[0647] User: Turns on the smart glasses device or smartphone and collects audio and video data of the conversation. For example, the moment a user starts speaking in a business meeting, audio and video data collection begins automatically.

[0648] Step 2:

[0649] Terminal: Noise filtering is performed on the collected audio data to remove unwanted noise, and video cleaning is performed on the video data to generate clearer images.

[0650] Step 3:

[0651] Terminal: After noise filtering and image cleaning, the data is sent to the server in real time via a secure communication protocol.

[0652] Step 4:

[0653] Server: Converts the received voice data into text data using voice recognition technology. Stores the converted text data.

[0654] Step 5:

[0655] Server: Analyzes the received video data and combines facial expression recognition algorithms and tone analysis of audio data to identify the user's emotions. For example, it recognizes whether the user is nervous or angry.

[0656] Step 6:

[0657] Server: Performs syntactic analysis on text data to extract components such as the subject, predicate, and object of a sentence. It also performs contextual analysis to understand the intention and topic of what is being said.

[0658] Step 7:

[0659] Server: Based on the results of syntactic and contextual analysis, summarizes what the other person is saying. For example, it summarizes it as "your boss is asking about the progress of the project."

[0660] Step 8:

[0661] Server: Based on the summarized information and the user's emotions recognized by the emotion engine, the server generates effective response candidates for the next utterance. For example, if the user is nervous, the server generates a response in a calm tone.

[0662] Step 9:

[0663] Server: Formats the generated summary and candidate responses and sends them to the user terminal. Data is transmitted end-to-end encrypted.

[0664] Step 10:

[0665] Device: A formatted summary and suggested responses are displayed on the smartglasses display or smartphone screen. Voice guidance is also possible if necessary. For example, the display might say, "Your boss is asking about the progress of the project. What do you think of the following response?"

[0666] Step 11:

[0667] User: Choose the appropriate response from the suggested responses and use it in the actual conversation.

[0668] Through this series of steps, the AI ​​conversation advisor can assist users in real-time conversations, avoiding misunderstandings and miscommunications, and providing appropriate responses that take emotions into account.

[0669] Example 2

[0670] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0671] Conventional conversation support systems were unable to fully grasp the intentions and emotions of users' remarks, resulting in misunderstandings and miscommunication. Furthermore, due to a lack of technology to recognize emotions from video data in addition to analyzing audio data, it was difficult to provide accurate responses in real time. This meant that users had to perform many manual operations to obtain the information they needed, making it difficult to support smooth communication.

[0672] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting audio data and video data of a conversation, means for transmitting the collected audio data and video data in real time, means for converting the transmitted audio data into text data, means for analyzing the syntax of the text data and recognizing the user's emotions, means for summarizing the analyzed data and information obtained from the emotion recognition, means for generating information that would be effective to utter next based on the summarized information, and means for presenting the generated information to the user. This makes it possible to accurately analyze the user's intentions and emotions based on the collected data and generate and present an appropriate response in real time.

[0673] "Means for collecting audio data and video data of a conversation" is a general term for devices equipped with a microphone and a camera for capturing the user's voice and video in real time.

[0674] The "means for transmitting the collected audio data and video data in real time" refers to a communication function for instantly transmitting the collected data to a server.

[0675] The "means for converting the transmitted voice data into text data" refers to a method for converting voice data into text format using voice recognition technology.

[0676] The "means for analyzing the syntax of the text data and recognizing the user's emotions" refers to a function for analyzing the grammatical structure of the text data using natural language processing technology and recognizing the user's emotions.

[0677] The "means for summarizing the analyzed data and information obtained from emotion recognition" refers to a function for summarizing information in order to display the analysis results in a concise manner.

[0678] "Means for generating information that would be effective to utter next based on the summarized information" refers to a technology that generates a response that would be effective for the user to utter next based on summarized information.

[0679] "Means for presenting the generated information to the user" refers to an interface including a display and a speaker for visually or audibly presenting the generated response candidates to the user.

[0680] MODE FOR CARRYING OUT THE INVENTION

[0681] This invention relates to a system that supports user conversations in real time by combining audio data and video data. The system consists of a terminal worn by the user and a server that processes the data.

[0682] Device configuration

[0683] Users use devices such as smart glasses and smartphones. These devices include the following hardware:

[0684] 1. Microphone: To collect audio data of conversation.

[0685] 2. Camera: To capture the user's facial expressions and collect video data.

[0686] 3. Display: For showing summarized information and possible responses.

[0687] 4. Communication module: To transmit collected data to the server in real time.

[0688] Server Configuration

[0689] The server includes the following software modules and functions:

[0690] 1. Speech recognition technology: Used to convert voice data into text data. Specifically, speech recognition engines include Google Cloud Speech-to-Text API and IBM Watson Speech to Text.

[0691] 2. Natural language processing technology: Used to analyze the syntax of text data and understand the user's intent. For example, libraries such as Spacy and NLTK are used.

[0692] 3. Emotion Engine: Recognizes user emotions from video data and analyzes voice tone.

[0693] 4. Generative AI models: Used to generate effective next utterances. For example, OpenAI's GPT-4.

[0694] 5. Summarization module: Summarizes the acquired information and presents it to the user.

[0695] Specific examples

[0696] Business Meeting Scenario

[0697] 1. User: The user puts on the smart glasses as soon as the business meeting begins.

[0698] 2. Terminal: Collects audio and video data, performs noise filtering and video preprocessing, and then sends it to the server.

[0699] 3. Server: Converts the voice data into text and performs syntax analysis and sentiment analysis. For example, it analyzes the speech data as "your boss is asking about the progress of the project" and recognizes that the user is nervous.

[0700] 4. Server: Generates response candidates and presents options such as:

[0701] "The project is on schedule and is currently 70% complete." (In a calm tone to ease tension)

[0702] "We're experiencing delays with some tasks, but we expect them to be completed by next week." (Providing reassurance with specific numbers)

[0703] "Would you like more detailed progress reports?" (Supportive attitude)

[0704] 5. Terminal: The generated summary and response suggestions are displayed on the screen. For example, it may be displayed in the form of "Your boss is asking about the progress of the project. What do you think of the following response?"

[0705] 6. User: Select the appropriate response from the presented options and execute it in the conversation.

[0706] Prompt Sentence Examples

[0707] "Please advise me on how to respond in a tense situation when someone asks me how the project is going."

[0708] "Generate phrases to speak in a calm tone when reporting an ongoing project to your boss."

[0709] The above is a specific embodiment for carrying out the present invention. This system effectively combines audio and video data to support user conversations in real time, thereby facilitating user communication and preventing misunderstandings and miscommunication.

[0710] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0711] Step 1: User initiates conversation

[0712] A user puts on smart glasses or a smartphone and starts a conversation. Specifically, the device automatically detects the start of a conversation, or the user manually instructs the device to start a conversation. The input is the user's voice and video, and the output is the device collecting these data.

[0713] Step 2: Collect and preprocess audio and video data

[0714] The device collects audio and video data of the conversation. It uses a microphone to pick up the audio and a camera to capture the user's face. The input is the collected audio and video data, and the output is preprocessed data. Specific preprocessing steps include reducing background noise from the audio data and adjusting the brightness and contrast of the video data.

[0715] Step 3: Send data from the device to the server

[0716] The terminal transmits the preprocessed data to the server in real time. The input is the preprocessed audio and video data, and the output is the data transmitted to the server. The data is converted into a packet format and transmitted using a secure communication protocol.

[0717] Step 4: Convert audio data to text

[0718] The server converts the received voice data into text data using voice recognition technology. The input is voice data and the output is text data. Specifically, it uses a voice recognition engine (e.g., Google Cloud Speech-to-Text API) to convert voice into text in real time.

[0719] Step 5: Parsing and Emotion Recognition

[0720] The server performs syntactic analysis of the text data and simultaneously performs emotion recognition using video data. The input is text data and video data, and the output is an analysis result that includes intent and emotion. A natural language processing library (e.g., Spacy) is used to identify the subject, predicate, and object of the sentence. At the same time, the emotion engine analyzes facial expressions and voice tone to recognize the user's emotions.

[0721] Step 6: Summarize and generate a response

[0722] The server summarizes the analysis results and generates effective information for the next user utterance. The input is the analysis results, and the output is summarized information and response candidates. A generative AI model (e.g., GPT-4) is used to generate multiple next utterances. Responses are customized based on past utterance patterns and the user's preferences.

[0723] Step 7: Summarize and present response candidates to the device

[0724] The server sends the generated summary and candidate responses to the terminal. The input is the summarized information and candidate responses, and the output is the information displayed on the terminal. The terminal receives this and displays on its display, "Your boss is asking about the progress of the project. What do you think of the following response?"

[0725] Step 8: User selects and executes the best response

[0726] The user selects an appropriate response from the candidate responses displayed on the device and executes it in the conversation. The input is the displayed candidate response, and the output is the confirmed response selected by the user. The user selects a response using the touch interface or voice commands and executes the response in the actual conversation.

[0727] (Application example 2)

[0728] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0729] In autonomous vehicles, communication with passengers and drivers can be difficult, and a major issue is the lack of appropriate instructions and information provided in emergencies. Furthermore, there is a need for a flexible system that can respond to various situations that arise while driving.

[0730] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for transmitting audio data and video data of a conversation in real time, means for analyzing the transmitted audio data and video data, and means for extracting the intention and emotion of the utterance from the analyzed data. This makes it possible to identify the intention and emotion of the speaker through the analysis of the audio and video data in real time and provide appropriate responses and instructions.

[0731] "Conversational voice data" refers to voice signals uttered by a user, and includes linguistic information and emotional information.

[0732] "Video data" refers to video signals collected to capture a user's facial expressions and movements, and includes visual information.

[0733] "Means for transmitting in real time" refers to a function for transmitting collected audio and video data to a server immediately without delay.

[0734] "Means for analysis" refers to software or algorithms used to analyze transmitted audio and video data, thereby enabling understanding of intent and emotion.

[0735] "Means of understanding intent" refers to the function of understanding the speaker's purpose and intention from the analyzed data.

[0736] "Means of capturing emotions" refers to the function of identifying the speaker's emotional state from the analyzed data.

[0737] "Means of summarization" refers to the function of summarizing analyzed information in a concise and easy-to-understand form.

[0738] "Means for generating information that will be effective to utter next" refers to the function for generating appropriate content to be uttered next based on the intention and emotion that has been grasped.

[0739] "Presentation means" refers to a device or function for visually or audibly presenting the generated information to the user.

[0740] "Means for providing appropriate instructions in an emergency" refers to a function that provides users with prompt and accurate instructions based on the results of data analysis in an emergency situation.

[0741] "Noise filtering" refers to the process of removing unwanted background noise from collected audio data.

[0742] "Image processing" refers to the process of converting collected video data into a form that is easier to clean and analyze.

[0743] "Normalizing the volume" refers to the process of adjusting the volume level of audio data to keep it constant.

[0744] System program generation and hardware / software configuration

[0745] System Configuration

[0746] This system includes multiple devices and servers for collecting and analyzing audio and video data of conversations in real time. Details of each element are given below.

[0747] 1. User Device:

[0748] Smart glasses: Worn by passengers or drivers, these glasses contain microphones and cameras for real-time audio and video capture.

[0749] Smartphones: Can be used as auxiliary devices to collect audio and video data.

[0750] 2. Server:

[0751] Speech recognition system: Uses the speech_recognition library to convert voice data into text.

[0752] Emotion analysis system: Uses the emotion_recognition library to identify emotions from collected audio and video data.

[0753] Natural language processing engine: Uses generative AI models to generate responses based on the speaker's intent and sentiment.

[0754] Data Processing

[0755] 1. Audio and video data collection and transmission:

[0756] Smart glasses or smartphones collect real-time audio and video data of conversations, which undergo local noise filtering and image processing before being sent to a server.

[0757] 2. Audio and video data analysis:

[0758] The server converts the received audio data into text data using the speech_recognition library, then performs syntactic analysis on the text data and combines it with context.

[0759] Next, the collected video data is analyzed using the emotion_recognition library to identify the speaker's emotions.

[0760] 3. Understanding intentions and emotions and summarizing information:

[0761] The system identifies the speaker's intention and emotion from the analyzed data and summarizes this information. For example, if the question "Where is the next exit?" is analyzed and the speaker is found to be impatient, it can be summarized as "The speaker is in a hurry to find out where the next exit is."

[0762] 4. Response generation and presentation:

[0763] Based on the summarized information, a generative AI model is used to generate appropriate responses, which are customized based on the user's past speech patterns and set preferences.

[0764] The generated response will be displayed on the smartglasses display or smartphone screen, and will also be provided as an audio guide if necessary.

[0765] Examples and prompts

[0766] Example 1: Emergency response on expressways

[0767] Scenario: Providing emergency information when a driver brakes suddenly.

[0768] Voice input: A passenger asks, "Where is the next exit?"

[0769] Emotion analysis: Detect anxiety from the tone of voice.

[0770] AI response: Instruct: "The next exit is 3 kilometers away. Please remain calm and keep driving."

[0771] Example prompt sentence:

[0772] Consider the user's nervousness and create a guiding message like this: 'Where is the next exit?' Answer in a calm tone.

[0773] Thus, the embodiment of the present invention includes a series of processes that collects audio and video data, analyzes it to understand the user's intentions and emotions, and provides appropriate information. In particular, it enables quick and accurate instructions in emergencies while driving, thereby improving the safety and comfort of the user.

[0774] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0775] Step 1:

[0776] The user device collects audio and video data of the conversation. Specifically, the microphone and camera built into smart glasses or smartphones record and record the user's speech and facial expressions in real time. The input is the user's voice and facial expressions, and the output is the collected audio and video data.

[0777] Step 2:

[0778] The terminal transmits collected audio and video data to the server in real time. Specifically, after noise filtering and preprocessing, the data is transmitted to the server via the network. The input is the collected audio and video data, and the output is the transmission of the preprocessed data to the server.

[0779] Step 3:

[0780] The server converts the received voice data into text data using the speech_recognition library. Specifically, it analyzes the voice signal using a speech recognition algorithm and converts the results into text format. The input is preprocessed voice data, and the output is text data.

[0781] Step 4:

[0782] The server analyzes the received video data using the emotion_recognition library to identify the user's emotions. Specifically, it analyzes facial expressions and vocal tone to identify the speaker's emotional state. The input is preprocessed video data, and the output is identified emotional information.

[0783] Step 5:

[0784] The server uses data obtained from speech recognition and emotion analysis to understand the speaker's intent and summarize the information. Specifically, it uses natural language processing algorithms to understand the gist of what is being said and create a concise summary. The input is text data and emotion information, and the output is summarized information.

[0785] Step 6:

[0786] The server uses the generative AI model to generate information that will be effective next time. Specifically, it generates an appropriate response based on the summarized information and the speaker's emotional state. The input is the summarized information and emotional information, and the output is the generated response.

[0787] Step 7:

[0788] The device presents the generated response. Specifically, the response is displayed on the smartglasses display or smartphone screen, and audio guidance is also provided if necessary. The input is the generated response, and the output is the information presented to the user.

[0789] This series of processes allows users to receive accurate information and instructions in real time. In particular, in emergencies, safety can be improved by providing prompt and appropriate instructions according to the user's emotional state.

[0790] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0791] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0792] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0793] [Third embodiment]

[0794] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0795] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0796] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0797] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0798] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0799] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0800] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0801] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0802] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0803] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0804] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0805] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0806] This invention relates to an "AI Conversation Advisor" that utilizes generative AI to provide conversation support and advice on what to say. This system collects and analyzes voice data in real time and provides appropriate responses to users, thereby supporting smooth communication.

[0807] System Overview

[0808] 1. User device: The user uses a smart glasses-type device or a smartphone. These devices are equipped with a microphone to collect conversations between the user and others in real time, and a display or audio output to show the results.

[0809] 2. Server: The collected voice data is sent to the server via the network, where it is analyzed using high-performance voice recognition and natural language processing technologies.

[0810] Program processing

[0811] Collection and transmission of voice data

[0812] First, when a user starts a conversation, the smart glasses or smartphone collects audio data. For example, when collecting audio during a meeting, collection begins automatically as soon as the meeting begins. The collected audio data undergoes pre-processing such as noise filtering to make it clear audio data. This audio data is then sent to a server in real time.

[0813] Analyzing voice data and understanding intent

[0814] The server uses speech recognition technology to convert the received voice data into text. The text data is then subjected to syntactic analysis, which identifies the subject, predicate, and object of the sentence and associates them with the context. Contextual and sentiment analysis are then performed to understand the intention and emotion behind the other person's remarks. For example, if the question includes "How is the project progressing?", it is interpreted as "You are asking about the progress of the project."

[0815] Summarization and response generation

[0816] The intent that is captured is summarized on the server and put into a form that is easy for the user to understand. For example, it may be summarized as "Your boss is asking about the progress of the project." The generative AI also generates multiple effective responses to utter next. Response candidates are customized based on the user's past speech patterns and set preferences.

[0817] Presentation of results

[0818] The summarized information and possible responses are sent to the device and displayed on the smartglasses display or smartphone screen. Voice guidance is also provided as needed. For example, the message might read, "Your boss is asking about the progress of a project. What do you think of the following response?"

[0819] Specific examples

[0820] Business Meeting Scenario

[0821] 1. User: Puts on smart glasses as soon as a business meeting begins.

[0822] 2. Terminal: Collects voice data, filters it, and then sends it to the server.

[0823] 3. Server: Converts the speech data into text and performs syntax and sentiment analysis. Example: "Your boss is asking about the progress of the project."

[0824] 4. Server: Generates candidate responses and presents options such as:

[0825] "The project is progressing as scheduled and is currently 70% complete."

[0826] "There are delays to some tasks, but we expect them to be completed by next week."

[0827] "Would you like a more detailed progress report?"

[0828] 5. Device: The generated summary and response suggestions are displayed on the smart glasses display. For example, it shows, "Your boss is asking about the progress of the project. What do you think of the following response?"

[0829] 6. User: Select the appropriate response from the suggested options and execute it in the conversation.

[0830] In this way, the "AI Conversation Advisor" will be able to assist users in their conversations in real time, allowing for smoother communication.

[0831] The processing flow will be explained below.

[0832] Step 1:

[0833] User: Turn on the smart glasses or smartphone and start collecting voice data of the conversation. For example, the moment the user starts speaking in a business meeting, voice data collection starts automatically.

[0834] Step 2:

[0835] Terminal: Noise filtering is performed on the collected voice data to remove background noise, etc. This allows for clear voice data to be obtained.

[0836] Step 3:

[0837] Terminal: Sends noise-filtered audio data to the server in real time. The data is transferred using a secure communication protocol.

[0838] Step 4:

[0839] Server: The received voice data is converted into text data using voice recognition technology. A highly accurate voice recognition engine is used in this process.

[0840] Step 5:

[0841] Server: Performs syntactic analysis on the text data to identify the components of a sentence, such as the subject, predicate, and object.

[0842] Step 6:

[0843] Server: Further contextual and sentiment analysis is performed to understand the intent and sentiment of the statement. For example, if the question "How is the project progressing?" is included, it will be interpreted as "Ask about the progress of the project."

[0844] Step 7:

[0845] Server: Summarizes the intent and generates information to present to the user in an easy-to-understand format. For example, it might summarize the intent as "The boss is asking about the progress of the project."

[0846] Step 8:

[0847] Server: Based on the generated summary, the server generates multiple effective responses for the next utterance. The responses are customized based on the user's past utterance patterns and preferences.

[0848] Step 9:

[0849] Server: Formats the generated summary and response candidates and sends them to the user's terminal.

[0850] Step 10:

[0851] Device: The summary and response candidates are displayed on the smartglasses display or smartphone screen. Voice guidance is also possible. For example, the message may say, "Your boss is asking about the progress of the project. What do you think of the following response?"

[0852] Step 11:

[0853] User: Choose the appropriate response from the suggested responses and use it in the actual conversation.

[0854] Through this series of steps, the "AI Conversation Advisor" supports users' conversations in real time, avoiding misunderstandings and miscommunication and ensuring smooth communication.

[0855] Example 1

[0856] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0857] During a conversation, it can be difficult for users to instantly grasp and respond appropriately. This is especially true in business meetings and important discussions, where it is necessary to respond appropriately on the spot. However, conventional technology lacks the means to analyze the content of a conversation in real time and provide appropriate responses. To solve this problem, a system that allows users to respond smoothly is needed.

[0858] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0859] In this invention, the server includes means for using a device to collect voice data, means for noise filtering the collected voice data in real time and transmitting the data to the server, means for converting the transmitted voice data into text using voice recognition technology, means for syntactically analyzing and sentiment analyzing the converted text data, means for extracting the intention of the utterance from the analyzed data, means for summarizing the intention of the utterance, means for generating information on effective next utterances using a generative AI model, and means for presenting the generated information on a user device, thereby enabling the user to grasp appropriate responses in real time and to communicate smoothly and effectively.

[0860] "Audio data" refers to audio signals of conversations or utterances made by a user.

[0861] "Device" refers to a hardware device for collecting audio data, including, for example, smart glasses and smartphones.

[0862] A "server" refers to a computer system that is connected via a network and processes and analyzes data.

[0863] "Noise filtering" refers to the process of removing unwanted background sounds and noise from collected audio data.

[0864] "Speech recognition technology" refers to technology for analyzing collected voice data and converting voice into text.

[0865] "Syntax analysis" refers to the process of analyzing the grammatical structure of text data and clarifying the relationships between the subject, predicate, object, and other parts of a sentence.

[0866] "Sentiment analysis" refers to the process of inferring the emotional state of a speaker from text data.

[0867] "Intention of speech" refers to understanding what the speaker intended based on the analyzed data.

[0868] "Summarization" means summarizing analyzed information into a concise and easy-to-understand format.

[0869] "Generative AI model" refers to an artificial intelligence model that performs natural language processing and is used to generate new text and responses.

[0870] The term "user device" refers to a device used by a user for displaying information and executing voice guidance.

[0871] This invention relates to an "AI Conversation Advisor" that utilizes generative AI to provide conversation support and advice on what to say. This system collects and analyzes voice data in real time and presents appropriate responses to users to support smooth communication.

[0872] System Overview

[0873] 1. User Devices

[0874] Users use smart glasses or smartphones, which are equipped with a microphone to collect conversations between the user and others in real time and a display or audio output to display the analysis results.

[0875] 2. Server

[0876] The collected voice data is sent to a server via a network. The server then uses high-performance voice recognition and natural language processing technologies to analyze the data. Specifically, it converts the voice data into text using voice recognition technology (e.g., Google Speech-to-Text API) and generates an optimal response using a generative AI model (e.g., OpenAI's GPT-4).

[0877] Program processing

[0878] First, when a user starts a conversation, the smart glasses or smartphone collects audio data. For example, when collecting audio during a meeting, collection begins automatically as soon as the meeting begins. The collected audio data undergoes pre-processing such as noise filtering on the device to create clear audio data. This audio data is then sent to the server in real time.

[0879] The server converts the received voice data into text using speech recognition technology. It then uses a generative AI model to perform syntax analysis and sentiment analysis on the text data to understand the intent and emotion of the other person's speech. For example, if the question "How is the project progressing?" is included, it can be interpreted as "A question about the progress of the project."

[0880] The intent that is captured is summarized on the server and put into a form that is easy for the user to understand. For example, it might summarize it as "your boss is asking about the progress of the project." It also uses a generative AI model to generate multiple effective next utterances. Response candidates are customized based on the user's past utterance patterns and set preferences.

[0881] The summarized information and possible responses are sent to the device and displayed on the smartglasses display or smartphone screen. Voice guidance is also provided as needed. For example, the message might read, "Your boss is asking about the progress of a project. What do you think of the following response?"

[0882] Specific examples

[0883] Business Meeting Scenario

[0884] 1. User: Puts on smart glasses as soon as a business meeting begins.

[0885] 2. Terminal: Collects voice data, filters it, and then sends it to the server.

[0886] 3. Server: Converts the speech data into text and performs syntax and sentiment analysis. Example: "Your boss is asking about the progress of the project."

[0887] 4. Server: Generates candidate responses and presents options such as:

[0888] "The project is progressing as scheduled and is currently 70% complete."

[0889] "There are delays to some tasks, but we expect them to be completed by next week."

[0890] "Would you like a more detailed progress report?"

[0891] 5. Device: The generated summary and response suggestions are displayed on the smart glasses display. For example, it shows, "Your boss is asking about the progress of the project. What do you think of the following response?"

[0892] 6. User: Select the appropriate response from the suggested options and execute it in the conversation.

[0893] Prompt Sentence Examples

[0894] Below are some example prompts to input to a generative AI model (e.g., GPT-4):

[0895] "If your boss is asking about the progress of a project, generate an appropriate response."

[0896] This will cause the AI ​​to generate a response like this:

[0897] 1. "The project is on schedule and is currently 70% complete."

[0898] 2. "We're running behind on some tasks, but we expect to complete them next week."

[0899] 3. "Would you like more detailed progress reports?"

[0900] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0901] Step 1: Collect and preprocess audio data

[0902] The user puts on the smart glasses or smartphone and launches the application.

[0903] Input: Real-time conversational audio produced by the user.

[0904] The device uses the device's microphone to capture audio data in real time. For example, if you're in a meeting, audio collection begins as soon as it starts.

[0905] The device performs noise filtering on the acquired voice data to remove background noise (for example, noise reduction processing using FFT).

[0906] Output: Clear audio data after filtering.

[0907] Step 2: Sending audio data

[0908] The device transmits the pre-processed voice data to the server in real time.

[0909] Input: Clear audio data after filtering.

[0910] Terminals may compress data to improve communication efficiency.

[0911] Output: The compressed audio data is sent over the network to the server.

[0912] Step 3: Convert audio data to text

[0913] The server converts the received voice data into text data using voice recognition technology (e.g., Google Speech-to-Text API).

[0914] Input: Compressed and transmitted audio data.

[0915] The server extracts linguistic features from the audio and converts it into text format.

[0916] Output: The converted text data.

[0917] Step 4: Parsing and sentiment analysis of text data

[0918] The server performs syntactic analysis on the generated text data, thereby identifying the subject, predicate, and object of the sentence.

[0919] Input: The converted text data.

[0920] The server then performs a sentiment analysis to assess the emotional state of the speaker.

[0921] Output: Parsed data and sentiment analysis results.

[0922] Step 5: Understand and summarize the intention of the statement

[0923] The server uses the analyzed data to understand the intent behind the statement.

[0924] Input: Syntactic and sentiment analysis results.

[0925] The server summarizes the intent of the conversation and condenses it into a form that is easy for the user to understand, such as "your boss is asking about the progress of a project."

[0926] Output: Summarized intent data.

[0927] Step 6: Generate candidate responses

[0928] The server uses a generative AI model (e.g., OpenAI's GPT-4) to generate multiple effective next utterances.

[0929] Input: Summarized intent data.

[0930] The server feeds example prompts to the model, which generates appropriate response patterns, taking into account the user's past speech patterns and preferences.

[0931] Output: The generated list of possible responses.

[0932] Step 7: Presenting results and user response

[0933] The server transmits the summarized information and candidate responses to the user terminal.

[0934] Input: The generated response candidate list.

[0935] The device will then display this information on the smartglasses display or smartphone screen, for example, in the form of, "Your boss is asking you about the progress of a project. What do you think of the following response?"

[0936] The user selects the appropriate response from the presented options and executes it in the actual conversation. Selection can be done by voice input or touch operation.

[0937] Output: The response selected by the user.

[0938] (Application example 1)

[0939] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0940] There is a need to improve the quality of customer interactions in brick-and-mortar stores and simultaneously increase sales staff efficiency and customer satisfaction. In particular, there is a lack of support tools to enable appropriate and prompt responses during customer service, and technology to ensure smooth communication with customers is needed. There is also a need for a system that can provide consistent, high-quality customer service that is not dependent on the experience or skills of staff.

[0941] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0942] In this invention, the server includes a preprocessing means for performing noise filtering on voice data collected using a smart device, a means for converting the voice data sent to the server into text and performing syntax analysis and sentiment analysis, and a means for displaying the generated response candidates and summaries on the display of the smart device. This makes it possible to support conversations with customers in real time using the smart device, allowing sales staff to respond to customers more quickly and accurately.

[0943] "Conversational voice data" refers to data in which the voice of a user or customer is recorded in digital format.

[0944] "Means for transmitting in real time" refers to technology for transmitting collected voice data to a server immediately without delay.

[0945] "Means for analyzing" refers to the technology or system that converts transmitted voice data into text data and understands grammar and context.

[0946] "Means for understanding the intention of a statement" refers to technology for extracting the speaker's intentions and emotions from analyzed text data.

[0947] "Means of summarizing" refers to the technique of summarizing the intent and information that has been extracted into a concise, to-the-point format.

[0948] The "means for generating effective information" is a technology for automatically generating information that is determined to be effective when spoken next.

[0949] "Means for presenting" refers to a technique for visually or audibly notifying the user of the generated information.

[0950] "Smart device" refers to an electronic device that can be carried or worn by a user and has the function of collecting and displaying voice data.

[0951] "Noise filtering" is a technique for removing background noise from audio data to obtain clear audio data.

[0952] "Syntax analysis" is a technique for analyzing the grammatical structure of text data and identifying elements such as subjects, predicates, and objects.

[0953] "Sentiment analysis" is a technology that identifies emotions from a speaker's text data and classifies the type of emotion.

[0954] "Response candidates" refer to multiple appropriate responses to be made next.

[0955] A "summary" is a concise summary of the main points and important information of a conversation.

[0956] "Means for displaying on a display" refers to technology for displaying the generated information on the screen of a smart device.

[0957] System configuration

[0958] A system for implementing the present invention includes the following hardware and software.

[0959] Smart device: A device carried or worn by the user that has the ability to collect voice data and display the results. Examples include smart glasses and smartphones.

[0960] Server: Has high-performance computing resources to perform voice data analysis, syntax analysis, and sentiment analysis. It mainly uses speech recognition APIs (e.g., Google Speech-to-Text API) and natural language processing APIs (e.g., OpenAI GPT-3).

[0961] Program Description

[0962] Collection and transmission of voice data

[0963] Smart devices (smart glasses or smartphones) collect the user's voice data in real time using a microphone built into the device. The collected voice data is filtered to make it clear and then sent to a server.

[0964] Analyzing voice data and understanding intent

[0965] The server converts the transmitted voice data into text using a speech recognition API (Google Speech-to-Text API). A natural language processing API (OpenAI GPT-3) is used to perform syntactic analysis and sentiment analysis on the converted text data. Syntactic analysis identifies the subject, predicate, and object, while sentiment analysis extracts the speaker's intention and emotion from the context.

[0966] Summarization and response generation

[0967] Based on the intent, the server summarizes the key points. The AI ​​then generates multiple effective responses for the next utterance. These responses are customized based on the user's past speech patterns and preferences.

[0968] Presentation of results

[0969] The summary and candidate responses generated by the server are displayed on the smart device's display. Voice guidance is also provided as needed. For example, a message such as, "A customer has a question about a new smartphone. What do you think of the following response?" can be displayed.

[0970] Specific examples

[0971] Scene: Customer service in a brick-and-mortar store

[0972] 1. A user puts on the smart glasses and starts a conversation with a customer.

[0973] 2. The smart device collects the voice data and sends it to the server along with noise filtering.

[0974] 3. The server converts the voice data into text and performs syntax and sentiment analysis. Example: "A customer is asking about the features of their smartphone."

[0975] 4. The server generates a candidate response, for example:

[0976] "This smartphone has an excellent camera."

[0977] "Long battery life and convenient"

[0978] Want to know more about the features?

[0979] 5. The generated summary and response suggestions are displayed on the smart glasses.

[0980] Prompt Sentence Examples

[0981] A customer has asked a question about a new smartphone. Generate possible responses.

[0982] In this way, the system can assist sales staff in serving customers in real time and improve customer satisfaction.

[0983] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0984] Step 1:

[0985] Smart devices (smart glasses or smartphones) collect voice data of conversations between users and customers in real time. The input is the voice data of the conversation, and the output is the collected voice data. The voice data is acquired using a microphone, and clear voice data is prepared by noise filtering. This voice data is then passed to the next step.

[0986] Step 2:

[0987] The smart device sends noise-filtered audio data to the server. The input is clear audio data, and the output is the audio data sent to the server. Here, the data is sent in real time using the device's communication module.

[0988] Step 3:

[0989] The server converts the transmitted voice data into text data. The input is voice data and the output is text data. The server uses a speech recognition API (Google Speech-to-Text API) to analyze the voice waveform data and generate a corresponding string.

[0990] Step 4:

[0991] The server analyzes the converted text data and identifies grammatical structures. The input is the text data, and the output is the results of the analysis. This process uses a natural language processing API (OpenAI GPT-3) to identify sentence elements such as subject, predicate, and object.

[0992] Step 5:

[0993] The server performs sentiment analysis based on the parsing results. The input is the parsing results, and the output is the intention of the utterance and the sentiment analysis results. The sentiment analysis module is used to determine the speaker's emotional state and intention. This information is used for further processing.

[0994] Step 6:

[0995] The server summarizes the main points of the speech based on the results of sentiment analysis. The input is the intention of the speech and the results of sentiment analysis, and the output is a summary. A summarization algorithm is used to extract the main points of the text and create a concise summary.

[0996] Step 7:

[0997] The server generates effective response candidates based on the summary. The input is the summary, and the output is multiple response candidates. Using a generative AI model (OpenAI GPT-3), multiple next-best utterances are generated.

[0998] Step 8:

[0999] The smart device receives the summary and candidate responses sent from the server and displays them on the display. The input is the summary and candidate responses, and the output is information provided to the user's visual sense. The display function of the device is used to visually present appropriate candidate responses to the user.

[1000] Step 9:

[1001] The user checks the response candidates presented by the smart device, selects the most appropriate one, and speaks to the customer. The input is the displayed response candidate, and the output is the response selected by the user. This enables smooth and effective customer service.

[1002] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1003] This invention relates to an "AI Conversation Advisor" that combines generative AI and an emotion engine to provide conversation support and advice on what to say. This system collects and analyzes voice data in real time, recognizes emotions, and then presents appropriate responses to the user, supporting smooth communication.

[1004] System Overview

[1005] 1. User device: The user uses a smart glass or smartphone, which is equipped with a microphone to collect real-time conversations between the user and others, a camera to capture the user's speech and facial expressions, and a display or audio output to show the results.

[1006] 2. Server: The collected audio and video data is sent to the server via the network. The server uses high-performance speech recognition technology, natural language processing technology, and an emotion engine to analyze the data.

[1007] Program processing

[1008] Collection and transmission of audio and video data

[1009] First, when a user starts a conversation, the smart glasses or smartphone collects audio and video data. For example, if collecting audio and facial expressions during a meeting, collection begins automatically as soon as the meeting begins. The collected data undergoes pre-processing such as noise filtering and video cleaning. This data is then sent to a server in real time.

[1010] Data analysis and understanding of intentions and emotions

[1011] The server converts the received voice data into text data using speech recognition technology. The text data is then subjected to syntactic analysis, which identifies the subject, predicate, and object of the sentence and associates them with the context. At the same time, the video data is analyzed by an emotion engine, which recognizes emotions from the user's facial expressions and tone of voice. For example, the question "How is the project progressing?" is analyzed as "You are asking about the progress of the project," and it is recognized that the user is feeling nervous.

[1012] Summarization and response generation

[1013] The intent and emotions that have been captured are summarized on the server, generating information to present to the user in an easy-to-understand format. For example, it may summarize the message as "Your boss is asking about the progress of the project." The generation AI also generates multiple effective responses to utter next. Response candidates are customized based on the user's past speech patterns and preferences. Additionally, the response content is adjusted based on the user's emotions as recognized by the emotion engine.

[1014] Presentation of results

[1015] The summarized information and possible responses are sent to the device and displayed on the smartglasses display or smartphone screen. Voice guidance is also provided as needed. For example, the message might read, "Your boss is asking about the progress of a project. What do you think of the following response?"

[1016] Specific examples

[1017] Business Meeting Scenario

[1018] 1. User: Puts on smart glasses as soon as a business meeting begins.

[1019] 2. Terminal: Collects audio and video data, performs noise filtering and video preprocessing, and then sends it to the server.

[1020] 3. Server: Converts the voice data into text and performs syntax analysis and sentiment analysis. For example, it analyzes "My boss is asking about the progress of the project" and recognizes that the user is nervous.

[1021] 4. Server: Generates response candidates and presents options such as:

[1022] "The project is on schedule and is currently 70% complete." (In a calm tone to ease tension)

[1023] "We're experiencing delays with some tasks, but we expect them to be completed by next week." (Providing reassurance with specific numbers)

[1024] "Would you like more detailed progress reports?" (Supportive attitude)

[1025] 5. Device: The generated summary and response suggestions are displayed on the smart glasses display. For example, it may say, "Your boss is asking about the progress of the project. What do you think of the following response?"

[1026] 6. User: Select the appropriate response from the presented options and execute it in the conversation.

[1027] In this way, the "AI Conversation Advisor" can assist users in conversations in real time, avoiding misunderstandings and miscommunication while providing appropriate responses that take emotions into account.

[1028] The processing flow will be explained below.

[1029] Step 1:

[1030] User: Turns on the smart glasses device or smartphone and collects audio and video data of the conversation. For example, the moment a user starts speaking in a business meeting, audio and video data collection begins automatically.

[1031] Step 2:

[1032] Terminal: Noise filtering is performed on the collected audio data to remove unwanted noise, and video cleaning is performed on the video data to generate clearer images.

[1033] Step 3:

[1034] Terminal: After noise filtering and image cleaning, the data is sent to the server in real time via a secure communication protocol.

[1035] Step 4:

[1036] Server: Converts the received voice data into text data using voice recognition technology. Stores the converted text data.

[1037] Step 5:

[1038] Server: Analyzes the received video data and combines facial expression recognition algorithms and tone analysis of audio data to identify the user's emotions. For example, it recognizes whether the user is nervous or angry.

[1039] Step 6:

[1040] Server: Performs syntactic analysis on text data to extract components such as the subject, predicate, and object of a sentence. It also performs contextual analysis to understand the intention and topic of what is being said.

[1041] Step 7:

[1042] Server: Based on the results of syntactic and contextual analysis, summarizes what the other person is saying. For example, it summarizes it as "your boss is asking about the progress of the project."

[1043] Step 8:

[1044] Server: Based on the summarized information and the user's emotions recognized by the emotion engine, the server generates effective response candidates for the next utterance. For example, if the user is nervous, the server generates a response in a calm tone.

[1045] Step 9:

[1046] Server: Formats the generated summary and candidate responses and sends them to the user terminal. Data is transmitted end-to-end encrypted.

[1047] Step 10:

[1048] Device: A formatted summary and suggested responses are displayed on the smartglasses display or smartphone screen. Voice guidance is also possible if necessary. For example, the display might say, "Your boss is asking about the progress of the project. What do you think of the following response?"

[1049] Step 11:

[1050] User: Choose the appropriate response from the suggested responses and use it in the actual conversation.

[1051] Through this series of steps, the AI ​​conversation advisor can assist users in real-time conversations, avoiding misunderstandings and miscommunications, and providing appropriate responses that take emotions into account.

[1052] Example 2

[1053] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1054] Conventional conversation support systems were unable to fully grasp the intentions and emotions of users' remarks, resulting in misunderstandings and miscommunication. Furthermore, due to a lack of technology to recognize emotions from video data in addition to analyzing audio data, it was difficult to provide accurate responses in real time. This meant that users had to perform many manual operations to obtain the information they needed, making it difficult to support smooth communication.

[1055] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting audio data and video data of a conversation, means for transmitting the collected audio data and video data in real time, means for converting the transmitted audio data into text data, means for analyzing the syntax of the text data and recognizing the user's emotions, means for summarizing the analyzed data and information obtained from the emotion recognition, means for generating information that would be effective to utter next based on the summarized information, and means for presenting the generated information to the user. This makes it possible to accurately analyze the user's intentions and emotions based on the collected data and generate and present an appropriate response in real time.

[1056] "Means for collecting audio data and video data of a conversation" is a general term for devices equipped with a microphone and a camera for capturing the user's voice and video in real time.

[1057] The "means for transmitting the collected audio data and video data in real time" refers to a communication function for instantly transmitting the collected data to a server.

[1058] The "means for converting the transmitted voice data into text data" refers to a method for converting voice data into text format using voice recognition technology.

[1059] The "means for analyzing the syntax of the text data and recognizing the user's emotions" refers to a function for analyzing the grammatical structure of the text data using natural language processing technology and recognizing the user's emotions.

[1060] The "means for summarizing the analyzed data and information obtained from emotion recognition" refers to a function for summarizing information in order to display the analysis results in a concise manner.

[1061] "Means for generating information that would be effective to utter next based on the summarized information" refers to a technology that generates a response that would be effective for the user to utter next based on summarized information.

[1062] "Means for presenting the generated information to the user" refers to an interface including a display and a speaker for visually or audibly presenting the generated response candidates to the user.

[1063] MODE FOR CARRYING OUT THE INVENTION

[1064] This invention relates to a system that supports user conversations in real time by combining audio data and video data. The system consists of a terminal worn by the user and a server that processes the data.

[1065] Device configuration

[1066] Users use devices such as smart glasses and smartphones. These devices include the following hardware:

[1067] 1. Microphone: To collect audio data of conversation.

[1068] 2. Camera: To capture the user's facial expressions and collect video data.

[1069] 3. Display: For showing summarized information and possible responses.

[1070] 4. Communication module: To transmit collected data to the server in real time.

[1071] Server Configuration

[1072] The server includes the following software modules and functions:

[1073] 1. Speech recognition technology: Used to convert voice data into text data. Specifically, speech recognition engines include Google Cloud Speech-to-Text API and IBM Watson Speech to Text.

[1074] 2. Natural language processing technology: Used to analyze the syntax of text data and understand the user's intent. For example, libraries such as Spacy and NLTK are used.

[1075] 3. Emotion Engine: Recognizes user emotions from video data and analyzes voice tone.

[1076] 4. Generative AI models: Used to generate effective next utterances. For example, OpenAI's GPT-4.

[1077] 5. Summarization module: Summarizes the acquired information and presents it to the user.

[1078] Specific examples

[1079] Business Meeting Scenario

[1080] 1. User: The user puts on the smart glasses as soon as the business meeting begins.

[1081] 2. Terminal: Collects audio and video data, performs noise filtering and video preprocessing, and then sends it to the server.

[1082] 3. Server: Converts the voice data into text and performs syntax analysis and sentiment analysis. For example, it analyzes the speech data as "your boss is asking about the progress of the project" and recognizes that the user is nervous.

[1083] 4. Server: Generates response candidates and presents options such as:

[1084] "The project is on schedule and is currently 70% complete." (In a calm tone to ease tension)

[1085] "We're experiencing delays with some tasks, but we expect them to be completed by next week." (Providing reassurance with specific numbers)

[1086] "Would you like more detailed progress reports?" (Supportive attitude)

[1087] 5. Terminal: The generated summary and response suggestions are displayed on the screen. For example, it may be displayed in the form of "Your boss is asking about the progress of the project. What do you think of the following response?"

[1088] 6. User: Select the appropriate response from the presented options and execute it in the conversation.

[1089] Prompt Sentence Examples

[1090] "Please advise me on how to respond in a tense situation when someone asks me how the project is going."

[1091] "Generate phrases to speak in a calm tone when reporting an ongoing project to your boss."

[1092] The above is a specific embodiment for carrying out the present invention. This system effectively combines audio and video data to support user conversations in real time, thereby facilitating user communication and preventing misunderstandings and miscommunication.

[1093] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1094] Step 1: User initiates conversation

[1095] A user puts on smart glasses or a smartphone and starts a conversation. Specifically, the device automatically detects the start of a conversation, or the user manually instructs the device to start a conversation. The input is the user's voice and video, and the output is the device collecting these data.

[1096] Step 2: Collect and preprocess audio and video data

[1097] The device collects audio and video data of the conversation. It uses a microphone to pick up the audio and a camera to capture the user's face. The input is the collected audio and video data, and the output is preprocessed data. Specific preprocessing steps include reducing background noise from the audio data and adjusting the brightness and contrast of the video data.

[1098] Step 3: Send data from the device to the server

[1099] The terminal transmits the preprocessed data to the server in real time. The input is the preprocessed audio and video data, and the output is the data transmitted to the server. The data is converted into a packet format and transmitted using a secure communication protocol.

[1100] Step 4: Convert audio data to text

[1101] The server converts the received voice data into text data using voice recognition technology. The input is voice data and the output is text data. Specifically, it uses a voice recognition engine (e.g., Google Cloud Speech-to-Text API) to convert voice into text in real time.

[1102] Step 5: Parsing and Emotion Recognition

[1103] The server performs syntactic analysis of the text data and simultaneously performs emotion recognition using video data. The input is text data and video data, and the output is an analysis result that includes intent and emotion. A natural language processing library (e.g., Spacy) is used to identify the subject, predicate, and object of the sentence. At the same time, the emotion engine analyzes facial expressions and voice tone to recognize the user's emotions.

[1104] Step 6: Summarize and generate a response

[1105] The server summarizes the analysis results and generates effective information for the next user utterance. The input is the analysis results, and the output is summarized information and response candidates. A generative AI model (e.g., GPT-4) is used to generate multiple next utterances. Responses are customized based on past utterance patterns and the user's preferences.

[1106] Step 7: Summarize and present response candidates to the device

[1107] The server sends the generated summary and candidate responses to the terminal. The input is the summarized information and candidate responses, and the output is the information displayed on the terminal. The terminal receives this and displays on its display, "Your boss is asking about the progress of the project. What do you think of the following response?"

[1108] Step 8: User selects and executes the best response

[1109] The user selects an appropriate response from the candidate responses displayed on the device and executes it in the conversation. The input is the displayed candidate response, and the output is the confirmed response selected by the user. The user selects a response using the touch interface or voice commands and executes the response in the actual conversation.

[1110] (Application example 2)

[1111] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1112] In autonomous vehicles, communication with passengers and drivers can be difficult, and a major issue is the lack of appropriate instructions and information provided in emergencies. Furthermore, there is a need for a flexible system that can respond to various situations that arise while driving.

[1113] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for transmitting audio data and video data of a conversation in real time, means for analyzing the transmitted audio data and video data, and means for extracting the intention and emotion of the utterance from the analyzed data. This makes it possible to identify the intention and emotion of the speaker through the analysis of the audio and video data in real time and provide appropriate responses and instructions.

[1114] "Conversational voice data" refers to voice signals uttered by a user, and includes linguistic information and emotional information.

[1115] "Video data" refers to video signals collected to capture a user's facial expressions and movements, and includes visual information.

[1116] "Means for transmitting in real time" refers to a function for transmitting collected audio and video data to a server immediately without delay.

[1117] "Means for analysis" refers to software or algorithms used to analyze transmitted audio and video data, thereby enabling understanding of intent and emotion.

[1118] "Means of understanding intent" refers to the function of understanding the speaker's purpose and intention from the analyzed data.

[1119] "Means of capturing emotions" refers to the function of identifying the speaker's emotional state from the analyzed data.

[1120] "Means of summarization" refers to the function of summarizing analyzed information in a concise and easy-to-understand form.

[1121] "Means for generating information that will be effective to utter next" refers to the function for generating appropriate content to be uttered next based on the intention and emotion that has been grasped.

[1122] "Presentation means" refers to a device or function for visually or audibly presenting the generated information to the user.

[1123] "Means for providing appropriate instructions in an emergency" refers to a function that provides users with prompt and accurate instructions based on the results of data analysis in an emergency situation.

[1124] "Noise filtering" refers to the process of removing unwanted background noise from collected audio data.

[1125] "Image processing" refers to the process of converting collected video data into a form that is easier to clean and analyze.

[1126] "Normalizing the volume" refers to the process of adjusting the volume level of audio data to keep it constant.

[1127] System program generation and hardware / software configuration

[1128] System Configuration

[1129] This system includes multiple devices and servers for collecting and analyzing audio and video data of conversations in real time. Details of each element are given below.

[1130] 1. User Device:

[1131] Smart glasses: Worn by passengers or drivers, these glasses contain microphones and cameras for real-time audio and video capture.

[1132] Smartphones: Can be used as auxiliary devices to collect audio and video data.

[1133] 2. Server:

[1134] Speech recognition system: Uses the speech_recognition library to convert voice data into text.

[1135] Emotion analysis system: Uses the emotion_recognition library to identify emotions from collected audio and video data.

[1136] Natural language processing engine: Uses generative AI models to generate responses based on the speaker's intent and sentiment.

[1137] Data Processing

[1138] 1. Audio and video data collection and transmission:

[1139] Smart glasses or smartphones collect real-time audio and video data of conversations, which undergo local noise filtering and image processing before being sent to a server.

[1140] 2. Audio and video data analysis:

[1141] The server converts the received audio data into text data using the speech_recognition library, then performs syntactic analysis on the text data and combines it with context.

[1142] Next, the collected video data is analyzed using the emotion_recognition library to identify the speaker's emotions.

[1143] 3. Understanding intentions and emotions and summarizing information:

[1144] The system identifies the speaker's intention and emotion from the analyzed data and summarizes this information. For example, if the question "Where is the next exit?" is analyzed and the speaker is found to be impatient, it can be summarized as "The speaker is in a hurry to find out where the next exit is."

[1145] 4. Response generation and presentation:

[1146] Based on the summarized information, a generative AI model is used to generate appropriate responses, which are customized based on the user's past speech patterns and set preferences.

[1147] The generated response will be displayed on the smartglasses display or smartphone screen, and will also be provided as an audio guide if necessary.

[1148] Examples and prompts

[1149] Example 1: Emergency response on expressways

[1150] Scenario: Providing emergency information when a driver brakes suddenly.

[1151] Voice input: A passenger asks, "Where is the next exit?"

[1152] Emotion analysis: Detect anxiety from the tone of voice.

[1153] AI response: Instruct: "The next exit is 3 kilometers away. Please remain calm and keep driving."

[1154] Example prompt sentence:

[1155] Consider the user's nervousness and create a guiding message like this: 'Where is the next exit?' Answer in a calm tone.

[1156] Thus, the embodiment of the present invention includes a series of processes that collects audio and video data, analyzes it to understand the user's intentions and emotions, and provides appropriate information. In particular, it enables quick and accurate instructions in emergencies while driving, thereby improving the safety and comfort of the user.

[1157] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1158] Step 1:

[1159] The user device collects audio and video data of the conversation. Specifically, the microphone and camera built into smart glasses or smartphones record and record the user's speech and facial expressions in real time. The input is the user's voice and facial expressions, and the output is the collected audio and video data.

[1160] Step 2:

[1161] The terminal transmits collected audio and video data to the server in real time. Specifically, after noise filtering and preprocessing, the data is transmitted to the server via the network. The input is the collected audio and video data, and the output is the transmission of the preprocessed data to the server.

[1162] Step 3:

[1163] The server converts the received voice data into text data using the speech_recognition library. Specifically, it analyzes the voice signal using a speech recognition algorithm and converts the results into text format. The input is preprocessed voice data, and the output is text data.

[1164] Step 4:

[1165] The server analyzes the received video data using the emotion_recognition library to identify the user's emotions. Specifically, it analyzes facial expressions and vocal tone to identify the speaker's emotional state. The input is preprocessed video data, and the output is identified emotional information.

[1166] Step 5:

[1167] The server uses data obtained from speech recognition and emotion analysis to understand the speaker's intent and summarize the information. Specifically, it uses natural language processing algorithms to understand the gist of what is being said and create a concise summary. The input is text data and emotion information, and the output is summarized information.

[1168] Step 6:

[1169] The server uses the generative AI model to generate information that will be effective next time. Specifically, it generates an appropriate response based on the summarized information and the speaker's emotional state. The input is the summarized information and emotional information, and the output is the generated response.

[1170] Step 7:

[1171] The device presents the generated response. Specifically, the response is displayed on the smartglasses display or smartphone screen, and audio guidance is also provided if necessary. The input is the generated response, and the output is the information presented to the user.

[1172] This series of processes allows users to receive accurate information and instructions in real time. In particular, in emergencies, safety can be improved by providing prompt and appropriate instructions according to the user's emotional state.

[1173] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1174] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1175] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1176] [Fourth embodiment]

[1177] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1178] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1179] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1180] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1181] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1182] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1183] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1184] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1185] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1186] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1187] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1188] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1189] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1190] This invention relates to an "AI Conversation Advisor" that utilizes generative AI to provide conversation support and advice on what to say. This system collects and analyzes voice data in real time and provides appropriate responses to users, thereby supporting smooth communication.

[1191] System Overview

[1192] 1. User device: The user uses a smart glasses-type device or a smartphone. These devices are equipped with a microphone to collect conversations between the user and others in real time, and a display or audio output to show the results.

[1193] 2. Server: The collected voice data is sent to the server via the network, where it is analyzed using high-performance voice recognition and natural language processing technologies.

[1194] Program processing

[1195] Collection and transmission of voice data

[1196] First, when a user starts a conversation, the smart glasses or smartphone collects audio data. For example, when collecting audio during a meeting, collection begins automatically as soon as the meeting begins. The collected audio data undergoes pre-processing such as noise filtering to make it clear audio data. This audio data is then sent to a server in real time.

[1197] Analyzing voice data and understanding intent

[1198] The server uses speech recognition technology to convert the received voice data into text. The text data is then subjected to syntactic analysis, which identifies the subject, predicate, and object of the sentence and associates them with the context. Contextual and sentiment analysis are then performed to understand the intention and emotion behind the other person's remarks. For example, if the question includes "How is the project progressing?", it is interpreted as "You are asking about the progress of the project."

[1199] Summarization and response generation

[1200] The intent that is captured is summarized on the server and put into a form that is easy for the user to understand. For example, it may be summarized as "Your boss is asking about the progress of the project." The generative AI also generates multiple effective responses to utter next. Response candidates are customized based on the user's past speech patterns and set preferences.

[1201] Presentation of results

[1202] The summarized information and possible responses are sent to the device and displayed on the smartglasses display or smartphone screen. Voice guidance is also provided as needed. For example, the message might read, "Your boss is asking about the progress of a project. What do you think of the following response?"

[1203] Specific examples

[1204] Business Meeting Scenario

[1205] 1. User: Puts on smart glasses as soon as a business meeting begins.

[1206] 2. Terminal: Collects voice data, filters it, and then sends it to the server.

[1207] 3. Server: Converts the speech data into text and performs syntax and sentiment analysis. Example: "Your boss is asking about the progress of the project."

[1208] 4. Server: Generates candidate responses and presents options such as:

[1209] "The project is progressing as scheduled and is currently 70% complete."

[1210] "There are delays to some tasks, but we expect them to be completed by next week."

[1211] "Would you like a more detailed progress report?"

[1212] 5. Device: The generated summary and response suggestions are displayed on the smart glasses display. For example, it shows, "Your boss is asking about the progress of the project. What do you think of the following response?"

[1213] 6. User: Select the appropriate response from the suggested options and execute it in the conversation.

[1214] In this way, the "AI Conversation Advisor" will be able to assist users in their conversations in real time, allowing for smoother communication.

[1215] The processing flow will be explained below.

[1216] Step 1:

[1217] User: Turn on the smart glasses or smartphone and start collecting voice data of the conversation. For example, the moment the user starts speaking in a business meeting, voice data collection starts automatically.

[1218] Step 2:

[1219] Terminal: Noise filtering is performed on the collected voice data to remove background noise, etc. This allows for clear voice data to be obtained.

[1220] Step 3:

[1221] Terminal: Sends noise-filtered audio data to the server in real time. The data is transferred using a secure communication protocol.

[1222] Step 4:

[1223] Server: The received voice data is converted into text data using voice recognition technology. A highly accurate voice recognition engine is used in this process.

[1224] Step 5:

[1225] Server: Performs syntactic analysis on the text data to identify the components of a sentence, such as the subject, predicate, and object.

[1226] Step 6:

[1227] Server: Further contextual and sentiment analysis is performed to understand the intent and sentiment of the statement. For example, if the question "How is the project progressing?" is included, it will be interpreted as "Ask about the progress of the project."

[1228] Step 7:

[1229] Server: Summarizes the intent and generates information to present to the user in an easy-to-understand format. For example, it might summarize the intent as "The boss is asking about the progress of the project."

[1230] Step 8:

[1231] Server: Based on the generated summary, the server generates multiple effective responses for the next utterance. The responses are customized based on the user's past utterance patterns and preferences.

[1232] Step 9:

[1233] Server: Formats the generated summary and response candidates and sends them to the user's terminal.

[1234] Step 10:

[1235] Device: The summary and response candidates are displayed on the smartglasses display or smartphone screen. Voice guidance is also possible. For example, the message may say, "Your boss is asking about the progress of the project. What do you think of the following response?"

[1236] Step 11:

[1237] User: Choose the appropriate response from the suggested responses and use it in the actual conversation.

[1238] Through this series of steps, the "AI Conversation Advisor" supports users' conversations in real time, avoiding misunderstandings and miscommunication and ensuring smooth communication.

[1239] Example 1

[1240] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1241] During a conversation, it can be difficult for users to instantly grasp and respond appropriately. This is especially true in business meetings and important discussions, where it is necessary to respond appropriately on the spot. However, conventional technology lacks the means to analyze the content of a conversation in real time and provide appropriate responses. To solve this problem, a system that allows users to respond smoothly is needed.

[1242] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1243] In this invention, the server includes means for using a device to collect voice data, means for noise filtering the collected voice data in real time and transmitting the data to the server, means for converting the transmitted voice data into text using voice recognition technology, means for syntactically analyzing and sentiment analyzing the converted text data, means for extracting the intention of the utterance from the analyzed data, means for summarizing the intention of the utterance, means for generating information on effective next utterances using a generative AI model, and means for presenting the generated information on a user device, thereby enabling the user to grasp appropriate responses in real time and to communicate smoothly and effectively.

[1244] "Audio data" refers to audio signals of conversations or utterances made by a user.

[1245] "Device" refers to a hardware device for collecting audio data, including, for example, smart glasses and smartphones.

[1246] A "server" refers to a computer system that is connected via a network and processes and analyzes data.

[1247] "Noise filtering" refers to the process of removing unwanted background sounds and noise from collected audio data.

[1248] "Speech recognition technology" refers to technology for analyzing collected voice data and converting voice into text.

[1249] "Syntax analysis" refers to the process of analyzing the grammatical structure of text data and clarifying the relationships between the subject, predicate, object, and other parts of a sentence.

[1250] "Sentiment analysis" refers to the process of inferring the emotional state of a speaker from text data.

[1251] "Intention of speech" refers to understanding what the speaker intended based on the analyzed data.

[1252] "Summarization" means summarizing analyzed information into a concise and easy-to-understand format.

[1253] "Generative AI model" refers to an artificial intelligence model that performs natural language processing and is used to generate new text and responses.

[1254] The term "user device" refers to a device used by a user for displaying information and executing voice guidance.

[1255] This invention relates to an "AI Conversation Advisor" that utilizes generative AI to provide conversation support and advice on what to say. This system collects and analyzes voice data in real time and presents appropriate responses to users to support smooth communication.

[1256] System Overview

[1257] 1. User Devices

[1258] Users use smart glasses or smartphones, which are equipped with a microphone to collect conversations between the user and others in real time and a display or audio output to display the analysis results.

[1259] 2. Server

[1260] The collected voice data is sent to a server via a network. The server then uses high-performance voice recognition and natural language processing technologies to analyze the data. Specifically, it converts the voice data into text using voice recognition technology (e.g., Google Speech-to-Text API) and generates an optimal response using a generative AI model (e.g., OpenAI's GPT-4).

[1261] Program processing

[1262] First, when a user starts a conversation, the smart glasses or smartphone collects audio data. For example, when collecting audio during a meeting, collection begins automatically as soon as the meeting begins. The collected audio data undergoes pre-processing such as noise filtering on the device to create clear audio data. This audio data is then sent to the server in real time.

[1263] The server converts the received voice data into text using speech recognition technology. It then uses a generative AI model to perform syntax analysis and sentiment analysis on the text data to understand the intent and emotion of the other person's speech. For example, if the question "How is the project progressing?" is included, it can be interpreted as "A question about the progress of the project."

[1264] The intent that is captured is summarized on the server and put into a form that is easy for the user to understand. For example, it might summarize it as "your boss is asking about the progress of the project." It also uses a generative AI model to generate multiple effective next utterances. Response candidates are customized based on the user's past utterance patterns and set preferences.

[1265] The summarized information and possible responses are sent to the device and displayed on the smartglasses display or smartphone screen. Voice guidance is also provided as needed. For example, the message might read, "Your boss is asking about the progress of a project. What do you think of the following response?"

[1266] Specific examples

[1267] Business Meeting Scenario

[1268] 1. User: Puts on smart glasses as soon as a business meeting begins.

[1269] 2. Terminal: Collects voice data, filters it, and then sends it to the server.

[1270] 3. Server: Converts the speech data into text and performs syntax and sentiment analysis. Example: "Your boss is asking about the progress of the project."

[1271] 4. Server: Generates candidate responses and presents options such as:

[1272] "The project is progressing as scheduled and is currently 70% complete."

[1273] "There are delays to some tasks, but we expect them to be completed by next week."

[1274] "Would you like a more detailed progress report?"

[1275] 5. Device: The generated summary and response suggestions are displayed on the smart glasses display. For example, it shows, "Your boss is asking about the progress of the project. What do you think of the following response?"

[1276] 6. User: Select the appropriate response from the suggested options and execute it in the conversation.

[1277] Prompt Sentence Examples

[1278] Below are some example prompts to input to a generative AI model (e.g., GPT-4):

[1279] "If your boss is asking about the progress of a project, generate an appropriate response."

[1280] This will cause the AI ​​to generate a response like this:

[1281] 1. "The project is on schedule and is currently 70% complete."

[1282] 2. "We're running behind on some tasks, but we expect to complete them next week."

[1283] 3. "Would you like more detailed progress reports?"

[1284] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1285] Step 1: Collect and preprocess audio data

[1286] The user puts on the smart glasses or smartphone and launches the application.

[1287] Input: Real-time conversational audio produced by the user.

[1288] The device uses the device's microphone to capture audio data in real time. For example, if you're in a meeting, audio collection begins as soon as it starts.

[1289] The device performs noise filtering on the acquired voice data to remove background noise (for example, noise reduction processing using FFT).

[1290] Output: Clear audio data after filtering.

[1291] Step 2: Sending audio data

[1292] The device transmits the pre-processed voice data to the server in real time.

[1293] Input: Clear audio data after filtering.

[1294] Terminals may compress data to improve communication efficiency.

[1295] Output: The compressed audio data is sent over the network to the server.

[1296] Step 3: Convert audio data to text

[1297] The server converts the received voice data into text data using voice recognition technology (e.g., Google Speech-to-Text API).

[1298] Input: Compressed and transmitted audio data.

[1299] The server extracts linguistic features from the audio and converts it into text format.

[1300] Output: The converted text data.

[1301] Step 4: Parsing and sentiment analysis of text data

[1302] The server performs syntactic analysis on the generated text data, thereby identifying the subject, predicate, and object of the sentence.

[1303] Input: The converted text data.

[1304] The server then performs a sentiment analysis to assess the emotional state of the speaker.

[1305] Output: Parsed data and sentiment analysis results.

[1306] Step 5: Understand and summarize the intention of the statement

[1307] The server uses the analyzed data to understand the intent behind the statement.

[1308] Input: Syntactic and sentiment analysis results.

[1309] The server summarizes the intent of the conversation and condenses it into a form that is easy for the user to understand, such as "your boss is asking about the progress of a project."

[1310] Output: Summarized intent data.

[1311] Step 6: Generate candidate responses

[1312] The server uses a generative AI model (e.g., OpenAI's GPT-4) to generate multiple effective next utterances.

[1313] Input: Summarized intent data.

[1314] The server feeds example prompts to the model, which generates appropriate response patterns, taking into account the user's past speech patterns and preferences.

[1315] Output: The generated list of possible responses.

[1316] Step 7: Presenting results and user response

[1317] The server transmits the summarized information and candidate responses to the user terminal.

[1318] Input: The generated response candidate list.

[1319] The device will then display this information on the smartglasses display or smartphone screen, for example, in the form of, "Your boss is asking you about the progress of a project. What do you think of the following response?"

[1320] The user selects the appropriate response from the presented options and executes it in the actual conversation. Selection can be done by voice input or touch operation.

[1321] Output: The response selected by the user.

[1322] (Application example 1)

[1323] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1324] There is a need to improve the quality of customer interactions in brick-and-mortar stores and simultaneously increase sales staff efficiency and customer satisfaction. In particular, there is a lack of support tools to enable appropriate and prompt responses during customer service, and technology to ensure smooth communication with customers is needed. There is also a need for a system that can provide consistent, high-quality customer service that is not dependent on the experience or skills of staff.

[1325] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1326] In this invention, the server includes a preprocessing means for performing noise filtering on voice data collected using a smart device, a means for converting the voice data sent to the server into text and performing syntax analysis and sentiment analysis, and a means for displaying the generated response candidates and summaries on the display of the smart device. This makes it possible to support conversations with customers in real time using the smart device, allowing sales staff to respond to customers more quickly and accurately.

[1327] "Conversational voice data" refers to data in which the voice of a user or customer is recorded in digital format.

[1328] "Means for transmitting in real time" refers to technology for transmitting collected voice data to a server immediately without delay.

[1329] "Means for analyzing" refers to the technology or system that converts transmitted voice data into text data and understands grammar and context.

[1330] "Means for understanding the intention of a statement" refers to technology for extracting the speaker's intentions and emotions from analyzed text data.

[1331] "Means of summarizing" refers to the technique of summarizing the intent and information that has been extracted into a concise, to-the-point format.

[1332] The "means for generating effective information" is a technology for automatically generating information that is determined to be effective when spoken next.

[1333] "Means for presenting" refers to a technique for visually or audibly notifying the user of the generated information.

[1334] "Smart device" refers to an electronic device that can be carried or worn by a user and has the function of collecting and displaying voice data.

[1335] "Noise filtering" is a technique for removing background noise from audio data to obtain clear audio data.

[1336] "Syntax analysis" is a technique for analyzing the grammatical structure of text data and identifying elements such as subjects, predicates, and objects.

[1337] "Sentiment analysis" is a technology that identifies emotions from a speaker's text data and classifies the type of emotion.

[1338] "Response candidates" refer to multiple appropriate responses to be made next.

[1339] A "summary" is a concise summary of the main points and important information of a conversation.

[1340] "Means for displaying on a display" refers to technology for displaying the generated information on the screen of a smart device.

[1341] System configuration

[1342] A system for implementing the present invention includes the following hardware and software.

[1343] Smart device: A device carried or worn by the user that has the ability to collect voice data and display the results. Examples include smart glasses and smartphones.

[1344] Server: Has high-performance computing resources to perform voice data analysis, syntax analysis, and sentiment analysis. It mainly uses speech recognition APIs (e.g., Google Speech-to-Text API) and natural language processing APIs (e.g., OpenAI GPT-3).

[1345] Program Description

[1346] Collection and transmission of voice data

[1347] Smart devices (smart glasses or smartphones) collect the user's voice data in real time using a microphone built into the device. The collected voice data is filtered to make it clear and then sent to a server.

[1348] Analyzing voice data and understanding intent

[1349] The server converts the transmitted voice data into text using a speech recognition API (Google Speech-to-Text API). A natural language processing API (OpenAI GPT-3) is used to perform syntactic analysis and sentiment analysis on the converted text data. Syntactic analysis identifies the subject, predicate, and object, while sentiment analysis extracts the speaker's intention and emotion from the context.

[1350] Summarization and response generation

[1351] Based on the intent, the server summarizes the key points. The AI ​​then generates multiple effective responses for the next utterance. These responses are customized based on the user's past speech patterns and preferences.

[1352] Presentation of results

[1353] The summary and candidate responses generated by the server are displayed on the smart device's display. Voice guidance is also provided as needed. For example, a message such as, "A customer has a question about a new smartphone. What do you think of the following response?" can be displayed.

[1354] Specific examples

[1355] Scene: Customer service in a brick-and-mortar store

[1356] 1. A user puts on the smart glasses and starts a conversation with a customer.

[1357] 2. The smart device collects the voice data and sends it to the server along with noise filtering.

[1358] 3. The server converts the voice data into text and performs syntax and sentiment analysis. Example: "A customer is asking about the features of their smartphone."

[1359] 4. The server generates a candidate response, for example:

[1360] "This smartphone has an excellent camera."

[1361] "Long battery life and convenient"

[1362] Want to know more about the features?

[1363] 5. The generated summary and response suggestions are displayed on the smart glasses.

[1364] Prompt Sentence Examples

[1365] A customer has asked a question about a new smartphone. Generate possible responses.

[1366] In this way, the system can assist sales staff in serving customers in real time and improve customer satisfaction.

[1367] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1368] Step 1:

[1369] Smart devices (smart glasses or smartphones) collect voice data of conversations between users and customers in real time. The input is the voice data of the conversation, and the output is the collected voice data. The voice data is acquired using a microphone, and clear voice data is prepared by noise filtering. This voice data is then passed to the next step.

[1370] Step 2:

[1371] The smart device sends noise-filtered audio data to the server. The input is clear audio data, and the output is the audio data sent to the server. Here, the data is sent in real time using the device's communication module.

[1372] Step 3:

[1373] The server converts the transmitted voice data into text data. The input is voice data and the output is text data. The server uses a speech recognition API (Google Speech-to-Text API) to analyze the voice waveform data and generate a corresponding string.

[1374] Step 4:

[1375] The server analyzes the converted text data and identifies grammatical structures. The input is the text data, and the output is the results of the analysis. This process uses a natural language processing API (OpenAI GPT-3) to identify sentence elements such as subject, predicate, and object.

[1376] Step 5:

[1377] The server performs sentiment analysis based on the parsing results. The input is the parsing results, and the output is the intention of the utterance and the sentiment analysis results. The sentiment analysis module is used to determine the speaker's emotional state and intention. This information is used for further processing.

[1378] Step 6:

[1379] The server summarizes the main points of the speech based on the results of sentiment analysis. The input is the intention of the speech and the results of sentiment analysis, and the output is a summary. A summarization algorithm is used to extract the main points of the text and create a concise summary.

[1380] Step 7:

[1381] The server generates effective response candidates based on the summary. The input is the summary, and the output is multiple response candidates. Using a generative AI model (OpenAI GPT-3), multiple next-best utterances are generated.

[1382] Step 8:

[1383] The smart device receives the summary and candidate responses sent from the server and displays them on the display. The input is the summary and candidate responses, and the output is information provided to the user's visual sense. The display function of the device is used to visually present appropriate candidate responses to the user.

[1384] Step 9:

[1385] The user checks the response candidates presented by the smart device, selects the most appropriate one, and speaks to the customer. The input is the displayed response candidate, and the output is the response selected by the user. This enables smooth and effective customer service.

[1386] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1387] This invention relates to an "AI Conversation Advisor" that combines generative AI and an emotion engine to provide conversation support and advice on what to say. This system collects and analyzes voice data in real time, recognizes emotions, and then presents appropriate responses to the user, supporting smooth communication.

[1388] System Overview

[1389] 1. User device: The user uses a smart glass or smartphone, which is equipped with a microphone to collect real-time conversations between the user and others, a camera to capture the user's speech and facial expressions, and a display or audio output to show the results.

[1390] 2. Server: The collected audio and video data is sent to the server via the network. The server uses high-performance speech recognition technology, natural language processing technology, and an emotion engine to analyze the data.

[1391] Program processing

[1392] Collection and transmission of audio and video data

[1393] First, when a user starts a conversation, the smart glasses or smartphone collects audio and video data. For example, if collecting audio and facial expressions during a meeting, collection begins automatically as soon as the meeting begins. The collected data undergoes pre-processing such as noise filtering and video cleaning. This data is then sent to a server in real time.

[1394] Data analysis and understanding of intentions and emotions

[1395] The server converts the received voice data into text data using speech recognition technology. The text data is then subjected to syntactic analysis, which identifies the subject, predicate, and object of the sentence and associates them with the context. At the same time, the video data is analyzed by an emotion engine, which recognizes emotions from the user's facial expressions and tone of voice. For example, the question "How is the project progressing?" is analyzed as "You are asking about the progress of the project," and it is recognized that the user is feeling nervous.

[1396] Summarization and response generation

[1397] The intent and emotions that have been captured are summarized on the server, generating information to present to the user in an easy-to-understand format. For example, it may summarize the message as "Your boss is asking about the progress of the project." The generation AI also generates multiple effective responses to utter next. Response candidates are customized based on the user's past speech patterns and preferences. Additionally, the response content is adjusted based on the user's emotions as recognized by the emotion engine.

[1398] Presentation of results

[1399] The summarized information and possible responses are sent to the device and displayed on the smartglasses display or smartphone screen. Voice guidance is also provided as needed. For example, the message might read, "Your boss is asking about the progress of a project. What do you think of the following response?"

[1400] Specific examples

[1401] Business Meeting Scenario

[1402] 1. User: Puts on smart glasses as soon as a business meeting begins.

[1403] 2. Terminal: Collects audio and video data, performs noise filtering and video preprocessing, and then sends it to the server.

[1404] 3. Server: Converts the voice data into text and performs syntax analysis and sentiment analysis. For example, it analyzes "My boss is asking about the progress of the project" and recognizes that the user is nervous.

[1405] 4. Server: Generates response candidates and presents options such as:

[1406] "The project is on schedule and is currently 70% complete." (In a calm tone to ease tension)

[1407] "We're experiencing delays with some tasks, but we expect them to be completed by next week." (Providing reassurance with specific numbers)

[1408] "Would you like more detailed progress reports?" (Supportive attitude)

[1409] 5. Device: The generated summary and response suggestions are displayed on the smart glasses display. For example, it may say, "Your boss is asking about the progress of the project. What do you think of the following response?"

[1410] 6. User: Select the appropriate response from the presented options and execute it in the conversation.

[1411] In this way, the "AI Conversation Advisor" can assist users in conversations in real time, avoiding misunderstandings and miscommunication while providing appropriate responses that take emotions into account.

[1412] The processing flow will be explained below.

[1413] Step 1:

[1414] User: Turns on the smart glasses device or smartphone and collects audio and video data of the conversation. For example, the moment a user starts speaking in a business meeting, audio and video data collection begins automatically.

[1415] Step 2:

[1416] Terminal: Noise filtering is performed on the collected audio data to remove unwanted noise, and video cleaning is performed on the video data to generate clearer images.

[1417] Step 3:

[1418] Terminal: After noise filtering and image cleaning, the data is sent to the server in real time via a secure communication protocol.

[1419] Step 4:

[1420] Server: Converts the received voice data into text data using voice recognition technology. Stores the converted text data.

[1421] Step 5:

[1422] Server: Analyzes the received video data and combines facial expression recognition algorithms and tone analysis of audio data to identify the user's emotions. For example, it recognizes whether the user is nervous or angry.

[1423] Step 6:

[1424] Server: Performs syntactic analysis on text data to extract components such as the subject, predicate, and object of a sentence. It also performs contextual analysis to understand the intention and topic of what is being said.

[1425] Step 7:

[1426] Server: Based on the results of syntactic and contextual analysis, summarizes what the other person is saying. For example, it summarizes it as "your boss is asking about the progress of the project."

[1427] Step 8:

[1428] Server: Based on the summarized information and the user's emotions recognized by the emotion engine, the server generates effective response candidates for the next utterance. For example, if the user is nervous, the server generates a response in a calm tone.

[1429] Step 9:

[1430] Server: Formats the generated summary and candidate responses and sends them to the user terminal. Data is transmitted end-to-end encrypted.

[1431] Step 10:

[1432] Device: A formatted summary and suggested responses are displayed on the smartglasses display or smartphone screen. Voice guidance is also possible if necessary. For example, the display might say, "Your boss is asking about the progress of the project. What do you think of the following response?"

[1433] Step 11:

[1434] User: Choose the appropriate response from the suggested responses and use it in the actual conversation.

[1435] Through this series of steps, the AI ​​conversation advisor can assist users in real-time conversations, avoiding misunderstandings and miscommunications, and providing appropriate responses that take emotions into account.

[1436] Example 2

[1437] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1438] Conventional conversation support systems were unable to fully grasp the intentions and emotions of users' remarks, resulting in misunderstandings and miscommunication. Furthermore, due to a lack of technology to recognize emotions from video data in addition to analyzing audio data, it was difficult to provide accurate responses in real time. This meant that users had to perform many manual operations to obtain the information they needed, making it difficult to support smooth communication.

[1439] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for collecting audio data and video data of a conversation, means for transmitting the collected audio data and video data in real time, means for converting the transmitted audio data into text data, means for analyzing the syntax of the text data and recognizing the user's emotions, means for summarizing the analyzed data and information obtained from the emotion recognition, means for generating information that would be effective to utter next based on the summarized information, and means for presenting the generated information to the user. This makes it possible to accurately analyze the user's intentions and emotions based on the collected data and generate and present an appropriate response in real time.

[1440] "Means for collecting audio data and video data of a conversation" is a general term for devices equipped with a microphone and a camera for capturing the user's voice and video in real time.

[1441] The "means for transmitting the collected audio data and video data in real time" refers to a communication function for instantly transmitting the collected data to a server.

[1442] The "means for converting the transmitted voice data into text data" refers to a method for converting voice data into text format using voice recognition technology.

[1443] The "means for analyzing the syntax of the text data and recognizing the user's emotions" refers to a function for analyzing the grammatical structure of the text data using natural language processing technology and recognizing the user's emotions.

[1444] The "means for summarizing the analyzed data and information obtained from emotion recognition" refers to a function for summarizing information in order to display the analysis results in a concise manner.

[1445] "Means for generating information that would be effective to utter next based on the summarized information" refers to a technology that generates a response that would be effective for the user to utter next based on summarized information.

[1446] "Means for presenting the generated information to the user" refers to an interface including a display and a speaker for visually or audibly presenting the generated response candidates to the user.

[1447] MODE FOR CARRYING OUT THE INVENTION

[1448] This invention relates to a system that supports user conversations in real time by combining audio data and video data. The system consists of a terminal worn by the user and a server that processes the data.

[1449] Device configuration

[1450] Users use devices such as smart glasses and smartphones. These devices include the following hardware:

[1451] 1. Microphone: To collect audio data of conversation.

[1452] 2. Camera: To capture the user's facial expressions and collect video data.

[1453] 3. Display: For showing summarized information and possible responses.

[1454] 4. Communication module: To transmit collected data to the server in real time.

[1455] Server Configuration

[1456] The server includes the following software modules and functions:

[1457] 1. Speech recognition technology: Used to convert voice data into text data. Specifically, speech recognition engines include Google Cloud Speech-to-Text API and IBM Watson Speech to Text.

[1458] 2. Natural language processing technology: Used to analyze the syntax of text data and understand the user's intent. For example, libraries such as Spacy and NLTK are used.

[1459] 3. Emotion Engine: Recognizes user emotions from video data and analyzes voice tone.

[1460] 4. Generative AI models: Used to generate effective next utterances. For example, OpenAI's GPT-4.

[1461] 5. Summarization module: Summarizes the acquired information and presents it to the user.

[1462] Specific examples

[1463] Business Meeting Scenario

[1464] 1. User: The user puts on the smart glasses as soon as the business meeting begins.

[1465] 2. Terminal: Collects audio and video data, performs noise filtering and video preprocessing, and then sends it to the server.

[1466] 3. Server: Converts the voice data into text and performs syntax analysis and sentiment analysis. For example, it analyzes the speech data as "your boss is asking about the progress of the project" and recognizes that the user is nervous.

[1467] 4. Server: Generates response candidates and presents options such as:

[1468] "The project is on schedule and is currently 70% complete." (In a calm tone to ease tension)

[1469] "We're experiencing delays with some tasks, but we expect them to be completed by next week." (Providing reassurance with specific numbers)

[1470] "Would you like more detailed progress reports?" (Supportive attitude)

[1471] 5. Terminal: The generated summary and response suggestions are displayed on the screen. For example, it may be displayed in the form of "Your boss is asking about the progress of the project. What do you think of the following response?"

[1472] 6. User: Select the appropriate response from the presented options and execute it in the conversation.

[1473] Prompt Sentence Examples

[1474] "Please advise me on how to respond in a tense situation when someone asks me how the project is going."

[1475] "Generate phrases to speak in a calm tone when reporting an ongoing project to your boss."

[1476] The above is a specific embodiment for carrying out the present invention. This system effectively combines audio and video data to support user conversations in real time, thereby facilitating user communication and preventing misunderstandings and miscommunication.

[1477] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1478] Step 1: User initiates conversation

[1479] A user puts on smart glasses or a smartphone and starts a conversation. Specifically, the device automatically detects the start of a conversation, or the user manually instructs the device to start a conversation. The input is the user's voice and video, and the output is the device collecting these data.

[1480] Step 2: Collect and preprocess audio and video data

[1481] The device collects audio and video data of the conversation. It uses a microphone to pick up the audio and a camera to capture the user's face. The input is the collected audio and video data, and the output is preprocessed data. Specific preprocessing steps include reducing background noise from the audio data and adjusting the brightness and contrast of the video data.

[1482] Step 3: Send data from the device to the server

[1483] The terminal transmits the preprocessed data to the server in real time. The input is the preprocessed audio and video data, and the output is the data transmitted to the server. The data is converted into a packet format and transmitted using a secure communication protocol.

[1484] Step 4: Convert audio data to text

[1485] The server converts the received voice data into text data using voice recognition technology. The input is voice data and the output is text data. Specifically, it uses a voice recognition engine (e.g., Google Cloud Speech-to-Text API) to convert voice into text in real time.

[1486] Step 5: Parsing and Emotion Recognition

[1487] The server performs syntactic analysis of the text data and simultaneously performs emotion recognition using video data. The input is text data and video data, and the output is an analysis result that includes intent and emotion. A natural language processing library (e.g., Spacy) is used to identify the subject, predicate, and object of the sentence. At the same time, the emotion engine analyzes facial expressions and voice tone to recognize the user's emotions.

[1488] Step 6: Summarize and generate a response

[1489] The server summarizes the analysis results and generates effective information for the next user utterance. The input is the analysis results, and the output is summarized information and response candidates. A generative AI model (e.g., GPT-4) is used to generate multiple next utterances. Responses are customized based on past utterance patterns and the user's preferences.

[1490] Step 7: Summarize and present response candidates to the device

[1491] The server sends the generated summary and candidate responses to the terminal. The input is the summarized information and candidate responses, and the output is the information displayed on the terminal. The terminal receives this and displays on its display, "Your boss is asking about the progress of the project. What do you think of the following response?"

[1492] Step 8: User selects and executes the best response

[1493] The user selects an appropriate response from the candidate responses displayed on the device and executes it in the conversation. The input is the displayed candidate response, and the output is the confirmed response selected by the user. The user selects a response using the touch interface or voice commands and executes the response in the actual conversation.

[1494] (Application example 2)

[1495] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1496] In autonomous vehicles, communication with passengers and drivers can be difficult, and a major issue is the lack of appropriate instructions and information provided in emergencies. Furthermore, there is a need for a flexible system that can respond to various situations that arise while driving.

[1497] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for transmitting audio data and video data of a conversation in real time, means for analyzing the transmitted audio data and video data, and means for extracting the intention and emotion of the utterance from the analyzed data. This makes it possible to identify the intention and emotion of the speaker through the analysis of the audio and video data in real time and provide appropriate responses and instructions.

[1498] "Conversational voice data" refers to voice signals uttered by a user, and includes linguistic information and emotional information.

[1499] "Video data" refers to video signals collected to capture a user's facial expressions and movements, and includes visual information.

[1500] "Means for transmitting in real time" refers to a function for transmitting collected audio and video data to a server immediately without delay.

[1501] "Means for analysis" refers to software or algorithms used to analyze transmitted audio and video data, thereby enabling understanding of intent and emotion.

[1502] "Means of understanding intent" refers to the function of understanding the speaker's purpose and intention from the analyzed data.

[1503] "Means of capturing emotions" refers to the function of identifying the speaker's emotional state from the analyzed data.

[1504] "Means of summarization" refers to the function of summarizing analyzed information in a concise and easy-to-understand form.

[1505] "Means for generating information that will be effective to utter next" refers to the function for generating appropriate content to be uttered next based on the intention and emotion that has been grasped.

[1506] "Presentation means" refers to a device or function for visually or audibly presenting the generated information to the user.

[1507] "Means for providing appropriate instructions in an emergency" refers to a function that provides users with prompt and accurate instructions based on the results of data analysis in an emergency situation.

[1508] "Noise filtering" refers to the process of removing unwanted background noise from collected audio data.

[1509] "Image processing" refers to the process of converting collected video data into a form that is easier to clean and analyze.

[1510] "Normalizing the volume" refers to the process of adjusting the volume level of audio data to keep it constant.

[1511] System program generation and hardware / software configuration

[1512] System Configuration

[1513] This system includes multiple devices and servers for collecting and analyzing audio and video data of conversations in real time. Details of each element are given below.

[1514] 1. User Device:

[1515] Smart glasses: Worn by passengers or drivers, these glasses contain microphones and cameras for real-time audio and video capture.

[1516] Smartphones: Can be used as auxiliary devices to collect audio and video data.

[1517] 2. Server:

[1518] Speech recognition system: Uses the speech_recognition library to convert voice data into text.

[1519] Emotion analysis system: Uses the emotion_recognition library to identify emotions from collected audio and video data.

[1520] Natural language processing engine: Uses generative AI models to generate responses based on the speaker's intent and sentiment.

[1521] Data Processing

[1522] 1. Audio and video data collection and transmission:

[1523] Smart glasses or smartphones collect real-time audio and video data of conversations, which undergo local noise filtering and image processing before being sent to a server.

[1524] 2. Audio and video data analysis:

[1525] The server converts the received audio data into text data using the speech_recognition library, then performs syntactic analysis on the text data and combines it with context.

[1526] Next, the collected video data is analyzed using the emotion_recognition library to identify the speaker's emotions.

[1527] 3. Understanding intentions and emotions and summarizing information:

[1528] The system identifies the speaker's intention and emotion from the analyzed data and summarizes this information. For example, if the question "Where is the next exit?" is analyzed and the speaker is found to be impatient, it can be summarized as "The speaker is in a hurry to find out where the next exit is."

[1529] 4. Response generation and presentation:

[1530] Based on the summarized information, a generative AI model is used to generate appropriate responses, which are customized based on the user's past speech patterns and set preferences.

[1531] The generated response will be displayed on the smartglasses display or smartphone screen, and will also be provided as an audio guide if necessary.

[1532] Examples and prompts

[1533] Example 1: Emergency response on expressways

[1534] Scenario: Providing emergency information when a driver brakes suddenly.

[1535] Voice input: A passenger asks, "Where is the next exit?"

[1536] Emotion analysis: Detect anxiety from the tone of voice.

[1537] AI response: Instruct: "The next exit is 3 kilometers away. Please remain calm and keep driving."

[1538] Example prompt sentence:

[1539] Consider the user's nervousness and create a guiding message like this: 'Where is the next exit?' Answer in a calm tone.

[1540] Thus, the embodiment of the present invention includes a series of processes that collects audio and video data, analyzes it to understand the user's intentions and emotions, and provides appropriate information. In particular, it enables quick and accurate instructions in emergencies while driving, thereby improving the safety and comfort of the user.

[1541] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1542] Step 1:

[1543] The user device collects audio and video data of the conversation. Specifically, the microphone and camera built into smart glasses or smartphones record and record the user's speech and facial expressions in real time. The input is the user's voice and facial expressions, and the output is the collected audio and video data.

[1544] Step 2:

[1545] The terminal transmits collected audio and video data to the server in real time. Specifically, after noise filtering and preprocessing, the data is transmitted to the server via the network. The input is the collected audio and video data, and the output is the transmission of the preprocessed data to the server.

[1546] Step 3:

[1547] The server converts the received voice data into text data using the speech_recognition library. Specifically, it analyzes the voice signal using a speech recognition algorithm and converts the results into text format. The input is preprocessed voice data, and the output is text data.

[1548] Step 4:

[1549] The server analyzes the received video data using the emotion_recognition library to identify the user's emotions. Specifically, it analyzes facial expressions and vocal tone to identify the speaker's emotional state. The input is preprocessed video data, and the output is identified emotional information.

[1550] Step 5:

[1551] The server uses data obtained from speech recognition and emotion analysis to understand the speaker's intent and summarize the information. Specifically, it uses natural language processing algorithms to understand the gist of what is being said and create a concise summary. The input is text data and emotion information, and the output is summarized information.

[1552] Step 6:

[1553] The server uses the generative AI model to generate information that will be effective next time. Specifically, it generates an appropriate response based on the summarized information and the speaker's emotional state. The input is the summarized information and emotional information, and the output is the generated response.

[1554] Step 7:

[1555] The device presents the generated response. Specifically, the response is displayed on the smartglasses display or smartphone screen, and audio guidance is also provided if necessary. The input is the generated response, and the output is the information presented to the user.

[1556] This series of processes allows users to receive accurate information and instructions in real time. In particular, in emergencies, safety can be improved by providing prompt and appropriate instructions according to the user's emotional state.

[1557] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1558] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1559] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1560] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1561] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1562] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1563] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1564] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1565] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1566] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1567] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1568] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1569] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1570] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1571] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1572] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1573] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1574] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1575] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1576] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1577] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1578] The following is further disclosed regarding the above embodiment.

[1579] (Claim 1)

[1580] a means for collecting audio data of the conversation;

[1581] means for transmitting the collected voice data in real time;

[1582] means for analyzing the transmitted voice data;

[1583] means for extracting the intention of the speech from the analyzed voice data;

[1584] A means for summarizing information obtained from the means for grasping the intention of the utterance;

[1585] A means for generating information that is effective to be uttered next based on the summarized information;

[1586] means for presenting the generated information;

[1587] A system including:

[1588] (Claim 2)

[1589] 10. The system of claim 1, further comprising pre-processing means for noise filtering and volume normalizing the user's voice data.

[1590] (Claim 3)

[1591] 10. The system of claim 1, further comprising means for customizing the generated information according to the user's past speech patterns and set preferences.

[1592] "Example 1"

[1593] (Claim 1)

[1594] using a device to collect audio data;

[1595] means for filtering the collected voice data in real time and transmitting the filtered voice data to a server;

[1596] means for converting the transmitted voice data into text using voice recognition technology;

[1597] means for syntactically and sentimentally analyzing the converted text data;

[1598] A means for extracting the intention of the utterance from the analyzed data;

[1599] means for summarizing the intent of the utterance;

[1600] A means of generating effective information to be spoken next using a generative AI model;

[1601] means for presenting the generated information on a user device;

[1602] A system including:

[1603] (Claim 2)

[1604] 10. The system of claim 1, further comprising pre-processing means for noise filtering and volume normalizing the user's voice data.

[1605] (Claim 3)

[1606] 10. The system of claim 1, further comprising means for customizing the generated information according to the user's past speech patterns and set preferences.

[1607] "Application Example 1"

[1608] (Claim 1)

[1609] a means for collecting audio data of the conversation;

[1610] means for transmitting the collected voice data in real time;

[1611] means for analyzing the transmitted voice data;

[1612] means for extracting the intention of the speech from the analyzed voice data;

[1613] A means for summarizing information obtained from the means for grasping the intention of the utterance;

[1614] A means for generating information that is effective to be uttered next based on the summarized information;

[1615] means for presenting the generated information;

[1616] a pre-processing means for performing noise filtering on the voice data collected using the smart device;

[1617] means for converting the voice data transmitted to the server into text and performing syntax analysis and sentiment analysis;

[1618] a means for displaying the generated answer candidates and summary on a display of the smart device;

[1619] A system including:

[1620] (Claim 2)

[1621] 10. The system of claim 1, further comprising pre-processing means for noise filtering and volume normalizing the user's voice data.

[1622] (Claim 3)

[1623] 10. The system of claim 1, further comprising means for customizing the generated information according to the user's past speech patterns and set preferences.

[1624] "Example 2: Combining Emotion Engines"

[1625] (Claim 1)

[1626] means for collecting audio and video data of the conversation;

[1627] means for transmitting the collected audio and video data in real time;

[1628] means for converting the transmitted voice data into text data;

[1629] means for analyzing the syntax of the text data and recognizing the user's emotions;

[1630] means for summarizing the analyzed data and information obtained from emotion recognition;

[1631] A means for generating information that is effective to be uttered next based on the summarized information;

[1632] means for presenting the generated information to a user;

[1633] A system including:

[1634] (Claim 2)

[1635] 2. The system of claim 1, further comprising means for pre-processing the user's audio and video data, in particular for noise filtering and video cleaning.

[1636] (Claim 3)

[1637] 10. The system of claim 1, further comprising means for customizing the generated information according to the user's past speech patterns and set preferences.

[1638] "Application example 2 when combining emotion engines"

[1639] (Claim 1)

[1640] a means for collecting audio data of the conversation;

[1641] means for transmitting the collected audio and video data in real time;

[1642] means for analyzing the transmitted audio data and video data;

[1643] A means for extracting the intention and emotion of the utterance from the analyzed data;

[1644] A means for summarizing information obtained from the means for grasping the intention and emotion of the utterance;

[1645] A means for generating information that is effective to be uttered next based on the summarized information;

[1646] means for presenting the generated information;

[1647] A means for identifying the speaker's emotions and providing appropriate instructions in an emergency;

[1648] A system including:

[1649] (Claim 2)

[1650] 10. The system of claim 1, further comprising pre-processing means for noise filtering and image processing the user's audio and video data, and for volume normalization.

[1651] (Claim 3)

[1652] 10. The system of claim 1, further comprising means for customizing the generated information according to the user's past speech patterns and set preferences. [Explanation of symbols]

[1653] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for collecting audio data of the conversation; means for transmitting the collected voice data in real time; means for analyzing the transmitted voice data; means for extracting the intention of the speech from the analyzed voice data; A means for summarizing information obtained from the means for grasping the intention of the utterance; A means for generating information that is effective to be uttered next based on the summarized information; means for presenting the generated information; A system including:

2. 2. The system of claim 1, further comprising pre-processing means for noise filtering and volume normalizing the user's voice data.

3. 10. The system of claim 1, further comprising means for customizing the generated information according to the user's past speech patterns and set preferences.

Images