System

Abstract

A system is provided.SOLUTION: A system includes a means for acquiring voice data in an online conference, a means for converting the acquired voice data into text data in real time, a means for analyzing the text data and specifying the intention of a user, a means for retrieving appropriate information on the basis of the specified intention, and a means for transmitting the retrieved information to a user terminal and displaying it on a user screen.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's business environment, online meetings have become commonplace, but it is becoming increasingly difficult for salespeople and other personnel to instantly obtain necessary information during meetings and provide appropriate responses and proposals. In particular, if appropriate materials and information cannot be presented in real time based on the flow of conversation and questions, the effectiveness and efficiency of business negotiations will decrease. For this reason, there is a need for a system that can analyze conversation content in real time during online meetings and instantly provide appropriate information. [Means for solving the problem]

[0005] The present invention provides a system including means for acquiring voice data during an online conference, means for converting the acquired voice data into text data in real time, means for analyzing the text data and identifying the user's intent, means for searching for appropriate information based on the identified intent, and means for transmitting the searched information to a user terminal and displaying it on the user's screen. This system enables users to instantly acquire necessary information during an online conference and make appropriate responses and suggestions.

[0006] An "online conference" is a system that allows multiple users to communicate in real time via voice and video over the Internet.

[0007] "Audio data" refers to data that represents sound in digital form and is acquired through a microphone or recording device.

[0008] "Text data" refers to data in sentence format that has been converted using voice recognition technology.

[0009] "User intent" refers to what a user is asking for or what information they need in a conversation or statement.

[0010] A "search tool" is an algorithm or method for retrieving the required information from a database or external source based on a specific intent.

[0011] A "user terminal" is a device such as a computer or smartphone used to conduct an online conference.

[0012] "Means for displaying on the screen" refers to a method for visually displaying acquired information on the display of a user terminal. [Brief explanation of the drawings]

[0013] [Figure 1]1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] This invention is a system that analyzes the content of online meetings in real time and displays appropriate information only on the user's screen. This system consists of a server and a user terminal.

[0035] System configuration

[0036] 1. User Device

[0037] A user terminal is a device such as a computer or smartphone that has the functionality to hold online meetings. The user terminal is responsible for capturing audio data, sending data to a server, receiving data from the server, and displaying the data on the screen.

[0038] 2. Server

[0039] It is a remote computer that receives voice data, analyzes it, converts it into text, identifies intent, retrieves and summarizes information, and transmits the data to a user terminal.

[0040] Program processing explanation

[0041] 1. Acquiring and sending audio data

[0042] When a user starts an online conference, the user's device will use a built-in or external microphone to capture the conference audio in real time, and the captured audio data will be divided into packets and sent to the server at regular intervals.

[0043] 2. Analysis of voice data and conversion to text

[0044] The server analyzes the voice data received from the user's device in real time and converts it into text data using voice recognition technology. This conversion is performed using a highly accurate voice recognition engine.

[0045] 3. User Intent Analysis

[0046] Using text generation AI, the server analyzes the user's statements based on text data and identifies the information and answers the user is looking for. Natural language processing (NLP) technology is used for intent analysis.

[0047] 4. Finding and summarizing relevant information

[0048] Based on the identified intent, the server searches for the appropriate information from internal databases and external sources, summarizes the information, and presents it in a format that is easily understood by the user. During the summarization, key points are extracted and bullet points, simple graphs, and illustrations are generated.

[0049] 5. Transmission and Display of Information

[0050] The server sends the summarized information to the user's device, which then displays it on the screen in a pop-up format, allowing the user to check the information without having to change their line of sight during the meeting.

[0051] Specific examples

[0052] Example 1: Providing product information

[0053] Imagine a user says, "Tell me more about this product" during an online meeting. The user's device captures this speech and sends the audio data to the server. The server analyzes the audio data and converts it into text data. If the text generation AI determines that "more product details are being sought," the server searches the company's internal database for product specifications and manuals. The server summarizes the information obtained and sends it to the user's device. The user's device then displays the summary information as a pop-up on the screen.

[0054] Example 2: Checking contract terms

[0055] Imagine a case where a user says, "Please tell me more about the terms of the contract." The user's device sends voice data to the server, which converts the voice data into text data. The text generation AI extracts the keyword "terms of contract," and the server retrieves the relevant contract terms document from the database and summarizes it. The summary information is sent to the user's device, which displays the information on the screen.

[0056] In this way, the present invention provides a system that allows users to instantly obtain necessary information during an online conference and provide appropriate answers and suggestions.

[0057] The processing flow will be explained below.

[0058] Step 1:

[0059] When a user starts an online conference, the user's device captures the conference audio in real time using a built-in or external microphone, and the captured audio data is divided into packets and sent to the server at regular intervals.

[0060] Step 2:

[0061] The server receives the voice data sent from the user terminal and stores it in a buffer.

[0062] Step 3:

[0063] The server uses speech recognition technology to convert the voice data into text data, using highly accurate speech recognition engines such as Google Speech-to-Text API and IBM Watson Speech to Text.

[0064] Step 4:

[0065] The server inputs the converted text data into a text generation AI (e.g., GPT-3) and analyzes what the user said.

[0066] Step 5:

[0067] The server extracts key keywords and questions from the results of the text generation AI analysis and identifies the user's intent, such as a specific request like "I want to know the price of a product."

[0068] Step 6:

[0069] Based on the user's identified intent, the server queries internal databases and external sources to find the required information.

[0070] Step 7:

[0071] The server analyzes the retrieved information and summarizes the key points, presenting them in a format that is easily understandable to the user (e.g., bullet points or key points).

[0072] Step 8:

[0073] The server sends the summarized information to the user's device, optionally specifying the display format (e.g., popup, sidebar).

[0074] Step 9:

[0075] The user device analyzes the received information and displays it on the user's screen in the specified format, allowing the user to check the information necessary for the conversation without having to change their line of sight during the meeting.

[0076] This processing flow allows users to instantly obtain the right information during online meetings and keep the conversation moving smoothly.

[0077] Example 1

[0078] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0079] Online meetings often present a challenge, making it difficult for users to quickly obtain the information they need. In particular, the lack of real-time information search and a summary of the information provided means users must interrupt the flow of the meeting to conduct research, reducing meeting efficiency. Furthermore, the accuracy and appropriateness of the information obtained can also be a problem.

[0080] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0081] In this invention, the server includes means for acquiring voice data, means for converting the acquired voice data into text data in real time, means for analyzing the text data and identifying the user's intention, means for searching for appropriate information based on the identified intention, means for summarizing the searched information, and means for transmitting the summarized information to a user terminal and displaying it on the user's screen, thereby enabling users to acquire necessary information in real time during an online conference and understand that information in a short amount of time.

[0082] "Voice data" refers to data used to record and transmit a user's speech or voice in digital form.

[0083] "Text data" refers to character information converted from voice data using voice recognition technology.

[0084] "Real-time" is a concept that refers to events and data processing occurring almost immediately.

[0085] "Means for analyzing text data" refers to technologies that use generative AI models and natural language processing techniques to understand the content and context of text data and identify intent.

[0086] "User intent" is the result of analyzing what the user is asking for through their statements and what information they need.

[0087] "Information retrieval means" refers to techniques for retrieving relevant information from internal databases and external sources based on the specified intent.

[0088] "Means for summarizing information" refers to techniques for extracting important points from acquired information and organizing them in a format that is easy for users to understand.

[0089] A "user terminal" (terminal) is a device, such as a computer or smartphone, that a user uses to conduct online meetings and display information.

[0090] "Speech recognition technology" is a technology for analyzing voice data and converting it into text information. It includes highly accurate voice recognition engines.

[0091] An "internal database" is a collection of data held within a company and is a searchable source of information.

[0092] "External sources" are sources other than internal databases, such as the Internet or other external data repositories.

[0093] The "means for converting into text data in real time" is a process for instantly converting acquired voice data into text data.

[0094] "Means for displaying on a user's screen" means a method for visually presenting information on the screen of a user's device, including formats such as pop-ups and sidebars.

[0095] "Generative AI models" refer to generative artificial intelligence such as GPT-4, which are used to analyze text data and identify user intent.

[0096] MODE FOR CARRYING OUT THE INVENTION

[0097] This invention is a system for users to acquire necessary information in real time during an online conference and visually display it. This system is mainly composed of a server and user terminals.

[0098] System Overview

[0099] User Device

[0100] The user terminal can be a device such as a PC or smartphone. The user terminal has the following functions:

[0101] Software for conducting online meetings (e.g., Zoom, Microsoft Teams)

[0102] Ability to capture audio data using built-in or external microphone

[0103] Communication function to send voice data to the server

[0104] A function to display information received from the server on the screen.

[0105] server

[0106] The server is a powerful computer that performs the following functions:

[0107] Receiving audio data

[0108] Parsing and converting voice data to text (using a speech recognition engine such as the Google Speech-to-Text API)

[0109] User intent analysis using generative AI models (e.g., GPT-4)

[0110] Information retrieval from internal databases and external sources

[0111] Summarizing the searched information and sending it to the user's device

[0112] Program processing explanation

[0113] Acquiring and sending audio data

[0114] When a user starts an online meeting, the device uses a microphone to capture audio data and transmits it to the server in packets at regular intervals, using the HTTP or WebSocket protocol.

[0115] Analysis of voice data and conversion to text

[0116] Once the server receives the voice data, it converts it into text using the Google Speech-to-Text API, which applies phonemic analysis, acoustic models, and language models.

[0117] User intent analysis

[0118] The server uses text generation AI (GPT-4) to analyze the text data and identify the user's intent. For example, it analyzes a statement such as "Please tell me more about this product" and identifies that the information being sought is product details.

[0119] Finding and summarizing relevant information

[0120] The server searches for information based on the specified intent from internal databases and external sources, then uses summarization tools to extract key points and present them in a format that is easy for the user to understand.

[0121] Sending and Displaying Information

[0122] The summarized information is then packetized and sent from the server to the user's device, which then displays the received information in a pop-up or sidebar format, allowing users to instantly check the information during the meeting.

[0123] Specific examples

[0124] Example 1: Providing product information

[0125] If a user says, "Tell me more about this product" during an online meeting, the device captures the speech and sends the audio data to the server. The server converts the audio data into text using the Google Speech-to-Text API and uses GPT-4 to determine that the user is requesting product details. The server then searches the company's database for product specifications and manuals and sends the summarized information to the user's device. The user's device then displays the summarized information as a pop-up on the screen.

[0126] Prompt Sentence Examples

[0127] Analyze audio data containing the phrase "Tell me more about this product" and use the following as a prompt to find information about the product details:

[0128] "Say: Tell me more about this product. Response: Please describe the specifications and key features of this product."

[0129] Example 2: Checking contract terms

[0130] If the user says, "Please tell me more about the terms of the contract," the device sends the voice data to the server, which converts the voice data into text using the Google Speech-to-Text API and analyzes the keyword "terms of contract" using GPT-4. Information about the identified terms of contract is retrieved from the database, summarized, sent to the user's device, and displayed on the screen.

[0131] Prompt Sentence Examples

[0132] Analyze audio data containing the phrase "Tell me more about the terms and conditions," and use the following as a prompt to find information about the terms and conditions:

[0133] "Statement: Please tell me more about the terms and conditions. Response: Please provide a summary of the key points of the applicable terms and conditions."

[0134] In this way, users can instantly obtain the information they need during an online meeting and conduct the meeting efficiently.

[0135] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0136] Step 1: User starts an online meeting

[0137] A user launches online conference software (e.g., Zoom, Microsoft Teams) on their device and starts a conference. This puts the device into a preparation state for the conference. The input is the user's voice, and the output is the device entering online conference mode.

[0138] Step 2: The device captures the audio data

[0139] The device uses a built-in or external microphone to capture the user's voice in real time, and stores the voice data in a buffer. The input is the user's voice, and the output is the stored voice data.

[0140] Step 3: The device sends the audio data to the server

[0141] The captured audio data is divided into packets at regular intervals and sent to the server using HTTP or WebSocket protocol. The input is the audio data stored in the buffer, and the output is the audio data packets sent to the server.

[0142] Step 4: The server receives the audio data

[0143] The server receives packets of voice data sent from the terminal and combines them into the original voice data. The input is the voice data packets and the output is the combined voice data.

[0144] Step 5: The server converts the audio data into text data.

[0145] The server uses speech recognition technology (e.g., Google Speech-to-Text API) to convert the combined voice data into text data in real time. This process involves phonemic analysis, acoustic models, and language models. The input is the combined voice data, and the output is the generated text data.

[0146] Step 6: The server analyzes the text data to determine the user's intent

[0147] The server uses a generative AI model (e.g., GPT-4) to analyze the generated text data and identify the user's intent. This analysis uses natural language processing techniques. The input is the generated text data, and the output is the identified user's intent.

[0148] Step 7: The server looks up the appropriate information

[0149] The server searches for the appropriate information from internal databases or external sources (e.g., internal company databases or external APIs) based on the identified user intent. It issues queries and retrieves the required data. The input is the identified user intent, and the output is the retrieved information.

[0150] Step 8: The server summarizes the retrieved information

[0151] The server uses a summarization tool to extract key points from the retrieved information and present it in a form that is easy for the user to understand. The input is the retrieved information and the output is the summarized information.

[0152] Step 9: The server sends the summarized information to the user terminal.

[0153] The summarized information is packetized and sent to the user terminal in the form of an HTTP POST request, etc. The input is the summarized information, and the output is the information packets sent to the user terminal.

[0154] Step 10: Your device will display the information

[0155] The user terminal analyzes the received information packets and displays them on the user screen in an appropriate format (e.g., a pop-up or sidebar). This allows users to check the necessary information during a meeting without changing their line of sight. The input is the received information packet, and the output is the information displayed on the screen.

[0156] (Application example 1)

[0157] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0158] In conventional online meetings, it was difficult for participants to instantly obtain the information they needed. It was also difficult for workers in factories to obtain information by voice without using their hands, or to give work instructions to robots. This created challenges for improving work efficiency and maintaining work accuracy.

[0159] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0160] In this invention, the server includes a means for converting voice data into text data in real time, a means for analyzing the text data to identify the user's intent, and a means for retrieving and summarizing appropriate information based on the identified intent, thereby enabling online conference participants and factory workers to immediately display appropriate information or perform work operations based on voice instructions.

[0161] An "online conference" is a conference system that allows multiple users in remote locations to communicate in real time using the Internet.

[0162] "Audio data" is digital data obtained by converting an audio signal acquired via an audio input device such as a microphone.

[0163] "Text data" is voice data that has been analyzed using voice recognition technology and its contents expressed as a string of characters (text).

[0164] "User intent identification" refers to technologies and algorithms that analyze text data and determine what it means and what the user wants.

[0165] "Means for retrieving appropriate information" refers to technology for retrieving relevant information from internal databases or external information sources based on the user's intent.

[0166] A "user terminal" is a device such as a computer or smartphone that a user uses to conduct an online conference.

[0167] "Means for displaying on the screen" refers to technology that displays the searched information on the screen of the user's device as a pop-up or notification so that the user can visually confirm the information.

[0168] "Factory work" refers to work carried out within a factory, such as manufacturing, assembling, and maintaining products.

[0169] "Voice instructions" are commands or requests given by a user to a system or robot through voice input.

[0170] "Analysis means" refers to the technology and algorithms used to process audio and text data and understand its content.

[0171] "Means for executing tasks" refers to the technology and algorithms that enable a robot to perform specific actions or tasks based on the user's intentions or voice instructions.

[0172] The present invention is a system that acquires and analyzes voice data during online meetings and factory work, and provides appropriate information. This system is composed of a user terminal, a server, and a robot.

[0173] System configuration

[0174] User Device

[0175] User terminals are devices used for online meetings and factory work. Examples include computers, smartphones, and tablets. User terminals are responsible for capturing voice data, sending data to the server, receiving data from the server, and displaying the data on the screen. A highly accurate voice recognition engine is used for voice recognition (e.g., Google Speech-to-Text API).

[0176] server

[0177] The server is a remote computer that receives voice data, analyzes it, converts it into text, identifies intent, searches for and summarizes information, and then transmits the data to the user's device. NLP technology (e.g., OpenAI's GPT-3) is used to convert voice data into text and analyze intent. It also summarizes search results and provides them to the user in an easy-to-understand format.

[0178] robot

[0179] A robot is a device that performs specific actions or tasks in a factory by receiving voice instructions. The robot uses a built-in microphone to capture the voice instructions and transmits them to a server.

[0180] System Operation

[0181] 1. Acquiring and sending audio data

[0182] When a user starts an online meeting or factory work, the user device or robot captures audio in real time using a built-in or external microphone, and the captured audio data is divided into packets and sent to the server.

[0183] 2. Analysis of voice data and conversion to text

[0184] The server analyzes the voice data received from the user's device or robot in real time and converts it into text data using voice recognition technology.

[0185] 3. User Intent Analysis

[0186] The server uses text generation AI to analyze the user's statements based on text data and identify the information or instructions the user is seeking. Natural language processing (NLP) technology is used for intent analysis.

[0187] 4. Finding and summarizing relevant information

[0188] Based on the identified intent, the server retrieves appropriate information from internal databases and external sources, summarizes the information, and presents it in a format that is easy for the user to understand.

[0189] 5. Transmission and Display of Information

[0190] The server sends the summarized information to the user's device or the robot. The user's device displays the received information in a pop-up format on the screen, and the robot provides the information through voice or a display.

[0191] Specific examples

[0192] Example 1: Providing product information

[0193] When a user says "Tell me more about this product" during an online meeting, the voice data is sent to the server for analysis. The server then searches for detailed product information and sends a summary to the user's device for display.

[0194] Example 2: Factory Work Order

[0195] When a worker in the factory issues a voice command such as "What is the next maintenance procedure for this machine?", the robot captures the voice and sends it to the server. The server searches for the maintenance procedure and sends summary information to the robot. The robot then provides the maintenance procedure by display or voice.

[0196] Example prompt sentence:

[0197] What is the next maintenance procedure?

[0198] Please explain how to replace the next filter.

[0199] Please display the relevant page in the manual.

[0200] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0201] Step 1: Acquire audio data

[0202] When a user starts an online meeting or factory work, the user terminal or robot uses an internal or external microphone to capture audio in real time. The audio data input is the user's speech, and the output is the captured audio data. This data is converted into a digital format and used for subsequent analysis. Specifically, the microphone captures the audio waveform and processes it as a digital signal.

[0203] Step 2: Sending audio data

[0204] The terminal divides the captured audio data into packets at regular intervals and sends them to the server. The input is the captured audio data, and the output is packetized audio data. Specifically, the audio data is sent to the server in real time via the network. This transmission uses UDP or TCP / IP.

[0205] Step 3: Analyze and convert audio data into text

[0206] The server analyzes the received voice data in real time and converts it into text data using speech recognition technology. The input is packetized voice data, and the output is text data. This conversion uses a highly accurate speech recognition engine (e.g., Google Speech-to-Text API). Specifically, the voice signal is matched with an acoustic model and converted into text.

[0207] Step 4: User Intent Analysis

[0208] The server uses a generative AI model (e.g., OpenAI's GPT-3) to analyze the text data, interpreting the user's speech and identifying their intent. The input is converted text data, and the output is data containing the user's intent. Specifically, the text data is analyzed using a natural language processing algorithm to identify the intent of the speech and the desired information.

[0209] Step 5: Finding and Summarizing Pertinent Information

[0210] Based on the identified intent, the server searches for relevant information from internal databases and external sources and generates a summary. The input is data containing the user's intent, and the output is summarized information. Specifically, a search query is sent to the database, and the retrieved information is summarized using natural language processing algorithms.

[0211] Step 6: Submit summary information

[0212] The server sends the summarized information to the user terminal or robot. The input is the summarized information, and the output is the transmitted information. Specifically, data packets are sent to the user terminal or robot via the network.

[0213] Step 7: Viewing information

[0214] The terminal or robot provides the received information to the user visually or audibly. The input is information sent from the server, and the output is a visual display or audio information provided to the user. Specific actions include a pop-up appearing on the terminal screen, or the robot giving instructions by voice.

[0215] Example prompt sentence:

[0216] What is the next maintenance procedure?

[0217] Please explain how to replace the next filter.

[0218] Please display the relevant page in the manual.

[0219] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0220] This system analyzes conversations in real time during online meetings, displays appropriate information only on the user's screen, and also recognizes the user's emotions and adjusts the information provided accordingly. This system consists of a server, a user terminal, and an emotion engine.

[0221] System configuration

[0222] 1. User Device

[0223] A user terminal is a device such as a computer or smartphone that has the functionality to conduct online meetings. The user terminal is responsible for capturing voice data, sending data to the server and emotion engine, receiving data from the server, and displaying the data on the screen.

[0224] 2. Server

[0225] It is a remote computer that receives voice and text data, analyzes it, analyzes emotions, identifies intent, retrieves and summarizes information, and transmits the data to a user terminal.

[0226] 3. Emotion Engine

[0227] This engine analyzes the user's speech and text data to recognize the user's emotions. The engine determines emotions from the tone, speed, pitch, and content of the speech.

[0228] Program processing explanation

[0229] 1. Acquiring and sending audio data

[0230] When a user starts an online conference, the user device captures the conference audio in real time using a built-in or external microphone. The captured audio data is divided into packets at regular intervals and sent to the server and emotion engine.

[0231] 2. Analysis of voice data and conversion to text

[0232] The server receives the voice data sent from the user's device and stores it in a buffer. It then converts the voice data into text data using voice recognition technology. This process uses a highly accurate voice recognition engine.

[0233] 3. User Intent Analysis

[0234] Using text generation AI, the server analyzes the user's statements based on text data and identifies the information and answers the user is looking for. Natural language processing (NLP) technology is used to analyze intent.

[0235] 4. Emotion analysis

[0236] The emotion engine analyzes the voice and text data sent from the user's device and recognizes the user's emotions using an algorithm that determines emotions from the tone, speed, pitch, and content of the speech.

[0237] 5. Finding and summarizing relevant information

[0238] The server searches for appropriate information from internal databases and external sources based on the identified intent and the results of sentiment analysis. The content and format of the information presented is adjusted according to the results of sentiment analysis. The retrieved information is summarized and formatted in a way that is easily understandable to the user.

[0239] 6. Transmission and Display of Information

[0240] The summarized information is sent from the server to the user's device. The user's device displays the received information on the screen in a pop-up format or other format, allowing the user to check the information without changing their line of sight during the meeting. Depending on the results of emotion analysis, if the user is feeling anxious, information with a more reassuring content and tone is provided.

[0241] Specific examples

[0242] Example 1: Providing product information

[0243] Imagine a user says, "Tell me more about this product" during an online meeting. The user's device captures this speech and sends the voice data to the server and emotion engine. The server analyzes the voice data and converts it into text data. The emotion engine analyzes the emotions from the user's speech, and if tension or impatience is detected, it selects materials that explain things in a gentle and easy-to-understand tone. The server searches the internal database for product specifications and manuals and summarizes the information it retrieves. This summary information is sent to the user's device and displayed on the screen.

[0244] Example 2: Checking contract terms

[0245] Consider a scenario where a user says, "Please tell me more about the terms of the contract." The user's device sends voice data to the server and emotion engine, which converts the voice data into text data. The text generation AI extracts the keyword "terms of contract," and the emotion engine analyzes whether the user is excited or anxious. If the user is anxious, detailed and transparent information is provided as a priority. The server retrieves the relevant contract terms document from the database and summarizes it. The summary information is sent to the user's device and displayed on the screen.

[0246] In this way, the present invention provides a system that allows users to instantly obtain necessary information during an online conference and provide appropriate answers and suggestions, and is also capable of providing information that takes into account the user's emotions.

[0247] The processing flow will be explained below.

[0248] Step 1:

[0249] When a user starts an online conference, the user device captures the conference audio in real time using a built-in or external microphone. The captured audio data is divided into packets at regular intervals and sent to the server and emotion engine.

[0250] Step 2:

[0251] The server receives the voice data sent from the user device and stores it in a buffer. At the same time, the emotion engine also receives the voice data and prepares it for analysis.

[0252] Step 3:

[0253] The server uses speech recognition technology to convert the voice data into text data, using a highly accurate speech recognition engine.

[0254] Step 4:

[0255] The server inputs the converted text data into a text generation AI (e.g., GPT-3) to analyze what the user said, thereby identifying the user's intent and the information they need.

[0256] Step 5:

[0257] The emotion engine analyzes the voice and text data sent from the user's device to recognize the user's emotions, taking into account voice tone, speed, pitch, and content.

[0258] Step 6:

[0259] The server combines the analysis results of the text generation AI with those of the emotion engine to comprehensively evaluate the user's intentions and emotions.

[0260] Step 7:

[0261] The server queries internal databases and external sources based on the overall evaluation to find the required information, and adjusts the content and format of the information provided depending on the results of the sentiment analysis.

[0262] Step 8:

[0263] The server analyzes the retrieved information and summarizes key points, formatting this summary to suit the user's emotional state (e.g., a polite, reassuring tone).

[0264] Step 9:

[0265] The server sends the summarized information to the user's device, along with instructions on how to display the information (e.g., popup, sidebar).

[0266] Step 10:

[0267] The user device analyzes the received information and displays it on the user's screen in the specified format, allowing the user to check the necessary information without changing their line of sight during the meeting.

[0268] This processing flow allows users to instantly obtain appropriate information during online meetings, and the information is provided in a manner that takes into consideration the user's feelings, allowing the meeting to proceed smoothly.

[0269] Example 2

[0270] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0271] Conventional online conference systems have difficulty understanding the context of conversations and providing appropriate information in real time. Furthermore, they do not provide information that takes users' emotions into consideration, which prevents them from increasing user satisfaction. Furthermore, they lack a means to efficiently aggregate data from multiple sources, making it difficult to provide necessary information immediately during a conference. There is a need to solve these issues and realize more effective information provision.

[0272] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0273] In this invention, the server includes means for converting user voice data into text data in real time, means for analyzing the text data to identify the user's intention, means for analyzing the user's emotions, means for searching for appropriate information based on the identified intention and the results of the emotion analysis, means for summarizing the searched information, and means for transmitting the summarized information to the user terminal and displaying it on the user's screen. This makes it possible to provide appropriate information immediately during a conference and to provide information that is tailored to the user's emotions.

[0274] An "online conference" is a method of holding a conference between multiple participants in remote locations by sharing audio and video in real time via the Internet.

[0275] "Voice data" refers to data in which the user's voice is recorded in digital format.

[0276] "Text data" is character string data generated by analyzing voice data.

[0277] "Analysis" is the act of analyzing data using data processing techniques to understand its meaning and intent.

[0278] "User intent" refers to the information, questions, or requests that a user is trying to convey through their speech.

[0279] "Emotions" are psychological states and feelings that can be inferred from a user's statements and actions.

[0280] "Emotion analysis" is a technique for determining a user's emotions based on voice tone, speed, pitch, and text content.

[0281] "Information retrieval" is the act of finding necessary information from databases or external sources.

[0282] "Summarizing" is the act of extracting important points and organizing information in a concise form.

[0283] A "user terminal" is a device used by a user, such as a computer or smartphone.

[0284] The present invention is a system that analyzes user conversation content in real time in an online conference system, recognizes the user's emotions, and provides appropriate information. This system is composed of a server, user terminals, and an emotion engine.

[0285] System Configuration

[0286] 1. User Device

[0287] The user terminal is a device such as a computer or a smartphone for conducting an online conference, which allows the user to start the conference, capture audio data, and transmit the necessary data to the server and the emotion engine.

[0288] 2. Server

[0289] The server is a remote computer responsible for receiving, analyzing, sentiment analysis, intent identification, information retrieval and summarization, and transmitting data to the user terminal.

[0290] 3. Emotion Engine

[0291] The emotion engine analyzes voice and text data to recognize the user's emotions. It determines the user's emotions from the voice tone, speed, pitch, and content of the speech.

[0292] Program processing explanation

[0293] 1. Acquiring and sending audio data

[0294] When a user starts an online conference, the user's device will capture the conference audio in real time using a built-in or external microphone. The captured audio data will be divided into packets and sent to the server and emotion engine. This involves encoding the audio data and sending an HTTP POST request.

[0295] 2. Analysis of voice data and conversion to text

[0296] The server stores the received voice data in a buffer and converts it into text data using speech recognition technology (e.g., Google Cloud Speech-to-Text), which is then stored in a data store.

[0297] 3. User Intent Analysis

[0298] The server uses a generative AI model (e.g., OpenAI GPT-4) to analyze the text data and understand what the user is saying. Specifically, it uses NLP techniques to identify the user's intent.

[0299] 4. Emotion analysis

[0300] The emotion engine uses software such as IBM Watson Tone Analyzer to analyze the tone, speed, pitch, and content of the voice and text data transmitted to identify the user's emotions.

[0301] 5. Finding and summarizing relevant information

[0302] The server searches for appropriate information from an internal database or external information sources based on the results of the identified intent and sentiment analysis. The retrieved information is then organized into an easy-to-understand format by a summarization engine. The summarized information is then sent from the server to the user's device.

[0303] 6. Transmission and Display of Information

[0304] The device displays the received information on the screen in a pop-up or slide-in format, allowing users to instantly check the information during a meeting, and the content and format of the information are adjusted according to the results of sentiment analysis.

[0305] Specific examples

[0306] 1. Providing product information

[0307] If a user says "Tell me more about this product" during an online meeting, the device captures the speech and sends the voice data to the server and emotion engine. The server converts the voice data into text data and analyzes it together with the emotion engine. If the user is nervous, the server selects materials explained in a gentle tone, sends them to the device, and displays them on the screen.

[0308] 2. Check the terms and conditions

[0309] If a user says, "Please tell me more about the terms of the contract," the device sends the voice data to the server and emotion engine, which converts the voice data into text. A text-generation AI is used to extract the keyword "terms of contract," and the emotion engine analyzes the user's emotions. If the user is anxious, a document containing a detailed explanation is selected, and the information is summarized and displayed on the device.

[0310] Prompt sentence examples

[0311] "Tell me more about this product"

[0312] "Please tell me more about the terms of the contract."

[0313] I'd like to know how to use this feature.

[0314] Please tell me the progress of the project.

[0315] As a result, the present invention provides a system that allows users to instantly obtain necessary information during an online conference and provide appropriate answers and suggestions, and also makes it possible to provide information that takes into account the user's emotions.

[0316] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0317] Step 1:

[0318] A user starts an online meeting.

[0319] Specific operation: A user starts a meeting using an online conferencing tool (e.g., Zoom or Teams). At this time, the device's microphone becomes active.

[0320] Input: User's speaking voice.

[0321] Output: Capture of conversational audio.

[0322] Step 2:

[0323] This is done by the terminal to capture voice data in real time and split it into packets.

[0324] Specific operation: The device captures audio data using a built-in or external microphone, and packets the audio data at regular intervals (e.g., every 10 seconds).

[0325] Input: The captured audio data.

[0326] Output: Packetized voice data.

[0327] Step 3:

[0328] The terminal transmits the captured voice data packets to the server and the emotion engine.

[0329] Specific operation: The device uses an HTTP POST request to send the captured audio data packets to the server and the emotion engine's API endpoint.

[0330] Input: Packetized voice data.

[0331] Output: Sending voice data to the server and emotion engine.

[0332] Step 4:

[0333] The server stores the received voice data in a buffer and converts it into text data using a voice recognition engine.

[0334] Specific operation: After the server receives the voice data, it stores it in a buffer and then converts it into text using voice recognition technology (e.g., Google Cloud Speech-to-Text).

[0335] Input: The audio data received by the server.

[0336] Output: The speech converted into text data.

[0337] Step 5:

[0338] The server uses a generative AI model to analyze the text data and identify the user's intent.

[0339] What it does: The server reads the stored text data, provides prompts to a generative AI model (e.g., OpenAI GPT-4), analyzes the content of the text data, and uses natural language processing (NLP) techniques to identify the user's intent.

[0340] Input: The converted text data.

[0341] Output: Identified user intent.

[0342] Step 6:

[0343] The emotion engine analyzes voice and text data to recognize the user's emotions.

[0344] What it does: The emotion engine uses algorithms (e.g., IBM Watson Tone Analyzer) to analyze the tone, speed, pitch, and content of speech based on the transmitted voice and text data to determine the user's emotions.

[0345] Input: Audio and text data.

[0346] Output: Perceived user emotion.

[0347] Step 7:

[0348] The server retrieves and summarizes appropriate information based on the intent and emotion information.

[0349] How it works: The server searches for relevant information from internal databases and external sources based on the results of intent analysis and sentiment analysis, then sends the search results to a summarization engine, which summarizes them in an easy-to-understand format.

[0350] Input: Intention and emotion information.

[0351] Output: Summarized information.

[0352] Step 8:

[0353] The server sends the summarized information to the terminal, which displays the information.

[0354] Specific operation: The server sends the summarized information to the device, the device decodes the received information, and the information is displayed in a pop-up or slide-in format in the user interface.

[0355] Input: Summarized information.

[0356] Output: Information displayed on the terminal.

[0357] This allows users to instantly obtain the information they need during a meeting and provides optimal support for what is being said.

[0358] (Application example 2)

[0359] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0360] In online meetings, it is not easy for users to instantly obtain the necessary information while concentrating on the conversation. Furthermore, since information is not provided taking into account the user's emotions, there is a problem that information is received uniformly. In the advertising industry in particular, it is important to provide information dynamically according to the client's reactions, but conventional systems are inadequate in this regard.

[0361] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0362] In this invention, the server includes means for acquiring voice data, means for converting the voice data into text data, means for analyzing the text data to identify the user's intention, means for transmitting the retrieved information to the user terminal and displaying it, means for recognizing the user's emotions based on the analyzed data, and means for dynamically adjusting the information according to the user's emotional state. This allows the user to instantly obtain the information they need while concentrating on the conversation, and makes it possible to provide information that takes their emotions into consideration.

[0363] "Audio data" refers to digital audio signals that record users' statements and conversations during online meetings.

[0364] "Text data" is voice data that has been analyzed and converted into a string of characters, and is a representation of what the user has said as text information.

[0365] "Intent" refers to the information or answer a user is seeking through their statement or question.

[0366] "Information retrieval means" refers to the techniques and methods used to find relevant information from internal databases and external sources based on the identified user intent.

[0367] "Emotion recognition means" refers to technologies or algorithms that identify a user's emotional state based on analyzed voice and text data.

[0368] "Dynamic adjustment means" refers to techniques or methods that change the content or format of the information provided in a timely manner in response to the recognized emotional state of the user.

[0369] A "server" is a remote computer system that performs processing such as analysis, retrieval, emotion recognition, and dynamic adjustment of voice and text data.

[0370] A "user terminal" is a device that conducts online conferences, acquires audio data, transmits and receives data to and from a server, and displays information on a screen.

[0371] The present invention relates to a system that enables users to efficiently collect information through online meetings and provides appropriate information based on their emotions. A specific embodiment of this system will be described below.

[0372] System Configuration

[0373] 1. User Device:

[0374] The user terminal is a device for conducting online meetings and acquiring audio data. This terminal can be a smartphone or a laptop. The user terminal captures audio data in real time using a built-in or external microphone and transmits the data to the server and emotion engine. It also has a display for displaying information.

[0375] 2. Server:

[0376] The server is a remote computer system that analyzes voice and text data and recognizes emotions. The server converts voice data into text using advanced speech recognition technologies (such as Google Speech-to-Text API or IBM Watson Speech to Text). It then uses natural language processing (NLP) technologies such as OpenAI GPT-4 to analyze user intent and perform information retrieval and summarization.

[0377] 3. Emotion Engine:

[0378] The emotion engine is an algorithm and technology that analyzes user speech and text data to recognize emotions. Sentiment analysis uses voice tone, speed, pitch, and facial expression analysis technologies (Amazon Rekognition and Google Cloud's emotion analysis API) to provide feedback based on the user's emotional state.

[0379] Program processing explanation

[0380] When a user starts an online meeting, the user's device captures audio in real time using a built-in microphone and sends the captured audio data to the server and emotion engine. The server converts the audio data into text using the Google Speech-to-Text API. Based on this text data, the user's intent is analyzed using OpenAI GPT-4.

[0381] Next, the emotion engine analyzes the voice and text data sent from the user device and recognizes the user's emotions using Amazon Rekognition or Google Cloud's emotion analysis API. After analyzing the user's intent and emotions, the server searches for and summarizes appropriate information from internal databases and external sources based on the identified intent and emotions.

[0382] The summarized information is sent from the server to the user's device, which then displays the received information on the screen.Specific examples and case studies will be presented, particularly in the advertising industry, where information is dynamically adjusted according to the client's emotional state during a presentation, providing a sense of security.

[0383] Examples of concrete examples and prompts

[0384] For example, if a client anxiously asks, "Can you give me some specific examples of success stories for this marketing strategy?", the system analyzes the voice and text data to recognize the anxious state. The server then searches an internal database for success stories and provides the information in a format that is easy for the user to understand.

[0385] Example prompt sentence:

[0386] User said: "Can you give me some concrete examples of success with this marketing strategy?"

[0387] Sentiment analysis results: "Users are feeling anxious"

[0388] Appropriate information: "Success stories, case studies, and easy-to-understand diagrams and charts"

[0389] Example of what the presenter sees:

[0390] Your client is feeling anxious. Please suggest the following success stories:

[0391] An example of a company's sales increase through a campaign

[0392] Industry Achievements and Statistics

[0393] Summary of customer satisfaction survey results

[0394] As described above, the present invention provides a system that enables users to efficiently collect information during an online conference and provide appropriate information based on emotions.

[0395] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0396] Step 1:

[0397] When an online conference starts, the user device captures voice data in real time using a built-in microphone. This voice data is sent as input to the server and emotion engine. Specifically, the online conference application acquires the voice signal from the microphone, divides it into data packets, and sends them at regular time intervals.

[0398] Step 2:

[0399] The server converts the voice data received from the user's device into text data using a highly accurate voice recognition engine (e.g., Google Speech-to-Text API, IBM Watson Speech to Text). This process outputs the voice data as text data. Specifically, it performs a spectral analysis of the voice signal, breaks it down into phonemes, and then reconstructs it into a word sequence.

[0400] Step 3:

[0401] The server uses natural language processing (NLP) technology such as OpenAI GPT-4 to analyze the user's intent using the generated text data. It understands the user's speech from the input text data, identifies their intent, and outputs that information. Specifically, it uses a word embedding model to understand the context and determine the information the user is looking for and the intent of their question.

[0402] Step 4:

[0403] The emotion engine analyzes the voice and text data sent from the server to recognize the user's emotions. It analyzes the input voice tone, speed, pitch, and content of speech and outputs the user's emotional state. Specifically, it uses a combination of facial expression recognition technology (e.g., Amazon Rekognition) and voice analysis algorithms (e.g., Google Cloud's emotion analysis API) to identify emotions.

[0404] Step 5:

[0405] The server searches for appropriate information from an internal database or external information sources based on the analysis results of the identified intent and emotion. The server matches the analysis results as input with the database, finds the most appropriate information, and outputs summarized information. Specifically, the server searches the database using pre-configured queries to extract highly relevant information.

[0406] Step 6:

[0407] The server searches for and sends the summarized information to the user's device. The user's device displays the received information in a pop-up format on the screen so that the user can easily check it during the meeting. The input summary information is then formatted into an appropriate format and visually presented to the user. Specifically, the information pop-up is displayed in a position that does not obstruct the user's line of sight, allowing the user to access detailed information as needed.

[0408] Through the above processing steps, the system allows users to obtain the information they need in real time during online meetings, and also to obtain information tailored to their emotions.

[0409] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0410] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0411] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0412] [Second embodiment]

[0413] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0414] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0415] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0416] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0417] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0418] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0419] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0420] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0421] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0422] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0423] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0424] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0425] This invention is a system that analyzes the content of online meetings in real time and displays appropriate information only on the user's screen. This system consists of a server and a user terminal.

[0426] System configuration

[0427] 1. User Device

[0428] A user terminal is a device such as a computer or smartphone that has the functionality to hold online meetings. The user terminal is responsible for capturing audio data, sending data to the server, receiving data from the server, and displaying the data on the screen.

[0429] 2. Server

[0430] It is a remote computer that receives voice data, analyzes it, converts it into text, identifies intent, retrieves and summarizes information, and transmits the data to a user terminal.

[0431] Program processing explanation

[0432] 1. Acquiring and sending audio data

[0433] When a user starts an online conference, the user's device will use a built-in or external microphone to capture the conference audio in real time, and the captured audio data will be divided into packets and sent to the server at regular intervals.

[0434] 2. Analysis of voice data and conversion to text

[0435] The server analyzes the voice data received from the user's device in real time and converts it into text data using voice recognition technology. This conversion is performed using a highly accurate voice recognition engine.

[0436] 3. User Intent Analysis

[0437] Using text generation AI, the server analyzes the user's statements based on text data and identifies the information and answers the user is looking for. Natural language processing (NLP) technology is used for intent analysis.

[0438] 4. Finding and summarizing relevant information

[0439] Based on the identified intent, the server searches for the appropriate information from internal databases and external sources, summarizes the information, and presents it in a format that is easily understood by the user. During the summarization, key points are extracted and bullet points, simple graphs, and illustrations are generated.

[0440] 5. Transmission and Display of Information

[0441] The server sends the summarized information to the user's device, which then displays it on the screen in a pop-up format, allowing the user to check the information without having to change their line of sight during the meeting.

[0442] Specific examples

[0443] Example 1: Providing product information

[0444] Imagine a user says, "Tell me more about this product" during an online meeting. The user's device captures this speech and sends the audio data to the server. The server analyzes the audio data and converts it into text data. If the text generation AI determines that "more product details are being sought," the server searches the company's internal database for product specifications and manuals. The server summarizes the information obtained and sends it to the user's device. The user's device then displays the summary information as a pop-up on the screen.

[0445] Example 2: Checking contract terms

[0446] Imagine a case where a user says, "Please tell me more about the terms of the contract." The user's device sends voice data to the server, which converts the voice data into text data. The text generation AI extracts the keyword "terms of contract," and the server retrieves the relevant contract terms document from the database and summarizes it. The summary information is sent to the user's device, which displays the information on the screen.

[0447] In this way, the present invention provides a system that allows users to instantly obtain necessary information during an online conference and provide appropriate answers and suggestions.

[0448] The processing flow will be explained below.

[0449] Step 1:

[0450] When a user starts an online conference, the user's device captures the conference audio in real time using a built-in or external microphone, and the captured audio data is divided into packets and sent to the server at regular intervals.

[0451] Step 2:

[0452] The server receives the voice data sent from the user terminal and stores it in a buffer.

[0453] Step 3:

[0454] The server uses speech recognition technology to convert the voice data into text data, using highly accurate speech recognition engines such as Google Speech-to-Text API and IBM Watson Speech to Text.

[0455] Step 4:

[0456] The server inputs the converted text data into a text generation AI (e.g., GPT-3) and analyzes what the user said.

[0457] Step 5:

[0458] The server extracts key keywords and questions from the results of the text generation AI analysis and identifies the user's intent, such as a specific request like "I want to know the price of a product."

[0459] Step 6:

[0460] Based on the user's identified intent, the server queries internal databases and external sources to find the required information.

[0461] Step 7:

[0462] The server analyzes the retrieved information and summarizes the key points, presenting them in a format that is easily understandable to the user (e.g., bullet points or key points).

[0463] Step 8:

[0464] The server sends the summarized information to the user's device, optionally specifying the display format (e.g., popup, sidebar).

[0465] Step 9:

[0466] The user device analyzes the received information and displays it on the user's screen in the specified format, allowing the user to check the information necessary for the conversation without having to change their line of sight during the meeting.

[0467] This processing flow allows users to instantly obtain the right information during online meetings and keep the conversation moving smoothly.

[0468] Example 1

[0469] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0470] Online meetings often present a challenge, making it difficult for users to quickly obtain the information they need. In particular, the lack of real-time information search and a summary of the information provided means users must interrupt the flow of the meeting to conduct research, reducing meeting efficiency. Furthermore, the accuracy and appropriateness of the information obtained can also be a problem.

[0471] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0472] In this invention, the server includes means for acquiring voice data, means for converting the acquired voice data into text data in real time, means for analyzing the text data and identifying the user's intention, means for searching for appropriate information based on the identified intention, means for summarizing the searched information, and means for transmitting the summarized information to a user terminal and displaying it on the user's screen, thereby enabling users to acquire necessary information in real time during an online conference and understand that information in a short amount of time.

[0473] "Voice data" refers to data used to record and transmit a user's speech or voice in digital form.

[0474] "Text data" refers to character information converted from voice data using voice recognition technology.

[0475] "Real-time" is a concept that refers to events and data processing occurring almost immediately.

[0476] "Means for analyzing text data" refers to technologies that use generative AI models and natural language processing techniques to understand the content and context of text data and identify intent.

[0477] "User intent" is the result of analyzing what the user is asking for through their statements and what information they need.

[0478] "Information retrieval means" refers to techniques for retrieving relevant information from internal databases and external sources based on the specified intent.

[0479] "Means for summarizing information" refers to techniques for extracting important points from acquired information and organizing them in a format that is easy for users to understand.

[0480] A "user terminal" (terminal) is a device, such as a computer or smartphone, that a user uses to conduct online meetings and display information.

[0481] "Speech recognition technology" is a technology for analyzing voice data and converting it into text information. It includes highly accurate voice recognition engines.

[0482] An "internal database" is a collection of data held within a company and is a searchable source of information.

[0483] "External sources" are sources other than internal databases, such as the Internet or other external data repositories.

[0484] The "means for converting into text data in real time" is a process for instantly converting acquired voice data into text data.

[0485] "Means for displaying on the user's screen" means a method for visually presenting information on the screen of the user's device, including formats such as pop-ups and sidebars.

[0486] "Generative AI models" refer to generative artificial intelligence such as GPT-4, which are used to analyze text data and identify user intent.

[0487] MODE FOR CARRYING OUT THE INVENTION

[0488] This invention is a system for users to acquire necessary information in real time during an online conference and visually display it. This system is mainly composed of a server and user terminals.

[0489] System Overview

[0490] User Device

[0491] The user terminal can be a device such as a PC or smartphone. The user terminal has the following functions:

[0492] Software for conducting online meetings (e.g., Zoom, Microsoft Teams)

[0493] Ability to capture audio data using built-in or external microphone

[0494] Communication function to send voice data to the server

[0495] A function to display information received from the server on the screen.

[0496] server

[0497] The server is a powerful computer that performs the following functions:

[0498] Receiving audio data

[0499] Parsing and converting voice data to text (using a speech recognition engine such as the Google Speech-to-Text API)

[0500] User intent analysis using generative AI models (e.g., GPT-4)

[0501] Information retrieval from internal databases and external sources

[0502] Summarizing the searched information and sending it to the user's device

[0503] Program processing explanation

[0504] Acquiring and sending audio data

[0505] When a user starts an online meeting, the device uses a microphone to capture audio data and transmits it to the server in packets at regular intervals, using the HTTP or WebSocket protocol.

[0506] Analysis of voice data and conversion to text

[0507] Once the server receives the voice data, it converts it into text using the Google Speech-to-Text API, which uses phonemic analysis, acoustic models, and language models.

[0508] User intent analysis

[0509] The server uses a text-generating AI (GPT-4) to analyze the text data and identify the user's intent. For example, it analyzes a statement such as "Please tell me more about this product" and identifies that the information being sought is product details.

[0510] Finding and summarizing relevant information

[0511] The server searches for information based on the specified intent from internal databases and external sources, then uses summarization tools to extract key points and present them in a format that is easy for the user to understand.

[0512] Sending and Displaying Information

[0513] The summarized information is then packetized and sent from the server to the user's device, which then displays the received information in a pop-up or sidebar format, allowing users to instantly check the information during the meeting.

[0514] Specific examples

[0515] Example 1: Providing product information

[0516] If a user says, "Tell me more about this product" during an online meeting, the device captures the speech and sends the audio data to the server. The server converts the audio data into text using the Google Speech-to-Text API and uses GPT-4 to determine that the user is requesting product details. The server then searches the company's database for product specifications and manuals and sends the summarized information to the user's device. The user's device then displays the summarized information as a pop-up on the screen.

[0517] Prompt Sentence Examples

[0518] Analyze audio data containing the phrase "Tell me more about this product" and use the following as a prompt to find information about the product details:

[0519] "Say: Tell me more about this product. Response: Please describe the specifications and key features of this product."

[0520] Example 2: Checking contract terms

[0521] If the user says, "Please tell me more about the terms of the contract," the device sends the voice data to the server, which converts the voice data into text using the Google Speech-to-Text API and analyzes the keyword "terms of contract" using GPT-4. Information about the identified terms of contract is retrieved from the database, summarized, sent to the user's device, and displayed on the screen.

[0522] Prompt Sentence Examples

[0523] Analyze audio data containing the phrase "Tell me more about the terms and conditions," and use the following as a prompt to find information about the terms and conditions:

[0524] "Statement: Please tell me more about the terms and conditions. Response: Please provide a summary of the key points of the applicable terms and conditions."

[0525] In this way, users can instantly obtain the information they need during an online meeting and conduct the meeting efficiently.

[0526] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0527] Step 1: User starts an online meeting

[0528] A user launches online conference software (e.g., Zoom, Microsoft Teams) on their device and starts a conference. This puts the device into a preparation state for the conference. The input is the user's voice, and the output is the device entering online conference mode.

[0529] Step 2: The device captures the audio data

[0530] The device uses a built-in or external microphone to capture the user's voice in real time, and stores the voice data in a buffer. The input is the user's voice, and the output is the stored voice data.

[0531] Step 3: The device sends the audio data to the server

[0532] The captured audio data is divided into packets at regular intervals and sent to the server using HTTP or WebSocket protocol. The input is the audio data stored in the buffer, and the output is the audio data packets sent to the server.

[0533] Step 4: The server receives the audio data

[0534] The server receives packets of voice data sent from the terminal and combines them into the original voice data. The input is the voice data packets and the output is the combined voice data.

[0535] Step 5: The server converts the audio data into text data.

[0536] The server uses speech recognition technology (e.g., Google Speech-to-Text API) to convert the combined voice data into text data in real time. This process involves phonemic analysis, acoustic models, and language models. The input is the combined voice data, and the output is the generated text data.

[0537] Step 6: The server analyzes the text data to determine the user's intent

[0538] The server uses a generative AI model (e.g., GPT-4) to analyze the generated text data and identify the user's intent. This analysis uses natural language processing techniques. The input is the generated text data, and the output is the identified user's intent.

[0539] Step 7: The server looks up the appropriate information

[0540] The server searches for the appropriate information from internal databases or external sources (e.g., internal company databases or external APIs) based on the identified user intent. It issues queries and retrieves the required data. The input is the identified user intent, and the output is the retrieved information.

[0541] Step 8: The server summarizes the retrieved information

[0542] The server uses a summarization tool to extract key points from the retrieved information and present it in a form that is easy for the user to understand. The input is the retrieved information and the output is the summarized information.

[0543] Step 9: The server sends the summarized information to the user terminal.

[0544] The summarized information is packetized and sent to the user terminal in the form of an HTTP POST request, etc. The input is the summarized information, and the output is the information packets sent to the user terminal.

[0545] Step 10: Your device will display the information

[0546] The user terminal analyzes the received information packets and displays them on the user screen in an appropriate format (e.g., a pop-up or sidebar). This allows users to check the necessary information during a meeting without changing their line of sight. The input is the received information packet, and the output is the information displayed on the screen.

[0547] (Application example 1)

[0548] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0549] In conventional online meetings, it was difficult for participants to instantly obtain the information they needed. It was also difficult for workers in factories to obtain information by voice without using their hands, or to give work instructions to robots. This created challenges for improving work efficiency and maintaining work accuracy.

[0550] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0551] In this invention, the server includes a means for converting voice data into text data in real time, a means for analyzing the text data to identify the user's intent, and a means for retrieving and summarizing appropriate information based on the identified intent, thereby enabling online conference participants and factory workers to immediately display appropriate information or perform work operations based on voice instructions.

[0552] An "online conference" is a conference system that allows multiple users in remote locations to communicate in real time using the Internet.

[0553] "Audio data" is a digital data obtained by converting an audio signal acquired via an audio input device such as a microphone.

[0554] "Text data" is voice data that has been analyzed using voice recognition technology and its contents expressed as a string of characters (text).

[0555] "User intent identification" refers to technologies and algorithms that analyze text data and determine what it means and what the user wants.

[0556] "Means for retrieving appropriate information" refers to technology for retrieving relevant information from internal databases or external information sources based on the user's intent.

[0557] A "user terminal" is a device such as a computer or smartphone that a user uses to conduct an online conference.

[0558] "Means for displaying on the screen" refers to technology that displays the searched information on the screen of the user's device as a pop-up or notification so that the user can visually confirm the information.

[0559] "Factory work" refers to work carried out within a factory, such as manufacturing, assembling, and maintaining products.

[0560] "Voice instructions" are commands or requests given by a user to a system or robot through voice input.

[0561] "Analysis means" refers to the technology and algorithms used to process audio and text data and understand its content.

[0562] "Means for executing tasks" refers to the technology and algorithms that enable a robot to perform specific actions or tasks based on the user's intentions or voice instructions.

[0563] The present invention is a system that acquires and analyzes voice data during online meetings and factory work, and provides appropriate information. This system is composed of a user terminal, a server, and a robot.

[0564] System configuration

[0565] User Device

[0566] User terminals are devices used for online meetings and factory work. Examples include computers, smartphones, and tablets. User terminals are responsible for capturing voice data, sending data to the server, receiving data from the server, and displaying the data on the screen. A highly accurate voice recognition engine is used for voice recognition (e.g., Google Speech-to-Text API).

[0567] server

[0568] The server is a remote computer that receives voice data, analyzes it, converts it into text, identifies intent, searches for and summarizes information, and then transmits the data to the user's device. NLP technology (e.g., OpenAI's GPT-3) is used to convert voice data into text and analyze intent. It also summarizes search results and provides them to the user in an easy-to-understand format.

[0569] robot

[0570] A robot is a device that performs specific actions or tasks in a factory by receiving voice instructions. The robot uses a built-in microphone to capture the voice instructions and transmits them to a server.

[0571] System Operation

[0572] 1. Acquiring and sending audio data

[0573] When a user starts an online meeting or factory work, the user device or robot captures audio in real time using a built-in or external microphone, and the captured audio data is divided into packets and sent to the server.

[0574] 2. Analysis of voice data and conversion to text

[0575] The server analyzes the voice data received from the user's device or robot in real time and converts it into text data using voice recognition technology.

[0576] 3. User Intent Analysis

[0577] The server uses text generation AI to analyze the user's statements based on text data and identify the information or instructions the user is seeking. Natural language processing (NLP) technology is used for intent analysis.

[0578] 4. Finding and summarizing relevant information

[0579] Based on the identified intent, the server retrieves appropriate information from internal databases and external sources, summarizes the information, and presents it in a format that is easy for the user to understand.

[0580] 5. Transmission and Display of Information

[0581] The server sends the summarized information to the user's device or the robot. The user's device displays the received information in a pop-up format on the screen, and the robot provides the information through voice or a display.

[0582] Specific examples

[0583] Example 1: Providing product information

[0584] When a user says "Tell me more about this product" during an online meeting, the voice data is sent to the server for analysis. The server then searches for detailed product information and sends a summary to the user's device for display.

[0585] Example 2: Factory Work Order

[0586] When a worker in the factory issues a voice command such as "What is the next maintenance procedure for this machine?", the robot captures the voice and sends it to the server. The server searches for the maintenance procedure and sends summary information to the robot. The robot then provides the maintenance procedure by display or voice.

[0587] Example prompt sentence:

[0588] What is the next maintenance procedure?

[0589] Please explain how to replace the following filters.

[0590] Please display the relevant page in the manual.

[0591] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0592] Step 1: Acquire audio data

[0593] When a user starts an online meeting or factory work, the user terminal or robot uses an internal or external microphone to capture audio in real time. The audio data input is the user's speech, and the output is the captured audio data. This data is converted into a digital format and used for subsequent analysis. Specifically, the microphone captures the audio waveform and processes it as a digital signal.

[0594] Step 2: Sending audio data

[0595] The terminal divides the captured audio data into packets at regular intervals and sends them to the server. The input is the captured audio data, and the output is packetized audio data. Specifically, the audio data is sent to the server in real time via the network. This transmission uses UDP or TCP / IP.

[0596] Step 3: Analyze and convert audio data into text

[0597] The server analyzes the received voice data in real time and converts it into text data using speech recognition technology. The input is packetized voice data, and the output is text data. This conversion uses a highly accurate speech recognition engine (e.g., Google Speech-to-Text API). Specifically, the voice signal is matched with an acoustic model and converted into text.

[0598] Step 4: User Intent Analysis

[0599] The server uses a generative AI model (e.g., OpenAI's GPT-3) to analyze the text data, interpreting the user's speech and identifying their intent. The input is converted text data, and the output is data containing the user's intent. Specifically, the text data is analyzed using a natural language processing algorithm to identify the intent of the speech and the desired information.

[0600] Step 5: Finding and Summarizing Pertinent Information

[0601] Based on the identified intent, the server searches for relevant information from internal databases and external sources and generates a summary. The input is data containing the user's intent, and the output is summarized information. Specifically, a search query is sent to the database, and the retrieved information is summarized using natural language processing algorithms.

[0602] Step 6: Submit summary information

[0603] The server sends the summarized information to the user terminal or robot. The input is the summarized information, and the output is the transmitted information. Specifically, data packets are sent to the user terminal or robot via the network.

[0604] Step 7: Viewing information

[0605] The terminal or robot provides the received information to the user visually or audibly. The input is information sent from the server, and the output is a visual display or audio information provided to the user. Specific actions include a pop-up appearing on the terminal screen, or the robot giving instructions by voice.

[0606] Example prompt sentence:

[0607] What is the next maintenance procedure?

[0608] Please explain how to replace the following filters.

[0609] Please display the relevant page in the manual.

[0610] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0611] This system analyzes conversations in real time during online meetings, displays appropriate information only on the user's screen, and also recognizes the user's emotions and adjusts the information provided accordingly. This system consists of a server, a user terminal, and an emotion engine.

[0612] System configuration

[0613] 1. User Device

[0614] A user terminal is a device such as a computer or smartphone that has the functionality to conduct online meetings. The user terminal is responsible for capturing voice data, sending data to the server and emotion engine, receiving data from the server, and displaying the data on the screen.

[0615] 2. Server

[0616] It is a remote computer that receives voice and text data, analyzes it, analyzes emotions, identifies intent, retrieves and summarizes information, and transmits the data to a user terminal.

[0617] 3. Emotion Engine

[0618] This engine analyzes the user's speech and text data to recognize the user's emotions. The engine determines emotions from the tone, speed, pitch, and content of the speech.

[0619] Program processing explanation

[0620] 1. Acquiring and sending audio data

[0621] When a user starts an online conference, the user device captures the conference audio in real time using a built-in or external microphone. The captured audio data is divided into packets at regular intervals and sent to the server and emotion engine.

[0622] 2. Analysis of voice data and conversion to text

[0623] The server receives the voice data sent from the user's device and stores it in a buffer. It then converts the voice data into text data using voice recognition technology. This process uses a highly accurate voice recognition engine.

[0624] 3. User Intent Analysis

[0625] Using text generation AI, the server analyzes the user's statements based on text data and identifies the information or answer the user is looking for. Natural language processing (NLP) technology is used for intent analysis.

[0626] 4. Emotion analysis

[0627] The emotion engine analyzes the voice and text data sent from the user's device and recognizes the user's emotions using an algorithm that determines emotions from the tone, speed, pitch, and content of the speech.

[0628] 5. Finding and summarizing relevant information

[0629] The server searches for appropriate information from internal databases and external sources based on the identified intent and the results of sentiment analysis, adjusts the content and format of the information presented depending on the results of sentiment analysis, and summarizes the information retrieved and presents it in a format that is easily understandable to the user.

[0630] 6. Transmission and Display of Information

[0631] The summarized information is sent from the server to the user's device. The user's device displays the received information on the screen in a pop-up format or other format, allowing the user to check the information without changing their line of sight during the meeting. Depending on the results of emotion analysis, if the user is feeling anxious, information with a more reassuring content and tone is provided.

[0632] Specific examples

[0633] Example 1: Providing product information

[0634] Imagine a user says, "Tell me more about this product" during an online meeting. The user's device captures this speech and sends the voice data to the server and emotion engine. The server analyzes the voice data and converts it into text data. The emotion engine analyzes the emotions from the user's speech, and if tension or impatience is detected, it selects materials that explain things in a gentle and easy-to-understand tone. The server searches the internal database for product specifications and manuals and summarizes the information it retrieves. This summary information is sent to the user's device and displayed on the screen.

[0635] Example 2: Checking contract terms

[0636] Consider a scenario where a user says, "Please tell me more about the terms of the contract." The user's device sends voice data to the server and emotion engine, which converts the voice data into text data. The text generation AI extracts the keyword "terms of contract," and the emotion engine analyzes whether the user is excited or anxious. If the user is anxious, detailed and transparent information is provided as a priority. The server retrieves the relevant contract terms document from the database and summarizes it. The summary information is sent to the user's device and displayed on the screen.

[0637] In this way, the present invention provides a system that allows users to instantly obtain necessary information during an online conference and provide appropriate answers and suggestions, and is also capable of providing information that takes into account the user's emotions.

[0638] The processing flow will be explained below.

[0639] Step 1:

[0640] When a user starts an online conference, the user device captures the conference audio in real time using a built-in or external microphone. The captured audio data is divided into packets at regular intervals and sent to the server and emotion engine.

[0641] Step 2:

[0642] The server receives the voice data sent from the user device and stores it in a buffer. At the same time, the emotion engine also receives the voice data and prepares it for analysis.

[0643] Step 3:

[0644] The server uses speech recognition technology to convert the voice data into text data, using a highly accurate speech recognition engine.

[0645] Step 4:

[0646] The server inputs the converted text data into a text generation AI (e.g., GPT-3) to analyze what the user said, thereby identifying the user's intent and the information they need.

[0647] Step 5:

[0648] The emotion engine analyzes the voice and text data sent from the user's device to recognize the user's emotions, taking into account voice tone, speed, pitch, and content.

[0649] Step 6:

[0650] The server combines the analysis results of the text generation AI with those of the emotion engine to comprehensively evaluate the user's intentions and emotions.

[0651] Step 7:

[0652] The server queries internal databases and external sources based on the overall evaluation to find the required information, and adjusts the content and format of the information provided depending on the results of the sentiment analysis.

[0653] Step 8:

[0654] The server analyzes the retrieved information and summarizes key points, formatting this summary to suit the user's emotional state (e.g., a polite, reassuring tone).

[0655] Step 9:

[0656] The server sends the summarized information to the user's device, along with instructions on how to display the information (e.g., popup, sidebar).

[0657] Step 10:

[0658] The user device analyzes the received information and displays it on the user's screen in the specified format, allowing the user to check the necessary information without changing their line of sight during the meeting.

[0659] This processing flow allows users to instantly obtain appropriate information during online meetings, and the information is provided in a manner that takes into consideration the user's feelings, allowing the meeting to proceed smoothly.

[0660] Example 2

[0661] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0662] Conventional online conference systems have difficulty understanding the context of conversations and providing appropriate information in real time. Furthermore, they do not provide information that takes users' emotions into consideration, which prevents them from increasing user satisfaction. Furthermore, they lack a means to efficiently aggregate data from multiple sources, making it difficult to provide necessary information immediately during a conference. There is a need to solve these issues and realize more effective information provision.

[0663] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0664] In this invention, the server includes means for converting user voice data into text data in real time, means for analyzing the text data to identify the user's intention, means for analyzing the user's emotions, means for searching for appropriate information based on the identified intention and the results of the emotion analysis, means for summarizing the searched information, and means for transmitting the summarized information to the user terminal and displaying it on the user's screen. This makes it possible to provide appropriate information immediately during a conference and to provide information that is tailored to the user's emotions.

[0665] An "online conference" is a method of holding a conference between multiple participants in remote locations by sharing audio and video in real time via the Internet.

[0666] "Voice data" refers to data in which the voice uttered by the user is recorded in digital format.

[0667] "Text data" is character string data generated by analyzing voice data.

[0668] "Analysis" is the act of analyzing data using data processing techniques to understand its meaning and intent.

[0669] "User intent" refers to the information, questions, or requests that a user is trying to convey through their speech.

[0670] "Emotions" are psychological states and feelings that can be inferred from a user's statements and actions.

[0671] "Emotion analysis" is a technique for determining a user's emotions based on voice tone, speed, pitch, and text content.

[0672] "Information retrieval" is the act of finding necessary information from databases or external sources.

[0673] "Summarizing" is the act of extracting important points and organizing information in a concise format.

[0674] A "user terminal" is a device used by a user, such as a computer or smartphone.

[0675] The present invention is a system that analyzes user conversation content in real time in an online conference system, recognizes the user's emotions, and provides appropriate information. This system is composed of a server, user terminals, and an emotion engine.

[0676] System Configuration

[0677] 1. User Device

[0678] The user terminal is a device such as a computer or a smartphone for conducting an online conference, which allows the user to start the conference, capture audio data, and transmit the necessary data to the server and the emotion engine.

[0679] 2. Server

[0680] The server is a remote computer responsible for receiving, analyzing, sentiment analysis, intent identification, information retrieval and summarization, and transmitting data to the user terminal.

[0681] 3. Emotion Engine

[0682] The emotion engine analyzes voice and text data to recognize the user's emotions. It determines the user's emotions from the voice tone, speed, pitch, and content of the speech.

[0683] Program processing explanation

[0684] 1. Acquiring and sending audio data

[0685] When a user starts an online conference, the user's device will capture the conference audio in real time using a built-in or external microphone. The captured audio data will be divided into packets and sent to the server and emotion engine. This involves encoding the audio data and sending an HTTP POST request.

[0686] 2. Analysis of voice data and conversion to text

[0687] The server stores the received voice data in a buffer and converts it into text data using speech recognition technology (e.g., Google Cloud Speech-to-Text), which is then stored in a data store.

[0688] 3. User Intent Analysis

[0689] The server uses a generative AI model (e.g., OpenAI GPT-4) to analyze the text data and understand what the user is saying. Specifically, it uses NLP techniques to identify the user's intent.

[0690] 4. Emotion analysis

[0691] The emotion engine uses software such as IBM Watson Tone Analyzer to analyze the tone, speed, pitch, and content of the voice and text data transmitted to identify the user's emotions.

[0692] 5. Finding and summarizing relevant information

[0693] The server searches for appropriate information from an internal database or external information sources based on the results of the identified intent and sentiment analysis. The retrieved information is then organized into an easy-to-understand format by a summarization engine. The summarized information is then sent from the server to the user's device.

[0694] 6. Transmission and Display of Information

[0695] The device displays the received information on the screen in a pop-up or slide-in format, allowing users to instantly check the information during a meeting, and the content and format of the information are adjusted according to the results of sentiment analysis.

[0696] Specific examples

[0697] 1. Providing product information

[0698] If a user says "Tell me more about this product" during an online meeting, the device captures the speech and sends the voice data to the server and emotion engine. The server converts the voice data into text data and analyzes it together with the emotion engine. If the user is nervous, the server selects materials explained in a gentle tone, sends them to the device, and displays them on the screen.

[0699] 2. Check the terms and conditions

[0700] If a user says, "Please tell me more about the terms of the contract," the device sends the voice data to the server and emotion engine, which converts the voice data into text. A text-generation AI is used to extract the keyword "terms of contract," and the emotion engine analyzes the user's emotions. If the user is anxious, a document containing a detailed explanation is selected, and the information is summarized and displayed on the device.

[0701] Prompt example

[0702] "Tell me more about this product"

[0703] "Please tell me more about the terms of the contract."

[0704] I'd like to know how to use this feature.

[0705] Please tell me the progress of the project.

[0706] As a result, the present invention provides a system that allows users to instantly obtain necessary information during an online conference and provide appropriate answers and suggestions, and also makes it possible to provide information that takes into account the user's emotions.

[0707] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0708] Step 1:

[0709] A user starts an online meeting.

[0710] Specific operation: A user starts a meeting using an online conferencing tool (e.g., Zoom or Teams). At this time, the device's microphone becomes active.

[0711] Input: User's speaking voice.

[0712] Output: Capture of conversational audio.

[0713] Step 2:

[0714] This is done by the terminal to capture voice data in real time and split it into packets.

[0715] Specific operation: The device captures audio data using a built-in or external microphone, and packets the audio data at regular intervals (e.g., every 10 seconds).

[0716] Input: The captured audio data.

[0717] Output: Packetized voice data.

[0718] Step 3:

[0719] The terminal transmits the captured voice data packets to the server and the emotion engine.

[0720] Specific operation: The device uses an HTTP POST request to send the captured audio data packets to the server and the emotion engine's API endpoint.

[0721] Input: Packetized voice data.

[0722] Output: Sending voice data to the server and emotion engine.

[0723] Step 4:

[0724] The server stores the received voice data in a buffer and converts it into text data using a voice recognition engine.

[0725] Specific operation: After the server receives the voice data, it stores it in a buffer and then converts it into text using voice recognition technology (e.g., Google Cloud Speech-to-Text).

[0726] Input: The audio data received by the server.

[0727] Output: The speech converted into text data.

[0728] Step 5:

[0729] The server uses a generative AI model to analyze the text data and identify the user's intent.

[0730] What it does: The server reads the stored text data, provides prompts to a generative AI model (e.g., OpenAI GPT-4), analyzes the content of the text data, and uses natural language processing (NLP) techniques to identify the user's intent.

[0731] Input: The converted text data.

[0732] Output: Identified user intent.

[0733] Step 6:

[0734] The emotion engine analyzes voice and text data to recognize the user's emotions.

[0735] What it does: The emotion engine uses algorithms (e.g., IBM Watson Tone Analyzer) to analyze the tone, speed, pitch, and content of speech based on the transmitted voice and text data to determine the user's emotions.

[0736] Input: Audio and text data.

[0737] Output: Perceived user emotion.

[0738] Step 7:

[0739] The server retrieves and summarizes appropriate information based on the intent and emotion information.

[0740] How it works: The server searches for relevant information from internal databases and external sources based on the results of intent analysis and sentiment analysis, then sends the search results to a summarization engine, which summarizes them in an easy-to-understand format.

[0741] Input: Intention and emotion information.

[0742] Output: Summarized information.

[0743] Step 8:

[0744] The server sends the summarized information to the terminal, which displays the information.

[0745] Specific operation: The server sends the summarized information to the device, the device decodes the received information, and the information is displayed in a pop-up or slide-in format in the user interface.

[0746] Input: Summarized information.

[0747] Output: Information displayed on the terminal.

[0748] This allows users to instantly obtain the information they need during a meeting and provides optimal support for what is being said.

[0749] (Application example 2)

[0750] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0751] In online meetings, it is difficult for users to instantly obtain the necessary information while concentrating on the conversation. Furthermore, since information is not provided taking into account the user's emotions, there is a problem that information is received uniformly. In the advertising industry in particular, it is important to provide information dynamically according to the client's reactions, but conventional systems are inadequate in this regard.

[0752] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0753] In this invention, the server includes means for acquiring voice data, means for converting the voice data into text data, means for analyzing the text data to identify the user's intention, means for transmitting the retrieved information to the user terminal and displaying it, means for recognizing the user's emotions based on the analyzed data, and means for dynamically adjusting the information according to the user's emotional state. This allows the user to instantly acquire the information they need while concentrating on the conversation, and makes it possible to provide information that takes their emotions into consideration.

[0754] "Audio data" refers to digital audio signals that record users' statements and conversations during online meetings.

[0755] "Text data" is voice data that has been analyzed and converted into a string of characters, and is a representation of what the user has said as text information.

[0756] "Intent" refers to the information or answer a user is seeking through their statement or question.

[0757] "Information retrieval means" refers to the technology or method used to find relevant information from internal databases or external sources based on the identified user intent.

[0758] "Emotion recognition means" refers to technologies or algorithms that identify a user's emotional state based on analyzed voice and text data.

[0759] "Dynamic adjustment means" refers to techniques or methods that change the content or format of the information provided in a timely manner in response to the recognized emotional state of the user.

[0760] A "server" is a remote computer system that performs processing such as analysis, retrieval, emotion recognition, and dynamic adjustment of voice and text data.

[0761] A "user terminal" is a device that conducts online conferences, acquires audio data, transmits and receives data to and from a server, and displays information on a screen.

[0762] The present invention relates to a system that enables users to efficiently collect information through online meetings and provides appropriate information based on their emotions. A specific embodiment of this system will be described below.

[0763] System Configuration

[0764] 1. User Device:

[0765] The user terminal is a device for conducting online meetings and acquiring audio data. This terminal can be a smartphone or a laptop. The user terminal captures audio data in real time using a built-in or external microphone and transmits the data to the server and emotion engine. It also has a display for displaying information.

[0766] 2. Server:

[0767] The server is a remote computer system that analyzes voice and text data and recognizes emotions. The server converts voice data into text using advanced speech recognition technologies (such as Google Speech-to-Text API or IBM Watson Speech to Text). It then uses natural language processing (NLP) technologies such as OpenAI GPT-4 to analyze user intent and perform information retrieval and summarization.

[0768] 3. Emotion Engine:

[0769] The emotion engine is an algorithm and technology that analyzes user speech and text data to recognize emotions. Sentiment analysis uses voice tone, speed, pitch, and facial expression analysis technologies (Amazon Rekognition and Google Cloud's emotion analysis API) to provide feedback based on the user's emotional state.

[0770] Program processing explanation

[0771] When a user starts an online meeting, the user's device captures audio in real time using a built-in microphone and sends the captured audio data to the server and emotion engine. The server converts the audio data into text using the Google Speech-to-Text API. Based on this text data, the user's intent is analyzed using OpenAI GPT-4.

[0772] Next, the emotion engine analyzes the voice and text data sent from the user device and recognizes the user's emotions using Amazon Rekognition or Google Cloud's emotion analysis API. After analyzing the user's intent and emotions, the server searches for and summarizes appropriate information from internal databases and external sources based on the identified intent and emotions.

[0773] The summarized information is sent from the server to the user's device, which then displays the received information on the screen.Specific examples and case studies will be presented, particularly in the advertising industry, where information is dynamically adjusted according to the client's emotional state during a presentation, providing a sense of security.

[0774] Examples of concrete examples and prompts

[0775] For example, if a client anxiously asks, "Can you give me some specific examples of success stories for this marketing strategy?", the system analyzes the voice and text data to recognize the anxious state. The server then searches an internal database for success stories and provides the information in a format that is easy for the user to understand.

[0776] Example prompt sentence:

[0777] User said: "Can you give me some concrete examples of success with this marketing strategy?"

[0778] Sentiment analysis results: "Users are feeling anxious"

[0779] Appropriate information: "Success stories, case studies, and easy-to-understand diagrams and charts"

[0780] Example of what the presenter sees:

[0781] Your client is feeling anxious. Please suggest the following success stories:

[0782] An example of a company's sales increase through a campaign

[0783] Industry Achievements and Statistics

[0784] Summary of customer satisfaction survey results

[0785] As described above, the present invention provides a system that enables users to efficiently collect information during an online conference and provide appropriate information based on emotions.

[0786] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0787] Step 1:

[0788] When an online conference starts, the user device captures voice data in real time using a built-in microphone. This voice data is sent as input to the server and emotion engine. Specifically, the online conference application acquires the voice signal from the microphone, divides it into data packets, and sends them at regular time intervals.

[0789] Step 2:

[0790] The server converts the voice data received from the user's device into text data using a highly accurate voice recognition engine (e.g., Google Speech-to-Text API, IBM Watson Speech to Text). This process outputs the voice data as text data. Specifically, it performs a spectral analysis of the voice signal, breaks it down into phonemes, and then reconstructs it into a word sequence.

[0791] Step 3:

[0792] The server uses natural language processing (NLP) technology such as OpenAI GPT-4 to analyze the user's intent using the generated text data. It understands the user's speech from the input text data, identifies their intent, and outputs that information. Specifically, it uses a word embedding model to understand the context and determine the information the user is looking for and the intent of their question.

[0793] Step 4:

[0794] The emotion engine analyzes the voice and text data sent from the server to recognize the user's emotions. It analyzes the input voice tone, speed, pitch, and content of speech and outputs the user's emotional state. Specifically, it uses a combination of facial expression recognition technology (e.g., Amazon Rekognition) and voice analysis algorithms (e.g., Google Cloud's emotion analysis API) to identify emotions.

[0795] Step 5:

[0796] The server searches for appropriate information from an internal database or external information sources based on the analysis results of the identified intent and emotion. The server matches the analysis results as input with the database, finds the most appropriate information, and outputs summarized information. Specifically, the server searches the database using pre-configured queries to extract highly relevant information.

[0797] Step 6:

[0798] The server searches for and sends the summarized information to the user's device. The user's device displays the received information in a pop-up format on the screen so that the user can easily check it during the meeting. The input summary information is then formatted into an appropriate format and visually presented to the user. Specifically, the information pop-up is displayed in a position that does not obstruct the user's line of sight, allowing the user to access detailed information as needed.

[0799] Through the above processing steps, the system allows users to obtain the information they need in real time during online meetings, and also to obtain information tailored to their emotions.

[0800] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0801] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0802] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0803] [Third embodiment]

[0804] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0805] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0806] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0807] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0808] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0809] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0810] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0811] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0812] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0813] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0814] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0815] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0816] This invention is a system that analyzes the content of online meetings in real time and displays appropriate information only on the user's screen. This system consists of a server and a user terminal.

[0817] System configuration

[0818] 1. User Device

[0819] A user terminal is a device such as a computer or smartphone that has the functionality to hold online meetings. The user terminal is responsible for capturing audio data, sending data to the server, receiving data from the server, and displaying the data on the screen.

[0820] 2. Server

[0821] It is a remote computer that receives voice data, analyzes it, converts it into text, identifies intent, retrieves and summarizes information, and transmits the data to a user terminal.

[0822] Program processing explanation

[0823] 1. Acquiring and sending audio data

[0824] When a user starts an online conference, the user's device will use a built-in or external microphone to capture the conference audio in real time, and the captured audio data will be divided into packets and sent to the server at regular intervals.

[0825] 2. Analysis of voice data and conversion to text

[0826] The server analyzes the voice data received from the user's device in real time and converts it into text data using voice recognition technology. This conversion is performed using a highly accurate voice recognition engine.

[0827] 3. User Intent Analysis

[0828] Using text generation AI, the server analyzes the user's statements based on text data and identifies the information and answers the user is looking for. Natural language processing (NLP) technology is used for intent analysis.

[0829] 4. Finding and summarizing relevant information

[0830] Based on the identified intent, the server searches for the appropriate information from internal databases and external sources, summarizes the information, and presents it in a format that is easily understood by the user. During the summarization, key points are extracted and bullet points, simple graphs, and illustrations are generated.

[0831] 5. Transmission and Display of Information

[0832] The server sends the summarized information to the user's device, which then displays it on the screen in a pop-up format, allowing the user to check the information without having to change their line of sight during the meeting.

[0833] Specific examples

[0834] Example 1: Providing product information

[0835] Imagine a user says, "Tell me more about this product" during an online meeting. The user's device captures this speech and sends the audio data to the server. The server analyzes the audio data and converts it into text data. If the text generation AI determines that "more product details are being sought," the server searches the company's internal database for product specifications and manuals. The server summarizes the information obtained and sends it to the user's device. The user's device then displays the summary information as a pop-up on the screen.

[0836] Example 2: Checking contract terms

[0837] Imagine a case where a user says, "Please tell me more about the terms of the contract." The user's device sends voice data to the server, which converts the voice data into text data. The text generation AI extracts the keyword "terms of contract," and the server retrieves the relevant contract terms document from the database and summarizes it. The summary information is sent to the user's device, which displays the information on the screen.

[0838] In this way, the present invention provides a system that allows users to instantly obtain necessary information during an online conference and provide appropriate answers and suggestions.

[0839] The processing flow will be explained below.

[0840] Step 1:

[0841] When a user starts an online conference, the user's device captures the conference audio in real time using a built-in or external microphone, and the captured audio data is divided into packets and sent to the server at regular intervals.

[0842] Step 2:

[0843] The server receives the voice data sent from the user terminal and stores it in a buffer.

[0844] Step 3:

[0845] The server uses speech recognition technology to convert the voice data into text data, using highly accurate speech recognition engines such as Google Speech-to-Text API and IBM Watson Speech to Text.

[0846] Step 4:

[0847] The server inputs the converted text data into a text generation AI (e.g., GPT-3) and analyzes what the user said.

[0848] Step 5:

[0849] The server extracts key keywords and questions from the results of the text generation AI analysis and identifies the user's intent, such as a specific request like "I want to know the price of a product."

[0850] Step 6:

[0851] Based on the user's identified intent, the server queries internal databases and external sources to find the required information.

[0852] Step 7:

[0853] The server analyzes the retrieved information and summarizes the key points, presenting them in a format that is easily understandable to the user (e.g., bullet points or key points).

[0854] Step 8:

[0855] The server sends the summarized information to the user's device, optionally specifying the display format (e.g., popup, sidebar).

[0856] Step 9:

[0857] The user device analyzes the received information and displays it on the user's screen in the specified format, allowing the user to check the information necessary for the conversation without having to change their line of sight during the meeting.

[0858] This processing flow allows users to instantly obtain the right information during online meetings and keep the conversation moving smoothly.

[0859] Example 1

[0860] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0861] Online meetings often present a challenge, making it difficult for users to quickly obtain the information they need. In particular, the lack of real-time information search and a summary of the information provided means users must interrupt the flow of the meeting to conduct research, reducing meeting efficiency. Furthermore, the accuracy and appropriateness of the information obtained can also be a problem.

[0862] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0863] In this invention, the server includes means for acquiring voice data, means for converting the acquired voice data into text data in real time, means for analyzing the text data and identifying the user's intention, means for searching for appropriate information based on the identified intention, means for summarizing the searched information, and means for transmitting the summarized information to a user terminal and displaying it on the user's screen, thereby enabling users to acquire necessary information in real time during an online conference and understand that information in a short amount of time.

[0864] "Voice data" refers to data used to record and transmit a user's speech or voice in digital form.

[0865] "Text data" refers to character information converted from voice data using voice recognition technology.

[0866] "Real-time" is a concept that refers to events and data processing occurring almost immediately.

[0867] "Means for analyzing text data" refers to technologies that use generative AI models and natural language processing techniques to understand the content and context of text data and identify intent.

[0868] "User intent" is the result of analyzing what the user is asking for through their statements and what information they need.

[0869] "Information retrieval means" refers to techniques for retrieving relevant information from internal databases and external sources based on the specified intent.

[0870] "Means for summarizing information" refers to techniques for extracting important points from acquired information and organizing them in a format that is easy for users to understand.

[0871] A "user terminal" (terminal) is a device, such as a computer or smartphone, that a user uses to conduct online meetings and display information.

[0872] "Speech recognition technology" is a technology for analyzing voice data and converting it into text information. It includes highly accurate voice recognition engines.

[0873] An "internal database" is a collection of data held within a company and is a searchable source of information.

[0874] "External sources" are sources other than internal databases, such as the Internet or other external data repositories.

[0875] The "means for converting into text data in real time" is a process for instantly converting acquired voice data into text data.

[0876] "Means for displaying on the user's screen" means a method for visually presenting information on the screen of the user's device, including formats such as pop-ups and sidebars.

[0877] "Generative AI models" refer to generative artificial intelligence such as GPT-4, which are used to analyze text data and identify user intent.

[0878] MODE FOR CARRYING OUT THE INVENTION

[0879] This invention is a system for users to acquire necessary information in real time during an online conference and visually display it. This system is mainly composed of a server and user terminals.

[0880] System Overview

[0881] User Device

[0882] The user terminal can be a device such as a PC or smartphone. The user terminal has the following functions:

[0883] Software for conducting online meetings (e.g., Zoom, Microsoft Teams)

[0884] Ability to capture audio data using built-in or external microphone

[0885] Communication function to send voice data to the server

[0886] A function to display information received from the server on the screen.

[0887] server

[0888] The server is a powerful computer that performs the following functions:

[0889] Receiving audio data

[0890] Parsing and converting voice data to text (using a speech recognition engine such as the Google Speech-to-Text API)

[0891] User intent analysis using generative AI models (e.g., GPT-4)

[0892] Information retrieval from internal databases and external sources

[0893] Summarizing the searched information and sending it to the user's device

[0894] Program processing explanation

[0895] Acquiring and sending audio data

[0896] When a user starts an online meeting, the device uses a microphone to capture audio data and transmits it to the server in packets at regular intervals, using the HTTP or WebSocket protocol.

[0897] Analysis of voice data and conversion to text

[0898] Once the server receives the voice data, it converts it into text using the Google Speech-to-Text API, which uses phonemic analysis, acoustic models, and language models.

[0899] User intent analysis

[0900] The server uses a text-generating AI (GPT-4) to analyze the text data and identify the user's intent. For example, it analyzes a statement such as "Please tell me more about this product" and identifies that the information being sought is product details.

[0901] Finding and summarizing relevant information

[0902] The server searches for information based on the specified intent from internal databases and external sources, then uses summarization tools to extract key points and present them in a format that is easy for the user to understand.

[0903] Sending and Displaying Information

[0904] The summarized information is then packetized and sent from the server to the user's device, which then displays the received information in a pop-up or sidebar format, allowing users to instantly check the information during the meeting.

[0905] Specific examples

[0906] Example 1: Providing product information

[0907] If a user says, "Tell me more about this product" during an online meeting, the device captures the speech and sends the audio data to the server. The server converts the audio data into text using the Google Speech-to-Text API and uses GPT-4 to determine that the user is requesting product details. The server then searches the company's database for product specifications and manuals and sends the summarized information to the user's device. The user's device then displays the summarized information as a pop-up on the screen.

[0908] Prompt Sentence Examples

[0909] Analyze audio data containing the phrase "Tell me more about this product" and use the following as a prompt to find information about the product details:

[0910] "Say: Tell me more about this product. Response: Please describe the specifications and key features of this product."

[0911] Example 2: Checking contract terms

[0912] If the user says, "Please tell me more about the terms of the contract," the device sends the voice data to the server, which converts the voice data into text using the Google Speech-to-Text API and analyzes the keyword "terms of contract" using GPT-4. Information about the identified terms of contract is retrieved from the database, summarized, sent to the user's device, and displayed on the screen.

[0913] Prompt Sentence Examples

[0914] Analyze audio data containing the phrase "Tell me more about the terms and conditions," and use the following as a prompt to find information about the terms and conditions:

[0915] "Statement: Please tell me more about the terms and conditions. Response: Please provide a summary of the key points of the applicable terms and conditions."

[0916] In this way, users can instantly obtain the information they need during an online meeting and conduct the meeting efficiently.

[0917] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0918] Step 1: User starts an online meeting

[0919] A user launches online conference software (e.g., Zoom, Microsoft Teams) on their device and starts a conference. This puts the device into a preparation state for the conference. The input is the user's voice, and the output is the device entering online conference mode.

[0920] Step 2: The device captures the audio data

[0921] The device uses a built-in or external microphone to capture the user's voice in real time, and stores the voice data in a buffer. The input is the user's voice, and the output is the stored voice data.

[0922] Step 3: The device sends the audio data to the server

[0923] The captured audio data is divided into packets at regular intervals and sent to the server using HTTP or WebSocket protocol. The input is the audio data stored in the buffer, and the output is the audio data packets sent to the server.

[0924] Step 4: The server receives the audio data

[0925] The server receives packets of voice data sent from the terminal and combines them into the original voice data. The input is the voice data packets and the output is the combined voice data.

[0926] Step 5: The server converts the audio data into text data.

[0927] The server uses speech recognition technology (e.g., Google Speech-to-Text API) to convert the combined voice data into text data in real time. This process involves phonemic analysis, acoustic models, and language models. The input is the combined voice data, and the output is the generated text data.

[0928] Step 6: The server analyzes the text data to determine the user's intent

[0929] The server uses a generative AI model (e.g., GPT-4) to analyze the generated text data and identify the user's intent. This analysis uses natural language processing techniques. The input is the generated text data, and the output is the identified user's intent.

[0930] Step 7: The server looks up the appropriate information

[0931] The server searches for the appropriate information from internal databases or external sources (e.g., internal company databases or external APIs) based on the identified user intent. It issues queries and retrieves the required data. The input is the identified user intent, and the output is the retrieved information.

[0932] Step 8: The server summarizes the retrieved information

[0933] The server uses a summarization tool to extract key points from the retrieved information and present it in a form that is easy for the user to understand. The input is the retrieved information and the output is the summarized information.

[0934] Step 9: The server sends the summarized information to the user terminal.

[0935] The summarized information is packetized and sent to the user terminal in the form of an HTTP POST request, etc. The input is the summarized information, and the output is the information packets sent to the user terminal.

[0936] Step 10: Your device will display the information

[0937] The user terminal analyzes the received information packets and displays them on the user screen in an appropriate format (e.g., a pop-up or sidebar). This allows users to check the necessary information during a meeting without changing their line of sight. The input is the received information packet, and the output is the information displayed on the screen.

[0938] (Application example 1)

[0939] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0940] In conventional online meetings, it was difficult for participants to instantly obtain the information they needed. It was also difficult for workers in factories to obtain information by voice without using their hands, or to give work instructions to robots. This created challenges for improving work efficiency and maintaining work accuracy.

[0941] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0942] In this invention, the server includes a means for converting voice data into text data in real time, a means for analyzing the text data to identify the user's intent, and a means for retrieving and summarizing appropriate information based on the identified intent, thereby enabling online conference participants and factory workers to immediately display appropriate information or perform work operations based on voice instructions.

[0943] An "online conference" is a conference system that allows multiple users in remote locations to communicate in real time using the Internet.

[0944] "Audio data" is a digital data obtained by converting an audio signal acquired via an audio input device such as a microphone.

[0945] "Text data" is voice data that has been analyzed using voice recognition technology and its contents expressed as a string of characters (text).

[0946] "User intent identification" refers to technologies and algorithms that analyze text data and determine what it means and what the user wants.

[0947] "Means for retrieving appropriate information" refers to technology for retrieving relevant information from internal databases or external information sources based on the user's intent.

[0948] A "user terminal" is a device such as a computer or smartphone that a user uses to conduct an online conference.

[0949] "Means for displaying on the screen" refers to technology that displays the searched information on the screen of the user's device as a pop-up or notification so that the user can visually confirm the information.

[0950] "Factory work" refers to work carried out within a factory, such as manufacturing, assembling, and maintaining products.

[0951] "Voice instructions" are commands or requests given by a user to a system or robot through voice input.

[0952] "Analysis means" refers to the technology and algorithms used to process audio and text data and understand its content.

[0953] "Means for executing tasks" refers to the technology and algorithms that enable a robot to perform specific actions or tasks based on the user's intentions or voice instructions.

[0954] The present invention is a system that acquires and analyzes voice data during online meetings and factory work, and provides appropriate information. This system is composed of a user terminal, a server, and a robot.

[0955] System configuration

[0956] User Device

[0957] User terminals are devices used for online meetings and factory work. Examples include computers, smartphones, and tablets. User terminals are responsible for capturing voice data, sending data to the server, receiving data from the server, and displaying the data on the screen. A highly accurate voice recognition engine is used for voice recognition (e.g., Google Speech-to-Text API).

[0958] server

[0959] The server is a remote computer that receives voice data, analyzes it, converts it into text, identifies intent, searches for and summarizes information, and then transmits the data to the user's device. NLP technology (e.g., OpenAI's GPT-3) is used to convert voice data into text and analyze intent. It also summarizes search results and provides them to the user in an easy-to-understand format.

[0960] robot

[0961] A robot is a device that performs specific actions or tasks in a factory by receiving voice instructions. The robot uses a built-in microphone to capture the voice instructions and transmits them to a server.

[0962] System Operation

[0963] 1. Acquiring and sending audio data

[0964] When a user starts an online meeting or factory work, the user device or robot captures audio in real time using a built-in or external microphone, and the captured audio data is divided into packets and sent to the server.

[0965] 2. Analysis of voice data and conversion to text

[0966] The server analyzes the voice data received from the user's device or robot in real time and converts it into text data using voice recognition technology.

[0967] 3. User Intent Analysis

[0968] The server uses text generation AI to analyze the user's statements based on text data and identify the information or instructions the user is seeking. Natural language processing (NLP) technology is used for intent analysis.

[0969] 4. Finding and summarizing relevant information

[0970] Based on the identified intent, the server retrieves appropriate information from internal databases and external sources, summarizes the information, and presents it in a format that is easy for the user to understand.

[0971] 5. Transmission and Display of Information

[0972] The server sends the summarized information to the user's device or the robot. The user's device displays the received information in a pop-up format on the screen, and the robot provides the information through voice or a display.

[0973] Specific examples

[0974] Example 1: Providing product information

[0975] When a user says "Tell me more about this product" during an online meeting, the voice data is sent to the server for analysis. The server then searches for detailed product information and sends a summary to the user's device for display.

[0976] Example 2: Factory Work Order

[0977] When a worker in the factory issues a voice command such as "What is the next maintenance procedure for this machine?", the robot captures the voice and sends it to the server. The server searches for the maintenance procedure and sends summary information to the robot. The robot then provides the maintenance procedure by display or voice.

[0978] Example prompt sentence:

[0979] What is the next maintenance procedure?

[0980] Please explain how to replace the following filters.

[0981] Please display the relevant page in the manual.

[0982] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0983] Step 1: Acquire audio data

[0984] When a user starts an online meeting or factory work, the user terminal or robot uses an internal or external microphone to capture audio in real time. The audio data input is the user's speech, and the output is the captured audio data. This data is converted into a digital format and used for subsequent analysis. Specifically, the microphone captures the audio waveform and processes it as a digital signal.

[0985] Step 2: Sending audio data

[0986] The terminal divides the captured audio data into packets at regular intervals and sends them to the server. The input is the captured audio data, and the output is packetized audio data. Specifically, the audio data is sent to the server in real time via the network. This transmission uses UDP or TCP / IP.

[0987] Step 3: Analyze and convert audio data into text

[0988] The server analyzes the received voice data in real time and converts it into text data using speech recognition technology. The input is packetized voice data, and the output is text data. This conversion uses a highly accurate speech recognition engine (e.g., Google Speech-to-Text API). Specifically, the voice signal is matched with an acoustic model and converted into text.

[0989] Step 4: User Intent Analysis

[0990] The server uses a generative AI model (e.g., OpenAI's GPT-3) to analyze the text data, interpreting the user's speech and identifying their intent. The input is converted text data, and the output is data containing the user's intent. Specifically, the text data is analyzed using a natural language processing algorithm to identify the intent of the speech and the desired information.

[0991] Step 5: Finding and Summarizing Pertinent Information

[0992] Based on the identified intent, the server searches for relevant information from internal databases and external sources and generates a summary. The input is data containing the user's intent, and the output is summarized information. Specifically, a search query is sent to the database, and the retrieved information is summarized using natural language processing algorithms.

[0993] Step 6: Submit summary information

[0994] The server sends the summarized information to the user terminal or robot. The input is the summarized information, and the output is the transmitted information. Specifically, data packets are sent to the user terminal or robot via the network.

[0995] Step 7: Viewing information

[0996] The terminal or robot provides the received information to the user visually or audibly. The input is information sent from the server, and the output is a visual display or audio information provided to the user. Specific actions include a pop-up appearing on the terminal screen, or the robot giving instructions by voice.

[0997] Example prompt sentence:

[0998] What is the next maintenance procedure?

[0999] Please explain how to replace the following filters.

[1000] Please display the relevant page in the manual.

[1001] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1002] This system analyzes conversations in real time during online meetings, displays appropriate information only on the user's screen, and also recognizes the user's emotions and adjusts the information provided accordingly. This system consists of a server, a user terminal, and an emotion engine.

[1003] System configuration

[1004] 1. User Device

[1005] A user terminal is a device such as a computer or smartphone that has the functionality to conduct online meetings. The user terminal is responsible for capturing voice data, sending data to the server and emotion engine, receiving data from the server, and displaying the data on the screen.

[1006] 2. Server

[1007] It is a remote computer that receives voice and text data, analyzes it, analyzes emotions, identifies intent, retrieves and summarizes information, and transmits the data to a user terminal.

[1008] 3. Emotion Engine

[1009] This engine analyzes the user's speech and text data to recognize the user's emotions. The engine determines emotions from the tone, speed, pitch, and content of the speech.

[1010] Program processing explanation

[1011] 1. Acquiring and sending audio data

[1012] When a user starts an online conference, the user device captures the conference audio in real time using a built-in or external microphone. The captured audio data is divided into packets at regular intervals and sent to the server and emotion engine.

[1013] 2. Analysis of voice data and conversion to text

[1014] The server receives the voice data sent from the user's device and stores it in a buffer. It then converts the voice data into text data using voice recognition technology. This process uses a highly accurate voice recognition engine.

[1015] 3. User Intent Analysis

[1016] Using text generation AI, the server analyzes the user's statements based on text data and identifies the information or answer the user is looking for. Natural language processing (NLP) technology is used for intent analysis.

[1017] 4. Emotion analysis

[1018] The emotion engine analyzes the voice and text data sent from the user's device and recognizes the user's emotions using an algorithm that determines emotions from the tone, speed, pitch, and content of the speech.

[1019] 5. Finding and summarizing relevant information

[1020] The server searches for appropriate information from internal databases and external sources based on the identified intent and the results of sentiment analysis, adjusts the content and format of the information presented depending on the results of sentiment analysis, and summarizes the information retrieved and presents it in a format that is easily understandable to the user.

[1021] 6. Transmission and Display of Information

[1022] The summarized information is sent from the server to the user's device. The user's device displays the received information on the screen in a pop-up format or other format, allowing the user to check the information without changing their line of sight during the meeting. Depending on the results of emotion analysis, if the user is feeling anxious, information with a more reassuring content and tone is provided.

[1023] Specific examples

[1024] Example 1: Providing product information

[1025] Imagine a user says, "Tell me more about this product" during an online meeting. The user's device captures this speech and sends the voice data to the server and emotion engine. The server analyzes the voice data and converts it into text data. The emotion engine analyzes the emotions from the user's speech, and if tension or impatience is detected, it selects materials that explain things in a gentle and easy-to-understand tone. The server searches the internal database for product specifications and manuals and summarizes the information it retrieves. This summary information is sent to the user's device and displayed on the screen.

[1026] Example 2: Checking contract terms

[1027] Consider a scenario where a user says, "Please tell me more about the terms of the contract." The user's device sends voice data to the server and emotion engine, which converts the voice data into text data. The text generation AI extracts the keyword "terms of contract," and the emotion engine analyzes whether the user is excited or anxious. If the user is anxious, detailed and transparent information is provided as a priority. The server retrieves the relevant contract terms document from the database and summarizes it. The summary information is sent to the user's device and displayed on the screen.

[1028] In this way, the present invention provides a system that allows users to instantly obtain necessary information during an online conference and provide appropriate answers and suggestions, and is also capable of providing information that takes into account the user's emotions.

[1029] The processing flow will be explained below.

[1030] Step 1:

[1031] When a user starts an online conference, the user device captures the conference audio in real time using a built-in or external microphone. The captured audio data is divided into packets at regular intervals and sent to the server and emotion engine.

[1032] Step 2:

[1033] The server receives the voice data sent from the user device and stores it in a buffer. At the same time, the emotion engine also receives the voice data and prepares it for analysis.

[1034] Step 3:

[1035] The server uses speech recognition technology to convert the voice data into text data, using a highly accurate speech recognition engine.

[1036] Step 4:

[1037] The server inputs the converted text data into a text generation AI (e.g., GPT-3) to analyze what the user said, thereby identifying the user's intent and the information they need.

[1038] Step 5:

[1039] The emotion engine analyzes the voice and text data sent from the user's device to recognize the user's emotions, taking into account voice tone, speed, pitch, and content.

[1040] Step 6:

[1041] The server combines the analysis results of the text generation AI with those of the emotion engine to comprehensively evaluate the user's intentions and emotions.

[1042] Step 7:

[1043] The server queries internal databases and external sources based on the overall evaluation to find the required information, and adjusts the content and format of the information provided depending on the results of the sentiment analysis.

[1044] Step 8:

[1045] The server analyzes the retrieved information and summarizes key points, formatting this summary to suit the user's emotional state (e.g., a polite, reassuring tone).

[1046] Step 9:

[1047] The server sends the summarized information to the user's device, along with instructions on how to display the information (e.g., popup, sidebar).

[1048] Step 10:

[1049] The user device analyzes the received information and displays it on the user's screen in the specified format, allowing the user to check the necessary information without changing their line of sight during the meeting.

[1050] This processing flow allows users to instantly obtain appropriate information during online meetings, and the information is provided in a manner that takes into consideration the user's feelings, allowing the meeting to proceed smoothly.

[1051] Example 2

[1052] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1053] Conventional online conference systems have difficulty understanding the context of conversations and providing appropriate information in real time. Furthermore, they do not provide information that takes users' emotions into consideration, which prevents them from increasing user satisfaction. Furthermore, they lack a means to efficiently aggregate data from multiple sources, making it difficult to provide necessary information immediately during a conference. There is a need to solve these issues and realize more effective information provision.

[1054] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1055] In this invention, the server includes means for converting user voice data into text data in real time, means for analyzing the text data to identify the user's intention, means for analyzing the user's emotions, means for searching for appropriate information based on the identified intention and the results of the emotion analysis, means for summarizing the searched information, and means for transmitting the summarized information to the user terminal and displaying it on the user's screen. This makes it possible to provide appropriate information immediately during a conference and to provide information that is tailored to the user's emotions.

[1056] An "online conference" is a method of holding a conference between multiple participants in remote locations by sharing audio and video in real time via the Internet.

[1057] "Voice data" refers to data in which the voice uttered by the user is recorded in digital format.

[1058] "Text data" is character string data generated by analyzing voice data.

[1059] "Analysis" is the act of analyzing data using data processing techniques to understand its meaning and intent.

[1060] "User intent" refers to the information, questions, or requests that a user is trying to convey through their speech.

[1061] "Emotions" are psychological states and feelings that can be inferred from a user's statements and actions.

[1062] "Emotion analysis" is a technique for determining a user's emotions based on voice tone, speed, pitch, and text content.

[1063] "Information retrieval" is the act of finding necessary information from databases or external sources.

[1064] "Summarizing" is the act of extracting important points and organizing information in a concise format.

[1065] A "user terminal" is a device used by a user, such as a computer or smartphone.

[1066] The present invention is a system that analyzes user conversation content in real time in an online conference system, recognizes the user's emotions, and provides appropriate information. This system is composed of a server, user terminals, and an emotion engine.

[1067] System Configuration

[1068] 1. User Device

[1069] The user terminal is a device such as a computer or a smartphone for conducting an online conference, which allows the user to start the conference, capture audio data, and transmit the necessary data to the server and the emotion engine.

[1070] 2. Server

[1071] The server is a remote computer responsible for receiving, analyzing, sentiment analysis, intent identification, information retrieval and summarization, and transmitting data to the user terminal.

[1072] 3. Emotion Engine

[1073] The emotion engine analyzes voice and text data to recognize the user's emotions. It determines the user's emotions from the voice tone, speed, pitch, and content of the speech.

[1074] Program processing explanation

[1075] 1. Acquiring and sending audio data

[1076] When a user starts an online conference, the user's device will capture the conference audio in real time using a built-in or external microphone. The captured audio data will be divided into packets and sent to the server and emotion engine. This involves encoding the audio data and sending an HTTP POST request.

[1077] 2. Analysis of voice data and conversion to text

[1078] The server stores the received voice data in a buffer and converts it into text data using speech recognition technology (e.g., Google Cloud Speech-to-Text), which is then stored in a data store.

[1079] 3. User Intent Analysis

[1080] The server uses a generative AI model (e.g., OpenAI GPT-4) to analyze the text data and understand what the user is saying. Specifically, it uses NLP techniques to identify the user's intent.

[1081] 4. Emotion analysis

[1082] The emotion engine uses software such as IBM Watson Tone Analyzer to analyze the tone, speed, pitch, and content of the voice and text data transmitted to identify the user's emotions.

[1083] 5. Finding and summarizing relevant information

[1084] The server searches for appropriate information from an internal database or external information sources based on the results of the identified intent and sentiment analysis. The retrieved information is then organized into an easy-to-understand format by a summarization engine. The summarized information is then sent from the server to the user's device.

[1085] 6. Transmission and Display of Information

[1086] The device displays the received information on the screen in a pop-up or slide-in format, allowing users to instantly check the information during a meeting, and the content and format of the information are adjusted according to the results of sentiment analysis.

[1087] Specific examples

[1088] 1. Providing product information

[1089] If a user says "Tell me more about this product" during an online meeting, the device captures the speech and sends the voice data to the server and emotion engine. The server converts the voice data into text data and analyzes it together with the emotion engine. If the user is nervous, the server selects materials explained in a gentle tone, sends them to the device, and displays them on the screen.

[1090] 2. Check the terms and conditions

[1091] If a user says, "Please tell me more about the terms of the contract," the device sends the voice data to the server and emotion engine, which converts the voice data into text. A text-generation AI is used to extract the keyword "terms of contract," and the emotion engine analyzes the user's emotions. If the user is anxious, a document containing a detailed explanation is selected, and the information is summarized and displayed on the device.

[1092] Prompt example

[1093] "Tell me more about this product"

[1094] "Please tell me more about the terms of the contract."

[1095] I'd like to know how to use this feature.

[1096] Please tell me the progress of the project.

[1097] As a result, the present invention provides a system that allows users to instantly obtain necessary information during an online conference and provide appropriate answers and suggestions, and also makes it possible to provide information that takes into account the user's emotions.

[1098] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1099] Step 1:

[1100] A user starts an online meeting.

[1101] Specific operation: A user starts a meeting using an online conferencing tool (e.g., Zoom or Teams). At this time, the device's microphone becomes active.

[1102] Input: User's speaking voice.

[1103] Output: Capture of conversational audio.

[1104] Step 2:

[1105] This is done by the terminal to capture voice data in real time and split it into packets.

[1106] Specific operation: The device captures audio data using a built-in or external microphone, and packets the audio data at regular intervals (e.g., every 10 seconds).

[1107] Input: The captured audio data.

[1108] Output: Packetized voice data.

[1109] Step 3:

[1110] The terminal transmits the captured voice data packets to the server and the emotion engine.

[1111] Specific operation: The device uses an HTTP POST request to send the captured audio data packets to the server and the emotion engine's API endpoint.

[1112] Input: Packetized voice data.

[1113] Output: Sending voice data to the server and emotion engine.

[1114] Step 4:

[1115] The server stores the received voice data in a buffer and converts it into text data using a voice recognition engine.

[1116] Specific operation: After the server receives the voice data, it stores it in a buffer and then converts it into text using voice recognition technology (e.g., Google Cloud Speech-to-Text).

[1117] Input: The audio data received by the server.

[1118] Output: The speech converted into text data.

[1119] Step 5:

[1120] The server uses a generative AI model to analyze the text data and identify the user's intent.

[1121] What it does: The server reads the stored text data, provides prompts to a generative AI model (e.g., OpenAI GPT-4), analyzes the content of the text data, and uses natural language processing (NLP) techniques to identify the user's intent.

[1122] Input: The converted text data.

[1123] Output: Identified user intent.

[1124] Step 6:

[1125] The emotion engine analyzes voice and text data to recognize the user's emotions.

[1126] What it does: The emotion engine uses algorithms (e.g., IBM Watson Tone Analyzer) to analyze the tone, speed, pitch, and content of speech based on the transmitted voice and text data to determine the user's emotions.

[1127] Input: Audio and text data.

[1128] Output: Perceived user emotion.

[1129] Step 7:

[1130] The server retrieves and summarizes appropriate information based on the intent and emotion information.

[1131] How it works: The server searches for relevant information from internal databases and external sources based on the results of intent analysis and sentiment analysis, then sends the search results to a summarization engine, which summarizes them in an easy-to-understand format.

[1132] Input: Intention and emotion information.

[1133] Output: Summarized information.

[1134] Step 8:

[1135] The server sends the summarized information to the terminal, which displays the information.

[1136] Specific operation: The server sends the summarized information to the device, the device decodes the received information, and the information is displayed in a pop-up or slide-in format in the user interface.

[1137] Input: Summarized information.

[1138] Output: Information displayed on the terminal.

[1139] This allows users to instantly obtain the information they need during a meeting and provides optimal support for what is being said.

[1140] (Application example 2)

[1141] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1142] In online meetings, it is difficult for users to instantly obtain the necessary information while concentrating on the conversation. Furthermore, since information is not provided taking into account the user's emotions, there is a problem that information is received uniformly. In the advertising industry in particular, it is important to provide information dynamically according to the client's reactions, but conventional systems are inadequate in this regard.

[1143] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1144] In this invention, the server includes means for acquiring voice data, means for converting the voice data into text data, means for analyzing the text data to identify the user's intention, means for transmitting the retrieved information to the user terminal and displaying it, means for recognizing the user's emotions based on the analyzed data, and means for dynamically adjusting the information according to the user's emotional state. This allows the user to instantly acquire the information they need while concentrating on the conversation, and makes it possible to provide information that takes their emotions into consideration.

[1145] "Audio data" refers to digital audio signals that record users' statements and conversations during online meetings.

[1146] "Text data" is voice data that has been analyzed and converted into a string of characters, and is a representation of what the user has said as text information.

[1147] "Intent" refers to the information or answer a user is seeking through their statement or question.

[1148] "Information retrieval means" refers to the technology or method used to find relevant information from internal databases or external sources based on the identified user intent.

[1149] "Emotion recognition means" refers to technologies or algorithms that identify a user's emotional state based on analyzed voice and text data.

[1150] "Dynamic adjustment means" refers to techniques or methods that change the content or format of the information provided in a timely manner in response to the recognized emotional state of the user.

[1151] A "server" is a remote computer system that performs processing such as analysis, retrieval, emotion recognition, and dynamic adjustment of voice and text data.

[1152] A "user terminal" is a device that conducts online conferences, acquires audio data, transmits and receives data to and from a server, and displays information on a screen.

[1153] The present invention relates to a system that enables users to efficiently collect information through online meetings and provides appropriate information based on their emotions. A specific embodiment of this system will be described below.

[1154] System Configuration

[1155] 1. User Device:

[1156] The user terminal is a device for conducting online meetings and acquiring audio data. This terminal can be a smartphone or a laptop. The user terminal captures audio data in real time using a built-in or external microphone and transmits the data to the server and emotion engine. It also has a display for displaying information.

[1157] 2. Server:

[1158] The server is a remote computer system that analyzes voice and text data and recognizes emotions. The server converts voice data into text using advanced speech recognition technologies (such as Google Speech-to-Text API or IBM Watson Speech to Text). It then uses natural language processing (NLP) technologies such as OpenAI GPT-4 to analyze user intent and perform information retrieval and summarization.

[1159] 3. Emotion Engine:

[1160] The emotion engine is an algorithm and technology that analyzes user speech and text data to recognize emotions. Sentiment analysis uses voice tone, speed, pitch, and facial expression analysis technologies (Amazon Rekognition and Google Cloud's emotion analysis API) to provide feedback based on the user's emotional state.

[1161] Program processing explanation

[1162] When a user starts an online meeting, the user's device captures audio in real time using a built-in microphone and sends the captured audio data to the server and emotion engine. The server converts the audio data into text using the Google Speech-to-Text API. Based on this text data, the user's intent is analyzed using OpenAI GPT-4.

[1163] Next, the emotion engine analyzes the voice and text data sent from the user device and recognizes the user's emotions using Amazon Rekognition or Google Cloud's emotion analysis API. After analyzing the user's intent and emotions, the server searches for and summarizes appropriate information from internal databases and external sources based on the identified intent and emotions.

[1164] The summarized information is sent from the server to the user's device, which then displays the received information on the screen.Specific examples and case studies will be presented, particularly in the advertising industry, where information is dynamically adjusted according to the client's emotional state during a presentation, providing a sense of security.

[1165] Examples of concrete examples and prompts

[1166] For example, if a client anxiously asks, "Can you give me some specific examples of success stories for this marketing strategy?", the system analyzes the voice and text data to recognize the anxious state. The server then searches an internal database for success stories and provides the information in a format that is easy for the user to understand.

[1167] Example prompt sentence:

[1168] User said: "Can you give me some concrete examples of success with this marketing strategy?"

[1169] Sentiment analysis results: "Users are feeling anxious"

[1170] Appropriate information: "Success stories, case studies, and easy-to-understand diagrams and charts"

[1171] Example of what the presenter sees:

[1172] Your client is feeling anxious. Please suggest the following success stories:

[1173] An example of a company's sales increase through a campaign

[1174] Industry Achievements and Statistics

[1175] Summary of customer satisfaction survey results

[1176] As described above, the present invention provides a system that enables users to efficiently collect information during an online conference and provide appropriate information based on emotions.

[1177] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1178] Step 1:

[1179] When an online conference starts, the user device captures voice data in real time using a built-in microphone. This voice data is sent as input to the server and emotion engine. Specifically, the online conference application acquires the voice signal from the microphone, divides it into data packets, and sends them at regular time intervals.

[1180] Step 2:

[1181] The server converts the voice data received from the user's device into text data using a highly accurate voice recognition engine (e.g., Google Speech-to-Text API, IBM Watson Speech to Text). This process outputs the voice data as text data. Specifically, it performs a spectral analysis of the voice signal, breaks it down into phonemes, and then reconstructs it into a word sequence.

[1182] Step 3:

[1183] The server uses natural language processing (NLP) technology such as OpenAI GPT-4 to analyze the user's intent using the generated text data. It understands the user's speech from the input text data, identifies their intent, and outputs that information. Specifically, it uses a word embedding model to understand the context and determine the information the user is looking for and the intent of their question.

[1184] Step 4:

[1185] The emotion engine analyzes the voice and text data sent from the server to recognize the user's emotions. It analyzes the input voice tone, speed, pitch, and content of speech and outputs the user's emotional state. Specifically, it uses a combination of facial expression recognition technology (e.g., Amazon Rekognition) and voice analysis algorithms (e.g., Google Cloud's emotion analysis API) to identify emotions.

[1186] Step 5:

[1187] The server searches for appropriate information from an internal database or external information sources based on the analysis results of the identified intent and emotion. The server matches the analysis results as input with the database, finds the most appropriate information, and outputs summarized information. Specifically, the server searches the database using pre-configured queries to extract highly relevant information.

[1188] Step 6:

[1189] The server searches for and sends the summarized information to the user's device. The user's device displays the received information in a pop-up format on the screen so that the user can easily check it during the meeting. The input summary information is then formatted into an appropriate format and visually presented to the user. Specifically, the information pop-up is displayed in a position that does not obstruct the user's line of sight, allowing the user to access detailed information as needed.

[1190] Through the above processing steps, the system allows users to obtain the information they need in real time during online meetings, and also to obtain information tailored to their emotions.

[1191] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1192] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1193] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1194] [Fourth embodiment]

[1195] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1196] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1197] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1198] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1199] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1200] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1201] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1202] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1203] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1204] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1205] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1206] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1207] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1208] This invention is a system that analyzes the content of online meetings in real time and displays appropriate information only on the user's screen. This system consists of a server and a user terminal.

[1209] System configuration

[1210] 1. User Device

[1211] A user terminal is a device such as a computer or smartphone that has the functionality to hold online meetings. The user terminal is responsible for capturing audio data, sending data to a server, receiving data from the server, and displaying the data on the screen.

[1212] 2. Server

[1213] It is a remote computer that receives voice data, analyzes it, converts it into text, identifies intent, retrieves and summarizes information, and transmits the data to a user terminal.

[1214] Program processing explanation

[1215] 1. Acquiring and sending audio data

[1216] When a user starts an online conference, the user's device will use a built-in or external microphone to capture the conference audio in real time, and the captured audio data will be divided into packets and sent to the server at regular intervals.

[1217] 2. Analysis of voice data and conversion to text

[1218] The server analyzes the voice data received from the user's device in real time and converts it into text data using voice recognition technology. This conversion is performed using a highly accurate voice recognition engine.

[1219] 3. User Intent Analysis

[1220] Using text generation AI, the server analyzes the user's statements based on text data and identifies the information and answers the user is looking for. Natural language processing (NLP) technology is used for intent analysis.

[1221] 4. Finding and summarizing relevant information

[1222] Based on the identified intent, the server searches for the appropriate information from internal databases and external sources, summarizes the information, and presents it in a format that is easily understood by the user. During the summarization, key points are extracted and bullet points, simple graphs, and illustrations are generated.

[1223] 5. Transmission and Display of Information

[1224] The server sends the summarized information to the user's device, which then displays it on the screen in a pop-up format, allowing the user to check the information without having to change their line of sight during the meeting.

[1225] Specific examples

[1226] Example 1: Providing product information

[1227] Imagine a user says, "Tell me more about this product" during an online meeting. The user's device captures this speech and sends the audio data to the server. The server analyzes the audio data and converts it into text data. If the text generation AI determines that "more product details are being sought," the server searches the company's internal database for product specifications and manuals. The server summarizes the information obtained and sends it to the user's device. The user's device then displays the summary information as a pop-up on the screen.

[1228] Example 2: Checking contract terms

[1229] Imagine a case where a user says, "Please tell me more about the terms of the contract." The user's device sends voice data to the server, which converts the voice data into text data. The text generation AI extracts the keyword "terms of contract," and the server retrieves the relevant contract terms document from the database and summarizes it. The summary information is sent to the user's device, which displays the information on the screen.

[1230] In this way, the present invention provides a system that allows users to instantly obtain necessary information during an online conference and provide appropriate answers and suggestions.

[1231] The processing flow will be explained below.

[1232] Step 1:

[1233] When a user starts an online conference, the user's device captures the conference audio in real time using a built-in or external microphone, and the captured audio data is divided into packets and sent to the server at regular intervals.

[1234] Step 2:

[1235] The server receives the voice data sent from the user terminal and stores it in a buffer.

[1236] Step 3:

[1237] The server uses speech recognition technology to convert the voice data into text data, using highly accurate speech recognition engines such as Google Speech-to-Text API and IBM Watson Speech to Text.

[1238] Step 4:

[1239] The server inputs the converted text data into a text generation AI (e.g., GPT-3) and analyzes what the user said.

[1240] Step 5:

[1241] The server extracts key keywords and questions from the results of the text generation AI analysis and identifies the user's intent, such as a specific request like "I want to know the price of a product."

[1242] Step 6:

[1243] Based on the user's identified intent, the server queries internal databases and external sources to find the required information.

[1244] Step 7:

[1245] The server analyzes the retrieved information and summarizes the key points, presenting them in a format that is easily understandable to the user (e.g., bullet points or key points).

[1246] Step 8:

[1247] The server sends the summarized information to the user's device, optionally specifying the display format (e.g., popup, sidebar).

[1248] Step 9:

[1249] The user device analyzes the received information and displays it on the user's screen in the specified format, allowing the user to check the information necessary for the conversation without having to change their line of sight during the meeting.

[1250] This processing flow allows users to instantly obtain the right information during online meetings and keep the conversation moving smoothly.

[1251] Example 1

[1252] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1253] Online meetings often present a challenge, making it difficult for users to quickly obtain the information they need. In particular, the lack of real-time information search and a summary of the information provided means users must interrupt the flow of the meeting to conduct research, reducing meeting efficiency. Furthermore, the accuracy and appropriateness of the information obtained can also be a problem.

[1254] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1255] In this invention, the server includes means for acquiring voice data, means for converting the acquired voice data into text data in real time, means for analyzing the text data and identifying the user's intention, means for searching for appropriate information based on the identified intention, means for summarizing the searched information, and means for transmitting the summarized information to a user terminal and displaying it on the user's screen, thereby enabling users to acquire necessary information in real time during an online conference and understand that information in a short amount of time.

[1256] "Voice data" refers to data used to record and transmit a user's speech or voice in digital form.

[1257] "Text data" refers to character information converted from voice data using voice recognition technology.

[1258] "Real-time" is a concept that refers to events and data processing occurring almost immediately.

[1259] "Means for analyzing text data" refers to technologies that use generative AI models and natural language processing techniques to understand the content and context of text data and identify intent.

[1260] "User intent" is the result of analyzing what the user is asking for through their statements and what information they need.

[1261] "Information retrieval means" refers to techniques for retrieving relevant information from internal databases and external sources based on the specified intent.

[1262] "Means for summarizing information" refers to techniques for extracting important points from acquired information and organizing them in a format that is easy for users to understand.

[1263] A "user terminal" (terminal) is a device, such as a computer or smartphone, that a user uses to conduct online meetings and display information.

[1264] "Speech recognition technology" is a technology for analyzing voice data and converting it into text information. It includes highly accurate voice recognition engines.

[1265] An "internal database" is a collection of data held within a company and is a searchable source of information.

[1266] "External sources" are sources other than internal databases, such as the Internet or other external data repositories.

[1267] The "means for converting into text data in real time" is a process for instantly converting acquired voice data into text data.

[1268] "Means for displaying on a user's screen" means a method for visually presenting information on the screen of a user's device, including formats such as pop-ups and sidebars.

[1269] "Generative AI models" refer to generative artificial intelligence such as GPT-4, which are used to analyze text data and identify user intent.

[1270] MODE FOR CARRYING OUT THE INVENTION

[1271] This invention is a system for users to acquire necessary information in real time during an online conference and visually display it. This system is mainly composed of a server and user terminals.

[1272] System Overview

[1273] User Device

[1274] The user terminal can be a device such as a PC or smartphone. The user terminal has the following functions:

[1275] Software for conducting online meetings (e.g., Zoom, Microsoft Teams)

[1276] Ability to capture audio data using built-in or external microphone

[1277] Communication function to send voice data to the server

[1278] A function to display information received from the server on the screen.

[1279] server

[1280] The server is a powerful computer that performs the following functions:

[1281] Receiving audio data

[1282] Parsing and converting voice data to text (using a speech recognition engine such as the Google Speech-to-Text API)

[1283] User intent analysis using generative AI models (e.g., GPT-4)

[1284] Information retrieval from internal databases and external sources

[1285] Summarizing the searched information and sending it to the user's device

[1286] Program processing explanation

[1287] Acquiring and sending audio data

[1288] When a user starts an online meeting, the device uses a microphone to capture audio data and transmits it to the server in packets at regular intervals, using the HTTP or WebSocket protocol.

[1289] Analysis of voice data and conversion to text

[1290] Once the server receives the voice data, it converts it into text using the Google Speech-to-Text API, which applies phonemic analysis, acoustic models, and language models.

[1291] User intent analysis

[1292] The server uses text generation AI (GPT-4) to analyze the text data and identify the user's intent. For example, it analyzes a statement such as "Please tell me more about this product" and identifies that the information being sought is product details.

[1293] Finding and summarizing relevant information

[1294] The server searches for information based on the specified intent from internal databases and external sources, then uses summarization tools to extract key points and present them in a format that is easy for the user to understand.

[1295] Sending and Displaying Information

[1296] The summarized information is then packetized and sent from the server to the user's device, which then displays the received information in a pop-up or sidebar format, allowing users to instantly check the information during the meeting.

[1297] Specific examples

[1298] Example 1: Providing product information

[1299] If a user says, "Tell me more about this product" during an online meeting, the device captures the speech and sends the audio data to the server. The server converts the audio data into text using the Google Speech-to-Text API and uses GPT-4 to determine that the user is requesting product details. The server then searches the company's database for product specifications and manuals and sends the summarized information to the user's device. The user's device then displays the summarized information as a pop-up on the screen.

[1300] Prompt Sentence Examples

[1301] Analyze audio data containing the phrase "Tell me more about this product" and use the following as a prompt to find information about the product details:

[1302] "Say: Tell me more about this product. Response: Please describe the specifications and key features of this product."

[1303] Example 2: Checking contract terms

[1304] If the user says, "Please tell me more about the terms of the contract," the device sends the voice data to the server, which converts the voice data into text using the Google Speech-to-Text API and analyzes the keyword "terms of contract" using GPT-4. Information about the identified terms of contract is retrieved from the database, summarized, sent to the user's device, and displayed on the screen.

[1305] Prompt Sentence Examples

[1306] Analyze audio data containing the phrase "Tell me more about the terms and conditions," and use the following as a prompt to find information about the terms and conditions:

[1307] "Statement: Please tell me more about the terms and conditions. Response: Please provide a summary of the key points of the applicable terms and conditions."

[1308] In this way, users can instantly obtain the information they need during an online meeting and conduct the meeting efficiently.

[1309] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1310] Step 1: User starts an online meeting

[1311] A user launches online conference software (e.g., Zoom, Microsoft Teams) on their device and starts a conference. This puts the device into a preparation state for the conference. The input is the user's voice, and the output is the device entering online conference mode.

[1312] Step 2: The device captures the audio data

[1313] The device uses a built-in or external microphone to capture the user's voice in real time, and stores the voice data in a buffer. The input is the user's voice, and the output is the stored voice data.

[1314] Step 3: The device sends the audio data to the server

[1315] The captured audio data is divided into packets at regular intervals and sent to the server using HTTP or WebSocket protocol. The input is the audio data stored in the buffer, and the output is the audio data packets sent to the server.

[1316] Step 4: The server receives the audio data

[1317] The server receives packets of voice data sent from the terminal and combines them into the original voice data. The input is the voice data packets and the output is the combined voice data.

[1318] Step 5: The server converts the audio data into text data.

[1319] The server uses speech recognition technology (e.g., Google Speech-to-Text API) to convert the combined voice data into text data in real time. This process involves phonemic analysis, acoustic models, and language models. The input is the combined voice data, and the output is the generated text data.

[1320] Step 6: The server analyzes the text data to determine the user's intent

[1321] The server uses a generative AI model (e.g., GPT-4) to analyze the generated text data and identify the user's intent. This analysis uses natural language processing techniques. The input is the generated text data, and the output is the identified user's intent.

[1322] Step 7: The server looks up the appropriate information

[1323] The server searches for the appropriate information from internal databases or external sources (e.g., internal company databases or external APIs) based on the identified user intent. It issues queries and retrieves the required data. The input is the identified user intent, and the output is the retrieved information.

[1324] Step 8: The server summarizes the retrieved information

[1325] The server uses a summarization tool to extract key points from the retrieved information and present it in a form that is easy for the user to understand. The input is the retrieved information and the output is the summarized information.

[1326] Step 9: The server sends the summarized information to the user terminal.

[1327] The summarized information is packetized and sent to the user terminal in the form of an HTTP POST request, etc. The input is the summarized information, and the output is the information packets sent to the user terminal.

[1328] Step 10: Your device will display the information

[1329] The user terminal analyzes the received information packets and displays them on the user screen in an appropriate format (e.g., a pop-up or sidebar). This allows users to check the necessary information during a meeting without changing their line of sight. The input is the received information packet, and the output is the information displayed on the screen.

[1330] (Application example 1)

[1331] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1332] In conventional online meetings, it was difficult for participants to instantly obtain the information they needed. It was also difficult for workers in factories to obtain information by voice without using their hands, or to give work instructions to robots. This created challenges for improving work efficiency and maintaining work accuracy.

[1333] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1334] In this invention, the server includes a means for converting voice data into text data in real time, a means for analyzing the text data to identify the user's intent, and a means for retrieving and summarizing appropriate information based on the identified intent, thereby enabling online conference participants and factory workers to immediately display appropriate information or perform work operations based on voice instructions.

[1335] An "online conference" is a conference system that allows multiple users in remote locations to communicate in real time using the Internet.

[1336] "Audio data" is digital data obtained by converting an audio signal acquired via an audio input device such as a microphone.

[1337] "Text data" is voice data that has been analyzed using voice recognition technology and its contents expressed as a string of characters (text).

[1338] "User intent identification" refers to technologies and algorithms that analyze text data and determine what it means and what the user wants.

[1339] "Means for retrieving appropriate information" refers to technology for retrieving relevant information from internal databases or external information sources based on the user's intent.

[1340] A "user terminal" is a device such as a computer or smartphone that a user uses to conduct an online conference.

[1341] "Means for displaying on the screen" refers to technology that displays the searched information on the screen of the user's device as a pop-up or notification so that the user can visually confirm the information.

[1342] "Factory work" refers to work carried out within a factory, such as manufacturing, assembling, and maintaining products.

[1343] "Voice instructions" are commands or requests given by a user to a system or robot through voice input.

[1344] "Analysis means" refers to the technology and algorithms used to process audio and text data and understand its content.

[1345] "Means for executing tasks" refers to the technology and algorithms that enable a robot to perform specific actions or tasks based on the user's intentions or voice instructions.

[1346] The present invention is a system that acquires and analyzes voice data during online meetings and factory work, and provides appropriate information. This system is composed of a user terminal, a server, and a robot.

[1347] System configuration

[1348] User Device

[1349] User terminals are devices used for online meetings and factory work. Examples include computers, smartphones, and tablets. User terminals are responsible for capturing voice data, sending data to the server, receiving data from the server, and displaying the data on the screen. A highly accurate voice recognition engine is used for voice recognition (e.g., Google Speech-to-Text API).

[1350] server

[1351] The server is a remote computer that receives voice data, analyzes it, converts it into text, identifies intent, searches for and summarizes information, and then transmits the data to the user's device. NLP technology (e.g., OpenAI's GPT-3) is used to convert voice data into text and analyze intent. It also summarizes search results and provides them to the user in an easy-to-understand format.

[1352] robot

[1353] A robot is a device that performs specific actions or tasks in a factory by receiving voice instructions. The robot uses a built-in microphone to capture the voice instructions and transmits them to a server.

[1354] System Operation

[1355] 1. Acquiring and sending audio data

[1356] When a user starts an online meeting or factory work, the user device or robot captures audio in real time using a built-in or external microphone, and the captured audio data is divided into packets and sent to the server.

[1357] 2. Analysis of voice data and conversion to text

[1358] The server analyzes the voice data received from the user's device or robot in real time and converts it into text data using voice recognition technology.

[1359] 3. User Intent Analysis

[1360] The server uses text generation AI to analyze the user's statements based on text data and identify the information or instructions the user is seeking. Natural language processing (NLP) technology is used for intent analysis.

[1361] 4. Finding and summarizing relevant information

[1362] Based on the identified intent, the server retrieves appropriate information from internal databases and external sources, summarizes the information, and presents it in a format that is easy for the user to understand.

[1363] 5. Transmission and Display of Information

[1364] The server sends the summarized information to the user's device or the robot. The user's device displays the received information in a pop-up format on the screen, and the robot provides the information through voice or a display.

[1365] Specific examples

[1366] Example 1: Providing product information

[1367] When a user says "Tell me more about this product" during an online meeting, the voice data is sent to the server for analysis. The server then searches for detailed product information and sends a summary to the user's device for display.

[1368] Example 2: Factory Work Order

[1369] When a worker in the factory issues a voice command such as "What is the next maintenance procedure for this machine?", the robot captures the voice and sends it to the server. The server searches for the maintenance procedure and sends summary information to the robot. The robot then provides the maintenance procedure by display or voice.

[1370] Example prompt sentence:

[1371] What is the next maintenance procedure?

[1372] Please explain how to replace the next filter.

[1373] Please display the relevant page in the manual.

[1374] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1375] Step 1: Acquire audio data

[1376] When a user starts an online meeting or factory work, the user terminal or robot uses an internal or external microphone to capture audio in real time. The audio data input is the user's speech, and the output is the captured audio data. This data is converted into a digital format and used for subsequent analysis. Specifically, the microphone captures the audio waveform and processes it as a digital signal.

[1377] Step 2: Sending audio data

[1378] The terminal divides the captured audio data into packets at regular intervals and sends them to the server. The input is the captured audio data, and the output is packetized audio data. Specifically, the audio data is sent to the server in real time via the network. This transmission uses UDP or TCP / IP.

[1379] Step 3: Analyze and convert audio data into text

[1380] The server analyzes the received voice data in real time and converts it into text data using speech recognition technology. The input is packetized voice data, and the output is text data. This conversion uses a highly accurate speech recognition engine (e.g., Google Speech-to-Text API). Specifically, the voice signal is matched with an acoustic model and converted into text.

[1381] Step 4: User Intent Analysis

[1382] The server uses a generative AI model (e.g., OpenAI's GPT-3) to analyze the text data, interpreting the user's speech and identifying their intent. The input is converted text data, and the output is data containing the user's intent. Specifically, the text data is analyzed using a natural language processing algorithm to identify the intent of the speech and the desired information.

[1383] Step 5: Finding and Summarizing Pertinent Information

[1384] Based on the identified intent, the server searches for relevant information from internal databases and external sources and generates a summary. The input is data containing the user's intent, and the output is summarized information. Specifically, a search query is sent to the database, and the retrieved information is summarized using natural language processing algorithms.

[1385] Step 6: Submit summary information

[1386] The server sends the summarized information to the user terminal or robot. The input is the summarized information, and the output is the transmitted information. Specifically, data packets are sent to the user terminal or robot via the network.

[1387] Step 7: Viewing information

[1388] The terminal or robot provides the received information to the user visually or audibly. The input is information sent from the server, and the output is a visual display or audio information provided to the user. Specific actions include a pop-up appearing on the terminal screen, or the robot giving instructions by voice.

[1389] Example prompt sentence:

[1390] What is the next maintenance procedure?

[1391] Please explain how to replace the next filter.

[1392] Please display the relevant page in the manual.

[1393] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1394] This system analyzes conversations in real time during online meetings, displays appropriate information only on the user's screen, and also recognizes the user's emotions and adjusts the information provided accordingly. This system consists of a server, a user terminal, and an emotion engine.

[1395] System configuration

[1396] 1. User Device

[1397] A user terminal is a device such as a computer or smartphone that has the functionality to conduct online meetings. The user terminal is responsible for capturing voice data, sending data to the server and emotion engine, receiving data from the server, and displaying the data on the screen.

[1398] 2. Server

[1399] It is a remote computer that receives voice and text data, analyzes it, analyzes emotions, identifies intent, retrieves and summarizes information, and transmits the data to a user terminal.

[1400] 3. Emotion Engine

[1401] This engine analyzes the user's speech and text data to recognize the user's emotions. The engine determines emotions from the tone, speed, pitch, and content of the speech.

[1402] Program processing explanation

[1403] 1. Acquiring and sending audio data

[1404] When a user starts an online conference, the user device captures the conference audio in real time using a built-in or external microphone. The captured audio data is divided into packets at regular intervals and sent to the server and emotion engine.

[1405] 2. Analysis of voice data and conversion to text

[1406] The server receives the voice data sent from the user's device and stores it in a buffer. It then converts the voice data into text data using voice recognition technology. This process uses a highly accurate voice recognition engine.

[1407] 3. User Intent Analysis

[1408] Using text generation AI, the server analyzes the user's statements based on text data and identifies the information and answers the user is looking for. Natural language processing (NLP) technology is used to analyze intent.

[1409] 4. Emotion analysis

[1410] The emotion engine analyzes the voice and text data sent from the user's device and recognizes the user's emotions using an algorithm that determines emotions from the tone, speed, pitch, and content of the speech.

[1411] 5. Finding and summarizing relevant information

[1412] The server searches for appropriate information from internal databases and external sources based on the identified intent and the results of sentiment analysis. The content and format of the information presented is adjusted according to the results of sentiment analysis. The retrieved information is summarized and formatted in a way that is easily understandable to the user.

[1413] 6. Transmission and Display of Information

[1414] The summarized information is sent from the server to the user's device. The user's device displays the received information on the screen in a pop-up format or other format, allowing the user to check the information without changing their line of sight during the meeting. Depending on the results of emotion analysis, if the user is feeling anxious, information with a more reassuring content and tone is provided.

[1415] Specific examples

[1416] Example 1: Providing product information

[1417] Imagine a user says, "Tell me more about this product" during an online meeting. The user's device captures this speech and sends the voice data to the server and emotion engine. The server analyzes the voice data and converts it into text data. The emotion engine analyzes the emotions from the user's speech, and if tension or impatience is detected, it selects materials that explain things in a gentle and easy-to-understand tone. The server searches the internal database for product specifications and manuals and summarizes the information it retrieves. This summary information is sent to the user's device and displayed on the screen.

[1418] Example 2: Checking contract terms

[1419] Consider a scenario where a user says, "Please tell me more about the terms of the contract." The user's device sends voice data to the server and emotion engine, which converts the voice data into text data. The text generation AI extracts the keyword "terms of contract," and the emotion engine analyzes whether the user is excited or anxious. If the user is anxious, detailed and transparent information is provided as a priority. The server retrieves the relevant contract terms document from the database and summarizes it. The summary information is sent to the user's device and displayed on the screen.

[1420] In this way, the present invention provides a system that allows users to instantly obtain necessary information during an online conference and provide appropriate answers and suggestions, and is also capable of providing information that takes into account the user's emotions.

[1421] The processing flow will be explained below.

[1422] Step 1:

[1423] When a user starts an online conference, the user device captures the conference audio in real time using a built-in or external microphone. The captured audio data is divided into packets at regular intervals and sent to the server and emotion engine.

[1424] Step 2:

[1425] The server receives the voice data sent from the user device and stores it in a buffer. At the same time, the emotion engine also receives the voice data and prepares it for analysis.

[1426] Step 3:

[1427] The server uses speech recognition technology to convert the voice data into text data, using a highly accurate speech recognition engine.

[1428] Step 4:

[1429] The server inputs the converted text data into a text generation AI (e.g., GPT-3) to analyze what the user said, thereby identifying the user's intent and the information they need.

[1430] Step 5:

[1431] The emotion engine analyzes the voice and text data sent from the user's device to recognize the user's emotions, taking into account voice tone, speed, pitch, and content.

[1432] Step 6:

[1433] The server combines the analysis results of the text generation AI with those of the emotion engine to comprehensively evaluate the user's intentions and emotions.

[1434] Step 7:

[1435] The server queries internal databases and external sources based on the overall evaluation to find the required information, and adjusts the content and format of the information provided depending on the results of the sentiment analysis.

[1436] Step 8:

[1437] The server analyzes the retrieved information and summarizes key points, formatting this summary to suit the user's emotional state (e.g., a polite, reassuring tone).

[1438] Step 9:

[1439] The server sends the summarized information to the user's device, along with instructions on how to display the information (e.g., popup, sidebar).

[1440] Step 10:

[1441] The user device analyzes the received information and displays it on the user's screen in the specified format, allowing the user to check the necessary information without changing their line of sight during the meeting.

[1442] This processing flow allows users to instantly obtain appropriate information during online meetings, and the information is provided in a manner that takes into consideration the user's feelings, allowing the meeting to proceed smoothly.

[1443] Example 2

[1444] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1445] Conventional online conference systems have difficulty understanding the context of conversations and providing appropriate information in real time. Furthermore, they do not provide information that takes users' emotions into consideration, which prevents them from increasing user satisfaction. Furthermore, they lack a means to efficiently aggregate data from multiple sources, making it difficult to provide necessary information immediately during a conference. There is a need to solve these issues and realize more effective information provision.

[1446] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1447] In this invention, the server includes means for converting user voice data into text data in real time, means for analyzing the text data to identify the user's intention, means for analyzing the user's emotions, means for searching for appropriate information based on the identified intention and the results of the emotion analysis, means for summarizing the searched information, and means for transmitting the summarized information to the user terminal and displaying it on the user's screen. This makes it possible to provide appropriate information immediately during a conference and to provide information that is tailored to the user's emotions.

[1448] An "online conference" is a method of holding a conference between multiple participants in remote locations by sharing audio and video in real time via the Internet.

[1449] "Voice data" refers to data in which the user's voice is recorded in digital format.

[1450] "Text data" is character string data generated by analyzing voice data.

[1451] "Analysis" is the act of analyzing data using data processing techniques to understand its meaning and intent.

[1452] "User intent" refers to the information, questions, or requests that a user is trying to convey through their speech.

[1453] "Emotions" are psychological states and feelings that can be inferred from a user's statements and actions.

[1454] "Emotion analysis" is a technique for determining a user's emotions based on voice tone, speed, pitch, and text content.

[1455] "Information retrieval" is the act of finding necessary information from databases or external sources.

[1456] "Summarizing" is the act of extracting important points and organizing information in a concise form.

[1457] A "user terminal" is a device used by a user, such as a computer or smartphone.

[1458] The present invention is a system that analyzes user conversation content in real time in an online conference system, recognizes the user's emotions, and provides appropriate information. This system is composed of a server, user terminals, and an emotion engine.

[1459] System Configuration

[1460] 1. User Device

[1461] The user terminal is a device such as a computer or a smartphone for conducting an online conference, which allows the user to start the conference, capture audio data, and transmit the necessary data to the server and the emotion engine.

[1462] 2. Server

[1463] The server is a remote computer responsible for receiving, analyzing, sentiment analysis, intent identification, information retrieval and summarization, and transmitting data to the user terminal.

[1464] 3. Emotion Engine

[1465] The emotion engine analyzes voice and text data to recognize the user's emotions. It determines the user's emotions from the voice tone, speed, pitch, and content of the speech.

[1466] Program processing explanation

[1467] 1. Acquiring and sending audio data

[1468] When a user starts an online conference, the user's device will capture the conference audio in real time using a built-in or external microphone. The captured audio data will be divided into packets and sent to the server and emotion engine. This involves encoding the audio data and sending an HTTP POST request.

[1469] 2. Analysis of voice data and conversion to text

[1470] The server stores the received voice data in a buffer and converts it into text data using speech recognition technology (e.g., Google Cloud Speech-to-Text), which is then stored in a data store.

[1471] 3. User Intent Analysis

[1472] The server uses a generative AI model (e.g., OpenAI GPT-4) to analyze the text data and understand what the user is saying. Specifically, it uses NLP techniques to identify the user's intent.

[1473] 4. Emotion analysis

[1474] The emotion engine uses software such as IBM Watson Tone Analyzer to analyze the tone, speed, pitch, and content of the voice and text data transmitted to identify the user's emotions.

[1475] 5. Finding and summarizing relevant information

[1476] The server searches for appropriate information from an internal database or external information sources based on the results of the identified intent and sentiment analysis. The retrieved information is then organized into an easy-to-understand format by a summarization engine. The summarized information is then sent from the server to the user's device.

[1477] 6. Transmission and Display of Information

[1478] The device displays the received information on the screen in a pop-up or slide-in format, allowing users to instantly check the information during a meeting, and the content and format of the information are adjusted according to the results of sentiment analysis.

[1479] Specific examples

[1480] 1. Providing product information

[1481] If a user says "Tell me more about this product" during an online meeting, the device captures the speech and sends the voice data to the server and emotion engine. The server converts the voice data into text data and analyzes it together with the emotion engine. If the user is nervous, the server selects materials explained in a gentle tone, sends them to the device, and displays them on the screen.

[1482] 2. Check the terms and conditions

[1483] If a user says, "Please tell me more about the terms of the contract," the device sends the voice data to the server and emotion engine, which converts the voice data into text. A text-generation AI is used to extract the keyword "terms of contract," and the emotion engine analyzes the user's emotions. If the user is anxious, a document containing a detailed explanation is selected, and the information is summarized and displayed on the device.

[1484] Prompt sentence examples

[1485] "Tell me more about this product"

[1486] "Please tell me more about the terms of the contract."

[1487] I'd like to know how to use this feature.

[1488] Please tell me the progress of the project.

[1489] As a result, the present invention provides a system that allows users to instantly obtain necessary information during an online conference and provide appropriate answers and suggestions, and also makes it possible to provide information that takes into account the user's emotions.

[1490] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1491] Step 1:

[1492] A user starts an online meeting.

[1493] Specific operation: A user starts a meeting using an online conferencing tool (e.g., Zoom or Teams). At this time, the device's microphone becomes active.

[1494] Input: User's speaking voice.

[1495] Output: Capture of conversational audio.

[1496] Step 2:

[1497] This is done by the terminal to capture voice data in real time and split it into packets.

[1498] Specific operation: The device captures audio data using a built-in or external microphone, and packets the audio data at regular intervals (e.g., every 10 seconds).

[1499] Input: The captured audio data.

[1500] Output: Packetized voice data.

[1501] Step 3:

[1502] The terminal transmits the captured voice data packets to the server and the emotion engine.

[1503] Specific operation: The device uses an HTTP POST request to send the captured audio data packets to the server and the emotion engine's API endpoint.

[1504] Input: Packetized voice data.

[1505] Output: Sending voice data to the server and emotion engine.

[1506] Step 4:

[1507] The server stores the received voice data in a buffer and converts it into text data using a voice recognition engine.

[1508] Specific operation: After the server receives the voice data, it stores it in a buffer and then converts it into text using voice recognition technology (e.g., Google Cloud Speech-to-Text).

[1509] Input: The audio data received by the server.

[1510] Output: The speech converted into text data.

[1511] Step 5:

[1512] The server uses a generative AI model to analyze the text data and identify the user's intent.

[1513] What it does: The server reads the stored text data, provides prompts to a generative AI model (e.g., OpenAI GPT-4), analyzes the content of the text data, and uses natural language processing (NLP) techniques to identify the user's intent.

[1514] Input: The converted text data.

[1515] Output: Identified user intent.

[1516] Step 6:

[1517] The emotion engine analyzes voice and text data to recognize the user's emotions.

[1518] What it does: The emotion engine uses algorithms (e.g., IBM Watson Tone Analyzer) to analyze the tone, speed, pitch, and content of speech based on the transmitted voice and text data to determine the user's emotions.

[1519] Input: Audio and text data.

[1520] Output: Perceived user emotion.

[1521] Step 7:

[1522] The server retrieves and summarizes appropriate information based on the intent and emotion information.

[1523] How it works: The server searches for relevant information from internal databases and external sources based on the results of intent analysis and sentiment analysis, then sends the search results to a summarization engine, which summarizes them in an easy-to-understand format.

[1524] Input: Intention and emotion information.

[1525] Output: Summarized information.

[1526] Step 8:

[1527] The server sends the summarized information to the terminal, which displays the information.

[1528] Specific operation: The server sends the summarized information to the device, the device decodes the received information, and the information is displayed in a pop-up or slide-in format in the user interface.

[1529] Input: Summarized information.

[1530] Output: Information displayed on the terminal.

[1531] This allows users to instantly obtain the information they need during a meeting and provides optimal support for what is being said.

[1532] (Application example 2)

[1533] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1534] In online meetings, it is not easy for users to instantly obtain the necessary information while concentrating on the conversation. Furthermore, since information is not provided taking into account the user's emotions, there is a problem that information is received uniformly. In the advertising industry in particular, it is important to provide information dynamically according to the client's reactions, but conventional systems are inadequate in this regard.

[1535] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1536] In this invention, the server includes means for acquiring voice data, means for converting the voice data into text data, means for analyzing the text data to identify the user's intention, means for transmitting the retrieved information to the user terminal and displaying it, means for recognizing the user's emotions based on the analyzed data, and means for dynamically adjusting the information according to the user's emotional state. This allows the user to instantly obtain the information they need while concentrating on the conversation, and makes it possible to provide information that takes their emotions into consideration.

[1537] "Audio data" refers to digital audio signals that record users' statements and conversations during online meetings.

[1538] "Text data" is voice data that has been analyzed and converted into a string of characters, and is a representation of what the user has said as text information.

[1539] "Intent" refers to the information or answer a user is seeking through their statement or question.

[1540] "Information retrieval means" refers to the techniques and methods used to find relevant information from internal databases and external sources based on the identified user intent.

[1541] "Emotion recognition means" refers to technologies or algorithms that identify a user's emotional state based on analyzed voice and text data.

[1542] "Dynamic adjustment means" refers to techniques or methods that change the content or format of the information provided in a timely manner in response to the recognized emotional state of the user.

[1543] A "server" is a remote computer system that performs processing such as analysis, retrieval, emotion recognition, and dynamic adjustment of voice and text data.

[1544] A "user terminal" is a device that conducts online conferences, acquires audio data, transmits and receives data to and from a server, and displays information on a screen.

[1545] The present invention relates to a system that enables users to efficiently collect information through online meetings and provides appropriate information based on their emotions. A specific embodiment of this system will be described below.

[1546] System Configuration

[1547] 1. User Device:

[1548] The user terminal is a device for conducting online meetings and acquiring audio data. This terminal can be a smartphone or a laptop. The user terminal captures audio data in real time using a built-in or external microphone and transmits the data to the server and emotion engine. It also has a display for displaying information.

[1549] 2. Server:

[1550] The server is a remote computer system that analyzes voice and text data and recognizes emotions. The server converts voice data into text using advanced speech recognition technologies (such as Google Speech-to-Text API or IBM Watson Speech to Text). It then uses natural language processing (NLP) technologies such as OpenAI GPT-4 to analyze user intent and perform information retrieval and summarization.

[1551] 3. Emotion Engine:

[1552] The emotion engine is an algorithm and technology that analyzes user speech and text data to recognize emotions. Sentiment analysis uses voice tone, speed, pitch, and facial expression analysis technologies (Amazon Rekognition and Google Cloud's emotion analysis API) to provide feedback based on the user's emotional state.

[1553] Program processing explanation

[1554] When a user starts an online meeting, the user's device captures audio in real time using a built-in microphone and sends the captured audio data to the server and emotion engine. The server converts the audio data into text using the Google Speech-to-Text API. Based on this text data, the user's intent is analyzed using OpenAI GPT-4.

[1555] Next, the emotion engine analyzes the voice and text data sent from the user device and recognizes the user's emotions using Amazon Rekognition or Google Cloud's emotion analysis API. After analyzing the user's intent and emotions, the server searches for and summarizes appropriate information from internal databases and external sources based on the identified intent and emotions.

[1556] The summarized information is sent from the server to the user's device, which then displays the received information on the screen.Specific examples and case studies will be presented, particularly in the advertising industry, where information is dynamically adjusted according to the client's emotional state during a presentation, providing a sense of security.

[1557] Examples of concrete examples and prompts

[1558] For example, if a client anxiously asks, "Can you give me some specific examples of success stories for this marketing strategy?", the system analyzes the voice and text data to recognize the anxious state. The server then searches an internal database for success stories and provides the information in a format that is easy for the user to understand.

[1559] Example prompt sentence:

[1560] User said: "Can you give me some concrete examples of success with this marketing strategy?"

[1561] Sentiment analysis results: "Users are feeling anxious"

[1562] Appropriate information: "Success stories, case studies, and easy-to-understand diagrams and charts"

[1563] Example of what the presenter sees:

[1564] Your client is feeling anxious. Please suggest the following success stories:

[1565] An example of a company's sales increase through a campaign

[1566] Industry Achievements and Statistics

[1567] Summary of customer satisfaction survey results

[1568] As described above, the present invention provides a system that enables users to efficiently collect information during an online conference and provide appropriate information based on emotions.

[1569] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1570] Step 1:

[1571] When an online conference starts, the user device captures voice data in real time using a built-in microphone. This voice data is sent as input to the server and emotion engine. Specifically, the online conference application acquires the voice signal from the microphone, divides it into data packets, and sends them at regular time intervals.

[1572] Step 2:

[1573] The server converts the voice data received from the user's device into text data using a highly accurate voice recognition engine (e.g., Google Speech-to-Text API, IBM Watson Speech to Text). This process outputs the voice data as text data. Specifically, it performs a spectral analysis of the voice signal, breaks it down into phonemes, and then reconstructs it into a word sequence.

[1574] Step 3:

[1575] The server uses natural language processing (NLP) technology such as OpenAI GPT-4 to analyze the user's intent using the generated text data. It understands the user's speech from the input text data, identifies their intent, and outputs that information. Specifically, it uses a word embedding model to understand the context and determine the information the user is looking for and the intent of their question.

[1576] Step 4:

[1577] The emotion engine analyzes the voice and text data sent from the server to recognize the user's emotions. It analyzes the input voice tone, speed, pitch, and content of speech and outputs the user's emotional state. Specifically, it uses a combination of facial expression recognition technology (e.g., Amazon Rekognition) and voice analysis algorithms (e.g., Google Cloud's emotion analysis API) to identify emotions.

[1578] Step 5:

[1579] The server searches for appropriate information from an internal database or external information sources based on the analysis results of the identified intent and emotion. The server matches the analysis results as input with the database, finds the most appropriate information, and outputs summarized information. Specifically, the server searches the database using pre-configured queries to extract highly relevant information.

[1580] Step 6:

[1581] The server searches for and sends the summarized information to the user's device. The user's device displays the received information in a pop-up format on the screen so that the user can easily check it during the meeting. The input summary information is then formatted into an appropriate format and visually presented to the user. Specifically, the information pop-up is displayed in a position that does not obstruct the user's line of sight, allowing the user to access detailed information as needed.

[1582] Through the above processing steps, the system allows users to obtain the information they need in real time during online meetings, and also to obtain information tailored to their emotions.

[1583] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1584] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1585] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1586] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1587] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1588] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1589] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1590] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1591] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1592] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1593] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1594] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1595] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1596] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1597] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1598] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1599] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1600] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1601] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1602] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1603] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1604] The following is further disclosed regarding the above embodiment.

[1605] (Claim 1)

[1606] A means for acquiring audio data in an online conference;

[1607] A means for converting the acquired voice data into text data in real time;

[1608] A means for analyzing text data and identifying user intent;

[1609] A means of retrieving appropriate information based on the identified intent;

[1610] means for transmitting the searched information to a user terminal and displaying it on the user's screen;

[1611] A system including:

[1612] (Claim 2)

[1613] 10. The system of claim 1, utilizing speech recognition technology to convert the speech data into text data.

[1614] (Claim 3)

[1615] 10. The system of claim 1, which uses an internal database to retrieve the appropriate information.

[1616] "Example 1"

[1617] (Claim 1)

[1618] A means for acquiring audio data in an online conference;

[1619] A means for converting the acquired voice data into text data in real time;

[1620] A means for analyzing text data and identifying user intent;

[1621] A means of retrieving appropriate information based on the identified intent;

[1622] a means for summarizing the retrieved information;

[1623] means for transmitting the summarized information to a user terminal and displaying it on the user's screen;

[1624] A system including:

[1625] (Claim 2)

[1626] 10. The system of claim 1, utilizing speech recognition technology to convert the speech data into text data.

[1627] (Claim 3)

[1628] 10. The system of claim 1, wherein the system uses an internal database and / or external sources to retrieve the appropriate information.

[1629] "Application Example 1"

[1630] (Claim 1)

[1631] A means for acquiring audio data in an online conference;

[1632] A means for converting the acquired voice data into text data in real time;

[1633] A means for analyzing text data and identifying user intent;

[1634] A means of retrieving appropriate information based on the identified intent;

[1635] means for transmitting the searched information to a user terminal and displaying it on the user's screen;

[1636] means for acquiring and analyzing voice instructions and performing the intended task;

[1637] a means for displaying appropriate information or performing work operations based on voice instructions during factory work;

[1638] A system including:

[1639] (Claim 2)

[1640] 10. The system of claim 1, utilizing speech recognition technology to convert the speech data into text data.

[1641] (Claim 3)

[1642] 10. The system of claim 1, which uses an internal database to retrieve the appropriate information.

[1643] "Example 2: Combining Emotion Engines"

[1644] (Claim 1)

[1645] A means for acquiring audio data in an online conference;

[1646] A means for converting the acquired voice data into text data in real time;

[1647] means for analyzing the text data and identifying the user's intent;

[1648] means for analyzing user emotions;

[1649] a means of retrieving appropriate information based on the determined intent and the results of the sentiment analysis;

[1650] a means for summarizing the retrieved information;

[1651] means for transmitting the summarized information to a user terminal and displaying it on the user screen;

[1652] A system including:

[1653] (Claim 2)

[1654] 10. The system of claim 1, utilizing speech recognition technology to convert the speech data into text data.

[1655] (Claim 3)

[1656] 10. The system of claim 1, which uses internal databases and external sources to retrieve appropriate information.

[1657] (Claim 4)

[1658] 10. The system of claim 1, wherein the system uses algorithms that analyze voice tone, rate, pitch, and text content for sentiment analysis.

[1659] (Claim 5)

[1660] 10. The system of claim 1, further comprising means for adjusting the content and format of the information in response to the user's emotions.

[1661] "Application example 2 when combining emotion engines"

[1662] (Claim 1)

[1663] A means for acquiring audio data in an online conference;

[1664] A means for converting the acquired voice data into text data in real time;

[1665] A means for analyzing text data and identifying user intent;

[1666] A means of retrieving appropriate information based on the identified intent;

[1667] means for transmitting the searched information to a user terminal and displaying it on the user's screen;

[1668] means for recognizing a user's emotion based on the analyzed text data and voice data;

[1669] means for dynamically adjusting the information provided in response to the user's emotional state;

[1670] A system including:

[1671] (Claim 2)

[1672] 10. The system of claim 1, utilizing speech recognition technology to convert the speech data into text data.

[1673] (Claim 3)

[1674] 10. The system of claim 1, which uses an internal database to retrieve the appropriate information. [Explanation of symbols]

[1675] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for acquiring audio data in an online conference; A means for converting the acquired voice data into text data in real time; A means for analyzing text data and identifying user intent; A means of retrieving appropriate information based on the identified intent; means for transmitting the searched information to a user terminal and displaying it on the user's screen; A system including:

2. 10. The system of claim 1, utilizing speech recognition techniques to convert the speech data into text data.

3. 10. The system of claim 1, which uses an internal database to retrieve the appropriate information.

Images