System

Abstract

A system is provided.SOLUTION: The system includes a means for converting voice data acquired in real time into text data, a means for specifying an unknown word or technical term from the text data, a means for retrieving the meaning and context of the specified word from Web data or past recording data, and a means for providing the retrieval result to the terminal of a user in real time.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's business environment and intercultural communication, there is a problem in that it is difficult to understand technical terms and difficult words used in online meetings on the spot. This problem makes it difficult for meeting participants to understand the meaning of words without interrupting the progress of the meeting, resulting in a decrease in the efficiency and effectiveness of the meeting. [Means for solving the problem]

[0005] The present invention provides a system that converts speech data acquired in real time into text data and identifies unknown words or technical terms from the text data. The meaning and context of the identified words are searched for in web data or past recorded data, and the search results are provided to the user's terminal in real time. Furthermore, the system transmits the search results as messages to an external interface and records them in a database or spreadsheet, allowing meeting participants to efficiently utilize the information.

[0006] "Audio data" refers to audio information including speech made during a conference.

[0007] "Text data" refers to information obtained by converting voice data into text.

[0008] "Unknown words or jargon" refers to technical terms that the meeting participants do not understand or that are used in a particular field.

[0009] "Means of identification" refers to technical methods for identifying and detecting unknown words or terminology.

[0010] "Search methods" refers to technical techniques for detecting the meaning and context of identified words from web data or historical record data.

[0011] "Web data" refers to public information and data that exists on the Internet.

[0012] "Past recorded data" refers to data that records information and statements used in previous meetings.

[0013] "User terminal" refers to a device such as a computer, smartphone, or tablet used by a user participating in a conference.

[0014] "Means of providing" refers to the technical method for displaying or notifying the acquired information on the user's terminal.

[0015] "Means of sending as a message to an external interface" refers to a technical method of sending a message using a communication means such as chat or email in order to convey the acquired information to the user.

[0016] "Means of recording in a database or spreadsheet" refers to a technological method of permanently storing captured information for future reference. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention relates to a real-time support system for technical terms and unknown words in online meetings. Each element of the system will be described in detail below.

[0039] First, to capture the audio data of users speaking during a meeting in real time, the user's device collects the audio data and converts it into text data using the Zoom API. The device accesses Zoom's live transcription function and specifies the meeting ID as an argument to obtain the content of the conversation as text information.

[0040] The captured text data is sent to a server, which analyzes it to identify unknown words and terminology. The server uses natural language processing (NLP) techniques to tokenize the text data and match it against a list of known words, including common and industry-standard words.

[0041] The identified unknown words are then processed by the server, which searches for their meaning and context from web data and past meeting records. The server queries web APIs to retrieve relevant information and searches past record databases for relevant context and previous usage examples.

[0042] Search results are provided from the server to the user's device. These results are formatted for user convenience and provided in real time. As a concrete example, consider the case where the word "ROI" is identified. The server retrieves the meaning of "ROI" (return on investment) from the web, adds detailed context based on usage examples from past meetings, and sends it to the user's device.

[0043] The server then sends the search results in the form of a message to an external interface, such as the Zoom chat, and simultaneously records this information in a database or spreadsheet using the Google Spreadsheet API, which is then saved for future reference, allowing for quick retrieval of past meeting information when needed.

[0044] As a concrete example, consider a situation where a user says during a meeting, "To maximize the ROI of this project, we first need to set KPIs." The speech is transcribed in real time, and the server identifies the words "ROI" and "KPI." The server then searches the internet and past meeting records for the meaning of these words, generates a message saying "ROI: Return on Investment, KPI: Key Performance Indicator," and sends it to the Zoom chat. This information is also recorded in a spreadsheet.

[0045] In this way, users can understand the meaning and context of technical terms in real time without disrupting the progress of the meeting, which can significantly improve the efficiency and understanding of meetings.

[0046] The processing flow will be explained below.

[0047] Step 1:

[0048] The device connects to the Zoom API to capture audio data in real time during the meeting, capturing what the user is saying as digital audio data on the spot.

[0049] Step 2:

[0050] The device converts the audio data it receives into text using Zoom's live transcription feature, which sends a request to Zoom using the meeting ID to convert the audio into text.

[0051] Step 3:

[0052] The terminal transmits the text data to the server, which then uploads the text data to the server via the network for analysis.

[0053] Step 4:

[0054] The server analyzes the received text data and uses natural language processing (NLP) technology to tokenize it, which divides the speech into individual words.

[0055] Step 5:

[0056] The server checks the tokenized words against a known word list, which contains common vocabulary and industry-standard terminology, to identify unknown words or technical terms.

[0057] Step 6:

[0058] The server queries web data and past record databases to find the meaning and context of the identified unknown words. It obtains relevant information from web APIs and gathers contextual information by searching past meeting records.

[0059] Step 7:

[0060] The server formats the collected information to generate semantic and contextual search results, including details such as word definitions and historical usage examples.

[0061] Step 8:

[0062] The server sends the formatted search results to the terminal. By sending the information to the user's terminal, the user can receive the information in real time.

[0063] Step 9:

[0064] The device will display the retrieved information to the user, and will also post the search results to the Zoom chat to share the information with other participants.

[0065] Step 10:

[0066] The server records the acquired information in a spreadsheet using the Google Spreadsheet API, so that the information can be referenced after the meeting.

[0067] Examples:

[0068] For example, if a user says during a meeting, "In order to maximize the ROI of this project, we first need to set KPIs," the process would go something like this:

[0069] 1. The device receives the speech and collects audio data using the Zoom API.

[0070] 2. The device converts the voice into text information and generates text data.

[0071] 3. The device sends the generated text data to the server.

[0072] 4. The server analyzes the received text data and identifies unknown words including "ROI" and "KPI."

[0073] 5. The server searches for the meaning and context of "ROI" and "KPI" from web data and historical record data.

[0074] 6. The server generates the results "ROI: Return on Investment" and "KPI: Key Performance Indicator."

[0075] 7. The server sends the generated results to the device, which displays the information to the user and posts it to the Zoom chat.

[0076] 8. The server records the obtained information in a spreadsheet using the Google Spreadsheet API.

[0077] In this way, users can understand the meaning and context of technical terms in real time without disrupting the flow of the meeting, allowing them to participate in the meeting efficiently.

[0078] Example 1

[0079] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0080] In online meetings, when technical terms or unfamiliar words appear, participants may be unable to keep up and the meeting progress may be delayed. Furthermore, when an unfamiliar word appears during a meeting, it is difficult to search for and understand it on the spot. This reduces the efficiency of the meeting and leads to insufficient understanding among participants. Furthermore, if there is no easy way to access the information later, it becomes difficult to reuse the information.

[0081] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0082] In this invention, the server includes a means for acquiring audio data of an online conference from a user's terminal in real time and converting the audio data into text data, a means for identifying unknown words or technical terms from the text data using natural language processing technology, and a means for searching for the meaning and context of the identified words from web data or past conference records. This allows users to grasp the meaning of technical terms and unknown words in real time, deepening their understanding without disrupting the progress of the conference. Furthermore, a means for sending search results as a message to an external interface and a means for recording them in a database or spreadsheet make it easy to reuse the same information later.

[0083] A "user terminal" is an electronic device such as a computer, tablet, or smartphone used by a user participating in an online conference.

[0084] "Audio data" refers to digital audio information recorded from an online conference conversation.

[0085] "Text data" is digital data that has been converted from voice data into character information.

[0086] "Natural language processing technology" is a technology for analyzing text data to identify unknown words and technical terms, and is a series of processing techniques including tokenization and morphological analysis.

[0087] "Unknown words or technical terms" are new words that are not included in known word lists or words that are technical and not commonly understood.

[0088] "Web data" refers to digital information that is publicly available on the Internet, including information that can be obtained through web pages or APIs.

[0089] "Past meeting records" refers to audio data and text data from previous meetings, the contents of which are stored in a database.

[0090] "Searching" is the act of referencing web data or past meeting records to find the meaning or context of a specified word.

[0091] "Formatting" refers to converting search results into something that is easy for users to understand, and organizing them into an appropriate format and structure.

[0092] "Providing in real time" refers to providing information immediately without waiting while users are conducting an online conference.

[0093] "Sending to an external interface" means sending the obtained information to another system or application, such as a messaging platform.

[0094] "Recording in a database or spreadsheet" means storing search results or meeting information in a digital format for future reference.

[0095] The present invention is a system that provides real-time support for technical terms and unknown words in online meetings. The system shown below is composed of multiple elements, including a user terminal, a server, an external interface, and a database or spreadsheet.

[0096] 1. Acquiring audio data and converting it to text

[0097] When a user starts an online meeting, the user's device captures the meeting's audio data in real time. Specifically, the user's device (PC, smartphone, tablet, etc.) uses the online meeting system to collect audio data. This device then converts the audio into text data using audio data capture technology such as the Zoom API. For example, the device accesses Zoom's live transcription function and converts the captured audio data into text in real time.

[0098] 2. Sending text data

[0099] The device sends the converted text data to the server using a communication protocol such as an HTTP POST request, where it is transferred to the server and prepared for analysis.

[0100] 3. Identifying Unknown Words and Terminology

[0101] The server processes the received text data using natural language processing (NLP) techniques, such as tokenizing the text and matching it against a list of known words to identify unknown words and terminology. This NLP includes morphological analysis, tokenization, and dictionary lookup of words.

[0102] 4. Searching for meaning and context

[0103] For the identified unknown words or terminology, the server searches for meaning and context from web APIs or a database of past meeting records. For example, the server may refer to web data or previous meeting records to clarify the meaning of the identified words. The program then performs a regular web search or internal resource search to extract relevant information.

[0104] 5. Providing search results

[0105] The server formats the search results for delivery to the user. Additionally, the server delivers the formatted information to the user's device in real time. This information is then sent as a message to an external interface, such as a Zoom chat. For example, the Zoom API can be used to post a description and context of the retrieved words to the chat. The same search results can also be saved to a database or spreadsheet using the Google Spreadsheet API for later reference.

[0106] Specific examples

[0107] For example, if a user says the following during a meeting:

[0108] "To maximize the ROI of this project, we first need to set KPIs."

[0109] The device captures this speech as audio data and converts it to text using the Zoom API:

[0110] "To maximize the ROI of this project, we first need to set KPIs."

[0111] The server identifies the words "ROI" and "KPI" and searches for their meanings in web data and past meeting records. For example, the server obtains results such as "ROI: Return on Investment" and "KPI: Key Performance Indicator."

[0112] The server sends the results as a message to the Zoom chat and simultaneously records them in a Google spreadsheet, providing real-time definitions of technical terms and supporting understanding during the meeting.

[0113] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0114] Step 1:

[0115] A user starts an online meeting. The user uses the online meeting system and is issued a meeting ID. The device starts collecting audio data via the Zoom API. The input is the trigger to start the meeting, and the output is the collected audio data.

[0116] Step 2:

[0117] The device uses Zoom's live transcription feature to convert audio data into text data in real time, directly using the Zoom API. The input is the collected audio data, and the output is the converted text data.

[0118] Step 3:

[0119] The terminal sends the converted text data to the server using an HTTP POST request. The input is text data, and the output is the transfer of the text data to the server.

[0120] Step 4:

[0121] The server analyzes the received text data using NLP techniques. Specifically, it tokenizes the text data and compares it with a list of known words to identify unknown words and technical terms. The input is the transmitted text data, and the output is the identified unknown words and technical terms.

[0122] Step 5:

[0123] The server searches for the meaning and context of the identified words from web data and a database of past meeting records. It uses a web API to obtain related information and reference past records. The input is the identified word, and the output is information about the meaning and context of that word.

[0124] Step 6:

[0125] The server formats the information it receives and converts it into something that is easy for the user to understand, for example, "ROI: Return on Investment." The input is the semantic and contextual information it receives, and the output is the formatted information.

[0126] Step 7:

[0127] The server delivers the formatted information to the user's device in real time. This information is sent as a message to an external interface such as Zoom chat. Furthermore, the information is recorded in a database or spreadsheet using the Google Spreadsheet API. The input is the formatted information, and the output is sending a message to Zoom chat and recording it in the database.

[0128] Specific examples of behavior:

[0129] For example, consider the case where a user says during a meeting, "To maximize the ROI of this project, we first need to set KPIs."

[0130] Step 1: A user starts a conference and audio data collection begins.

[0131] Step 2: The device translates the speech into "To maximize the ROI of this project, we first need to set KPIs."

[0132] Step 3: The device sends this text data to the server.

[0133] Step 4: The server identifies the words "ROI" and "KPI."

[0134] Step 5: The server obtains information such as "ROI: Return on Investment" and "KPI: Key Performance Indicator" from web data and past meeting records.

[0135] Step 6: The server formats this information.

[0136] Step 7: The server sends the formatted information to the Zoom chat and records it in a Google spreadsheet.

[0137] (Application example 1)

[0138] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0139] When a large amount of technical terms and unfamiliar words are used in online meetings or virtual stores, participants and customers often cannot understand their meaning or context, resulting in a decrease in communication efficiency. This can also impair the customer's purchasing experience, so a solution is needed.

[0140] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0141] In this invention, the server includes means for converting speech data acquired in real time into text data, means for identifying unknown words or technical terms from the text data, means for searching for the meaning and context of the identified words from web data or past recorded data, means for providing the search results to the user's terminal in real time, means for providing customer support functions in a virtual store, and means for overlaying related information on an interface used by the customer. This enables users and customers to understand the meaning of technical terms and unknown words used during meetings or while shopping in real time, thereby improving communication efficiency and the purchasing experience.

[0142] The "means for converting voice data acquired in real time into text data" refers to a technical means for instantly converting the voice spoken by a user into text information.

[0143] "Means for identifying unknown words or technical terms from text data" refers to technical means for analyzing acquired character information to find words that are not generally recognized or technical terms used in a specific field.

[0144] "Means for searching for the meaning and context of identified words from web data or historical records" refers to technical means for searching for the meaning and usage examples of identified unknown words or technical terms from data on the Internet or historical records.

[0145] The "means for providing search results to a user's terminal in real time" refers to a technical means for instantly displaying the obtained search results on the device used by the user.

[0146] "Means for providing customer support functions in virtual stores" refers to technical means for providing support functions to enable customers to enjoy shopping comfortably in stores operated online.

[0147] The "means for overlaying and displaying relevant information on the interface used by the customer" refers to a technical means for overlaying and displaying relevant information on the display device used by the customer.

[0148] To implement this invention, the following system configuration is required. First, the system collects voice data uttered by the user in real time and converts it into text data. This is done using the Google Speech Recognition API and the speech_recognition library. This voice data can be obtained from various input devices used by the user, such as smartphones, smart glasses, and head-mounted displays.

[0149] Next, the server analyzes the retrieved text data to identify unknown words and terminology. This step uses natural language processing (NLP) techniques to tokenize the text data and match it with a list of known words. For each unknown word identified, the server searches for its meaning and context from web data and historical record data, and sends a query to a web API to retrieve its meaning. It also searches historical record databases for related context and previous usage examples.

[0150] Specifically, when words such as "ROI" (return on investment) or "KPI" (key performance indicator) are identified, the server retrieves their meanings from the Internet and past meeting records to add detailed context. In a virtual store, if a customer says, "I want to know what the ROI of this product is," the system retrieves the meaning of "ROI" and related information in real time and displays it as an overlay on the customer's interface.

[0151] This allows customers to understand the meaning of technical terms and unknown words used in the virtual store in real time, improving the shopping experience. The server then sends the search results in the form of messages to an external interface, which can include, for example, the virtual store's chat function. At the same time, the information is recorded in a database or spreadsheet using the Google Spreadsheet API for future reference.

[0152] Examples of prompt sentences include:

[0153] "What is ROI?"

[0154] "I want to know how this product's ROI will be affected."

[0155] It is entered in the form:

[0156] This allows you to understand the meaning and context of technical terms and unknown words in real time, significantly improving meeting efficiency and purchasing experiences.

[0157] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0158] Step 1:

[0159] The device collects the user's voice data and acquires it in real time. The device can be a smartphone, smart glasses, or a head-mounted display. The input is the user's voice data, and the output is the acquired voice data. The collected voice data is passed to the Google Speech Recognition API, where it is prepared to be converted into text data.

[0160] Step 2:

[0161] The device converts the acquired voice data into text data using the Google Speech Recognition API. This process uses the speech_recognition library. The input is voice data, and the output is the converted text data. This text data is analyzed in the next step.

[0162] Step 3:

[0163] The server receives the text data sent from the terminal and tokenizes it using natural language processing (NLP) technology. The input is the text data, and the output is a tokenized word list. This process separates each word in the text.

[0164] Step 4:

[0165] The server matches the tokenized word list against a known word list to identify unknown words or terminology. The input is the tokenized word list, and the output is a list of identified unknown words. The known word list includes common words and industry-standard words.

[0166] Step 5:

[0167] The server searches for the meanings of identified unknown words from web data and historical data. This process involves sending queries using external web APIs to retrieve information. It also searches historical databases for relevant contexts and usage examples. The input is a list of identified unknown words, and the output is the meaning and context information for each word.

[0168] Step 6:

[0169] The server formats the acquired semantic and contextual information into a message and provides it to the user's device in real time. The input is the semantic and contextual information of each word, and the output is a formatted message, which is overlaid on the device's interface.

[0170] Step 7:

[0171] The server sends the search results as a message to an external interface, for example, a chat function of a virtual store. The input is a formatted message, and the output is a message displayed in the chat interface used by the user.

[0172] Step 8:

[0173] The server simultaneously records the retrieved information in a database or spreadsheet using the Google Spreadsheet API. The input is a formatted message, and the output is a recorded database or spreadsheet. This allows for quick retrieval of the information later.

[0174] Step 9:

[0175] It understands the meaning of technical terms and unknown words in real time based on the relevant information provided by the user through the interface. The input is a formatted message, and the output is improved user comprehension, thereby improving the user's shopping experience in the virtual store.

[0176] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0177] This invention combines a real-time support system for technical terms and unknown words in online meetings with an emotion engine that recognizes the user's emotions and optimizes the provision of support information. Each element of the system is described in detail below.

[0178] First, to obtain real-time voice data from users during a meeting, the user's device collects the voice data and converts it into text data using a voice recognition function. The device connects to a voice recognition API and converts the voice data into text information in real time.

[0179] The captured text data is sent to a server, which analyzes it to identify unknown words and terminology. The server uses natural language processing (NLP) techniques to tokenize the text data and match it against a list of known words, including common and industry-standard words.

[0180] The identified unknown words are then processed by the server, which searches for their meaning and context from web data and historical records. The server collects semantic and contextual information by querying web APIs to retrieve relevant information and searching historical databases for relevant context and previous usage examples.

[0181] The search results are delivered to the user's device from the server, and are displayed in real time in a user-friendly format, including detailed information such as word definitions and past usage examples. The most important information is posted to the Zoom chat and shared with other participants.

[0182] Furthermore, the system is equipped with an emotion engine that recognizes emotions from the user's speech and facial expressions. The device inputs the user's facial expression data and voice tone into an emotion analysis module to identify the user's current emotional state. The server combines this emotional information with the importance of words to automatically adjust the priority of information provision.

[0183] A specific use case is when a user feels stressed during a meeting. For example, if the user says, "To maximize the ROI of this project, we first need to set KPIs," and an emotion indicating stress is recognized, the server identifies the words "ROI" and "KPI" and searches for their meaning and context, while recognizing the user's stress level and adjusting the priority of information provision. If an emotion indicating stress is recognized, the server provides detailed assistance information and sends additional messages to help the user relax.

[0184] Additionally, search results are recorded in a spreadsheet using the Google Sheets API, and the records are saved for future reference, allowing users to quickly search for past meeting information when needed.

[0185] In this way, users can understand the meaning and context of technical terms in real time without disrupting the flow of the meeting, while receiving optimal support according to their emotional state, significantly improving meeting efficiency and understanding.

[0186] The processing flow will be explained below.

[0187] Step 1:

[0188] The device captures audio data during the meeting in real time, connects to a speech recognition API, and collects the user's speech as digital audio data.

[0189] Step 2:

[0190] The device converts the acquired voice data into text data in real time, allowing the spoken content to be recorded as text information.

[0191] Step 3:

[0192] The terminal transmits the converted text data to the server, which then sends the text data over the network for analysis on the server side.

[0193] Step 4:

[0194] The server analyzes the received text data and uses natural language processing (NLP) technology to tokenize it, dividing the speech into words.

[0195] Step 5:

[0196] The server checks the tokenized words against a known word list, which includes common vocabulary and industry-standard terminology, to identify unknown words or technical terms.

[0197] Step 6:

[0198] The server queries web data and historical record databases to find the meaning and context of the identified unknown words, using Web APIs to retrieve relevant information and collect relevant contextual information from the historical record database.

[0199] Step 7:

[0200] The server formats the collected information and generates search results that include semantic and contextual information, optionally adding details such as word definitions and previous usage examples.

[0201] Step 8:

[0202] The server provides the results to the user's device, which displays the information to the user in real time and can also post it to the Zoom chat.

[0203] Step 9:

[0204] The device inputs the user's facial expression data and voice tone into an emotion engine to recognize the user's emotional state, which then analyzes the user's emotions in real time.

[0205] Step 10:

[0206] The server automatically adjusts the priority of information provision based on the recognized emotional state. If the user is feeling stressed, detailed assistance information and relaxation messages will be provided first.

[0207] Step 11:

[0208] The server records the search results in a Google Spreadsheet, using the Google Spreadsheet API to store the retrieved information for future reference.

[0209] Examples:

[0210] Let's take the example of a user saying during a meeting, "To maximize the ROI of this project, we first need to set KPIs."

[0211] 1. The device receives the speech and collects voice data using a speech recognition API.

[0212] 2. The device converts the voice data into text data in real time.

[0213] 3. The device sends the generated text data to the server.

[0214] 4. The server analyzes the received text data and identifies the unknown words "ROI" and "KPI."

[0215] 5. The server searches for the meaning and context of "ROI" and "KPI" from web data and historical record databases.

[0216] 6. The server generates the results "ROI: Return on Investment" and "KPI: Key Performance Indicator."

[0217] 7. The server sends the results to the user's device, which displays the information in real time and also posts it to the Zoom chat.

[0218] 8. The device inputs the user's facial expression data and voice tone into an emotion engine to analyze the user's emotional state.

[0219] 9. The server adjusts the priority of information provision based on the recognized emotional state. If the user is feeling stressed, it will provide detailed assistance information and also send a relaxation message.

[0220] 10. The server records the search results in a spreadsheet using the Google Spreadsheet API.

[0221] In this way, users can understand the meaning and context of technical terms in real time without disrupting the flow of the meeting, while receiving optimal support according to their emotional state, significantly improving meeting efficiency and understanding.

[0222] Example 2

[0223] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0224] When online meetings contain a large amount of technical terms and unfamiliar words, participants may find it difficult to understand them, reducing the efficiency of the meeting and the level of understanding of the content. Furthermore, participants' emotional states can affect their comprehension and concentration, but existing systems lack support that takes this into account. Therefore, there is a need for a system that can recognize users' emotions and provide optimal information, along with real-time support for technical terms and unfamiliar words.

[0225] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0226] In this invention, the server includes means for converting speech data acquired in real time into text data, means for identifying unknown words or technical terms from the text data, means for searching for the meaning and context of the identified words from web data or past recorded data, means for detecting the user's emotions, and means for adjusting the order in which search results are presented based on the emotions. This makes it easier for the user to understand the meaning of technical terms and unknown words in real time, and enables the user to receive information optimal for their emotional state.

[0227] "Voice data" refers to information that includes a user's speech or voice, expressed in digital form.

[0228] "Text data" is voice data converted into text information, and is data written in natural language.

[0229] "Unknown words" are words or phrases that are not included in a predefined word list and require context and semantic understanding.

[0230] "Jargon" refers to specialized words and phrases used in a particular field or industry that are necessary for understanding that field.

[0231] "Web data" is information publicly available on the Internet and is data obtained using natural language processing and other technologies.

[0232] "Historical data" refers to information collected or generated in the past or data stored in a database that is used for reference or retrieval.

[0233] "Search results" are data containing information about the meaning and context of identified words that are provided to the user.

[0234] A "terminal" is a computing device used by a user to collect voice data, display text data, and perform other processing.

[0235] A "server" is a computing device that provides services over a network, such as analyzing, retrieving, generating, and providing data.

[0236] "Emotion" refers to the user's mental state or psychological reaction, and indicates states such as stress, relaxation, and concentration.

[0237] "Means for detecting emotions" refers to technology or devices that analyze the user's voice and facial expression data to identify their emotional state.

[0238] "Means for adjusting the order in which search results are presented" refers to techniques or methods for determining the priority of information to be presented based on the user's emotional state or other factors.

[0239] The present invention provides a real-time support system for technical terms and unknown words in online meetings, and further combines it with an emotion engine that recognizes the user's emotions and optimizes the provision of support information. An embodiment of this system will be described in detail.

[0240] First, the user's device collects audio data in real time to capture the user's speech during the meeting. A microphone is connected to the device to capture what is being said during the meeting. Next, the device uses a speech recognition API (e.g., Google Speech-to-Text API) to convert the collected audio data into text data. The audio data is sent to the API, and the returned text data is obtained.

[0241] The captured text data is sent from the device to a server, where it is analyzed. The server first tokenizes the text data (for example, breaking down "To maximize the ROI of this project, we first need to set KPIs" into individual words) and identifies unknown words and technical terms by comparing them with a list of known words. This list of known words includes pre-defined common words and industry-standard words.

[0242] The server then searches for the meaning and context of the identified unknown words using a Web API (e.g., Google Knowledge Graph API), and also searches for related contexts and previous usage examples from a historical record database to gather semantic and contextual information, resulting in the meaning and usage examples of words such as "ROI" and "KPI."

[0243] The search results are delivered to the user's device in real time from the server. Specifically, detailed text including word definitions, past usage examples, and related information is sent to the user's device. Important information is shared with other participants using the chat function of the meeting app (e.g., Zoom).

[0244] The system is also equipped with an emotion engine that recognizes emotions from the user's speech and facial expression data. The device uses a camera and microphone to collect the user's facial expression data and voice tone, which are then input into an emotion analysis module. The server analyzes this data to identify the user's emotional state. For example, a deep learning model can be used to generate emotion labels such as "stress," "relaxed," and "concentrated."

[0245] The server adjusts the order of search results based on the identified emotional information and the importance of unknown words. If the user is feeling stressed, detailed explanations and relaxation messages will be provided first. In this way, users can easily understand the meaning of technical terms and unknown words in real time during meetings and receive the most appropriate information according to their emotional state.

[0246] Finally, the search results are recorded in a spreadsheet using the Google Spreadsheet API from the server, allowing users to quickly reference past meeting information and avoid having to search for the same information again if necessary.

[0247] As a concrete example, suppose a user says during a meeting, "To maximize the ROI of this project, we first need to set KPIs." The device collects this speech as voice data and uses a speech recognition API to convert it into text data: "To maximize the ROI of this project, we first need to set KPIs." This text data is sent to the server, where "ROI" and "KPI" are identified as unknown words. The server searches for their meaning and context and provides detailed information to the user's device. If the user is recognized as feeling stressed, detailed support information and a relaxation message are sent. This information is recorded in a Google spreadsheet and can be referenced later.

[0248] An example prompt might be, "To get the most out of your real-time support during the meeting, please explain specifically what your ROI and KPIs mean. Also, please elaborate on other metrics that are important to the success of this project."

[0249] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0250] Step 1: Collect speech data and send it to the speech recognition API

[0251] Audio data of users speaking during a meeting is collected in real time using a microphone connected to the user's device. The device then sends the collected audio data to a speech recognition API (e.g., Google Speech-to-Text API). The input is real-time audio data, and the output is text data returned by the speech recognition API.

[0252] Step 2: Convert the audio data to text

[0253] The device receives the text data returned from the speech recognition API. Specific operations include processing HTTP requests to the speech recognition API and responses. The input is audio data from the speech recognition API, and the output is text data.

[0254] Step 3: Send the text data to the server

[0255] The terminal sends the converted text data to the server using an HTTP POST request, where the input is the text data and the output is a confirmation of transmission to the server.

[0256] Step 4: Tokenize the text data

[0257] The server tokenizes the received text data. Specifically, it splits the text into words. The input is the text data, and the output is a tokenized word list.

[0258] Step 5: Identify unknown words and terminology

[0259] The server checks the tokenized word list against a list of known words to identify unknown words or terminology. Specific operations include database query comparisons against the known word list. The input is the tokenized word list, and the output is the unknown word list.

[0260] Step 6: Search for the meaning and context of unknown words

[0261] The server queries the identified unknown words to a Web API (e.g., Google Knowledge Graph API) and an internal database to find their meaning and context. Specific operations include generating and sending API queries and analyzing the retrieved data. The input is a list of unknown words, and the output is semantic and context information.

[0262] Step 7: Format the search results and send them to your device

[0263] The server formats the semantic and contextual information it has obtained into a form that is easy for the user to use. Specific operations include converting it to HTML or text format. The input is the semantic and contextual information, and the output is the formatted information. The server then sends the formatted information to the device. The input is the formatted information, and the output is a confirmation of transmission to the device.

[0264] Step 8: Collect emotional data and identify emotional states

[0265] The device inputs the user's facial expression data and voice tone into the emotion analysis module. Specific operations include collecting data using a camera and microphone and analyzing it using an emotion analysis algorithm. The input is facial expression data and voice tone, and the output is the user's emotional state.

[0266] Step 9: Adjust the order of information based on emotional information

[0267] The server combines the acquired emotional information with the importance information of unknown words and automatically adjusts the order in which search results are presented. Specific operations include prioritizing the results based on the emotional state and the importance of words. The input is emotional information and importance information, and the output is the adjusted order in which the results are presented.

[0268] Step 10: Send search results to an external interface and record them in a spreadsheet

[0269] The server sends the search results as a message to an external interface (e.g., the chat function of a conference app). It then records the search results in a spreadsheet using the Google Spreadsheet API. Specific operations include sending and recording data using the API. The input is the search results, and the output is confirmation of sending to the external interface and recording in the spreadsheet.

[0270] (Application example 2)

[0271] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0272] When a driver in an autonomous vehicle encounters technical terms or unfamiliar words while driving, it is difficult to provide the necessary information for driving due to a lack of meaning and context information. It is also difficult to provide information at the appropriate time based on the driver's emotional state. This can lead to a decrease in driving safety and efficiency.

[0273] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for converting voice data acquired in real time into text data, means for identifying unknown words or technical terms from the text data, means for searching for the meaning and context of the identified words from web data or past recorded data, means for providing the search results to the user's terminal in real time, means for analyzing facial expression data or voice tone to recognize the driver's emotional state, and means for adjusting the priority of information provision based on the recognized emotional state. This makes it possible to provide information at the optimal timing according to the driver's emotional state while supporting the driver's understanding of technical terms and unknown words.

[0274] "Audio data" is a digital representation of an audio signal captured in real time.

[0275] "Text data" refers to data that has been converted into character information based on voice data.

[0276] "Unknown words" refer to technical terms or new terms that are not generally understood by users.

[0277] "Terminology" refers to specialized terms used in a particular industry or field.

[0278] "Web data" is data that includes information on the Internet.

[0279] "Previously recorded data" is data that has been previously acquired or stored.

[0280] "Providing in real time" means providing information instantly, without delay.

[0281] "User terminal" refers to the device used by a User.

[0282] "Emotional state" refers to the psychological state of a user as perceived from their facial expression and tone of voice.

[0283] "Facial expression data" refers to data relating to the facial expression of the user captured by a camera or the like.

[0284] "Voice tone" refers to the voice characteristics obtained from the intonation and stress of the voice.

[0285] "Adjusting the priority of information provision" means providing information at an appropriate time depending on the importance and urgency of the information.

[0286] An "external interface" is an interface for connecting a system to external equipment or devices.

[0287] An "assist message" is a supplementary message to support or encourage the user.

[0288] A "database" is a system for systematically organizing and storing data.

[0289] A spreadsheet is a tool for managing, analyzing, and displaying data in rows and columns.

[0290] This invention is a system that provides real-time support to drivers of self-driving vehicles when they encounter technical terms or unknown words while driving. The system collects and analyzes the user's voice data and facial expression data to provide appropriate support information.

[0291] System configuration

[0292] The system consists of the following hardware and software components:

[0293] Hardware:

[0294] Microphone: Collects driver voice inside the vehicle.

[0295] Camera: Collects driver facial expression data.

[0296] On-board computer: processes data and provides information.

[0297] software:

[0298] Speech recognition API (e.g., Google Speech-to-Text API): Converts voice data into text data in real time.

[0299] Natural Language Processing (NLP) engines (e.g., spaCy, NLTK): Analyze text data and identify terminology and unknown words.

[0300] Emotion recognition module (e.g., Face API, Microsoft Azure Emotion API): Analyzes facial expression data and voice tone to recognize the driver's emotional state.

[0301] Real-time data collection and processing system: Collects voice and facial expression data and analyzes it in real time.

[0302] Program processing

[0303] The server collects voice data in real time and converts it into text using a speech recognition API. It then analyzes the text using an NLP engine to identify unknown words and technical terms. The meaning and context of the identified words are obtained from web data and historically recorded data. This meaning and context information is then provided to the user's device in real time.

[0304] Furthermore, the driver's facial expression data and voice tone captured by the camera are input into the emotion recognition module to recognize the driver's emotional state. The server adjusts the priority of information provision based on the recognized emotional state. If emotions indicating stress are recognized, detailed assistance information and relaxation messages will be provided.

[0305] For example, if a driver hears the term "ADAS (Advanced Driver Assistance Systems)" for the first time while driving, the system will respond as follows:

[0306] 1. Listen to the driver's voice and convert it into text.

[0307] 2. Identify "ADAS" using an NLP engine.

[0308] 3. Obtain the definition of "ADAS" and related information from the Web API.

[0309] 4. Cameras analyze the driver's facial expressions to identify their emotional state.

[0310] 5. Display information on the display and add assistance messages as needed.

[0311] Prompt Sentence Examples

[0312] A driver in a self-driving vehicle encounters the word "ADAS" for the first time. How does the system respond in the next step?

[0313] 1. Listen to the driver's voice and convert it into text.

[0314] 2. Identify "ADAS" using the NLP engine.

[0315] 3. Get the meaning of "ADAS" from the Web API.

[0316] 4. Analyze the driver's emotional state using a camera.

[0317] 5. Information is displayed on the screen, and if the driver is feeling stressed, a message to relax is also displayed.

[0318] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0319] Step 1:

[0320] The user's voice is collected by a microphone. The input is real-time voice data, and the output is digitized voice data. The device sends this voice data to a speech recognition API (e.g., Google Speech-to-Text API) to convert it into text information. The output is text data.

[0321] Step 2:

[0322] The device sends text data obtained from the speech recognition API to the server, which then analyzes the text data using a natural language processing (NLP) engine (e.g., spaCy, NLTK). The input is text data, and the output is a tokenized word list. The server identifies unknown words and technical terms from this list.

[0323] Step 3:

[0324] The server searches for the meaning and context of identified unknown words and technical terms from web data or historical record data. The input is a list of identified words, and the output is the meaning and context information of each word. The server uses a web API to obtain related information and searches for related context from the historical record database.

[0325] Step 4:

[0326] The camera collects the driver's facial expression data in real time. The input is real-time facial expression data, and the output is digitized facial expression data. The device sends this facial expression data to an emotion recognition module (e.g., Face API, Microsoft Azure Emotion API) for analysis. The output is the driver's emotional state.

[0327] Step 5:

[0328] The server integrates the emotional state obtained from the emotion recognition module with the semantic and contextual information of the words obtained in step 3. The input is the emotional state and semantic / contextual information, and the output is a list of information provision with adjusted priorities. The server automatically adjusts the priority of information provision according to the emotional state.

[0329] Step 6:

[0330] The server sends the prioritized information list to the terminal. The terminal displays the meaning of words and context information in real time on the display. If the driver shows signs of stress or anxiety, the terminal also displays a relaxation message. The input is the information list, and the output is the displayed information.

[0331] Prompt Sentence Examples

[0332] A driver in a self-driving vehicle encounters the word "ADAS" for the first time. How does the system respond in the next step?

[0333] 1. Listen to the driver's voice and convert it into text.

[0334] 2. Identify "ADAS" using the NLP engine.

[0335] 3. Get the meaning of "ADAS" from the Web API.

[0336] 4. Analyze the driver's emotional state using a camera.

[0337] 5. Information is displayed on the screen, and if the driver is feeling stressed, a message to relax is also displayed.

[0338] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0339] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0340] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0341] [Second embodiment]

[0342] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0343] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0344] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0345] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0346] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0347] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0348] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0349] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0350] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0351] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0352] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0353] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0354] The present invention relates to a real-time support system for technical terms and unknown words in online meetings. Each element of the system will be described in detail below.

[0355] First, to capture the audio data of users speaking during a meeting in real time, the user's device collects the audio data and converts it into text data using the Zoom API. The device accesses Zoom's live transcription function and specifies the meeting ID as an argument to obtain the content of the conversation as text information.

[0356] The captured text data is sent to a server, which analyzes it to identify unknown words and terminology. The server uses natural language processing (NLP) techniques to tokenize the text data and match it against a list of known words, including common and industry-standard words.

[0357] The identified unknown words are then processed by the server, which searches for their meaning and context from web data and past meeting records. The server queries web APIs to retrieve relevant information and searches past record databases for relevant context and previous usage examples.

[0358] Search results are provided from the server to the user's device. These results are formatted for user convenience and provided in real time. As a concrete example, consider the case where the word "ROI" is identified. The server retrieves the meaning of "ROI" (return on investment) from the web, adds detailed context based on usage examples from past meetings, and sends it to the user's device.

[0359] The server then sends the search results in the form of a message to an external interface, such as the Zoom chat, and simultaneously records this information in a database or spreadsheet using the Google Spreadsheet API, which is then saved for future reference, allowing for quick retrieval of past meeting information when needed.

[0360] As a concrete example, consider a situation where a user says during a meeting, "To maximize the ROI of this project, we first need to set KPIs." The speech is transcribed in real time, and the server identifies the words "ROI" and "KPI." The server then searches the internet and past meeting records for the meaning of these words, generates a message saying "ROI: Return on Investment, KPI: Key Performance Indicator," and sends it to the Zoom chat. This information is also recorded in a spreadsheet.

[0361] In this way, users can understand the meaning and context of technical terms in real time without disrupting the progress of the meeting, which can significantly improve the efficiency and understanding of meetings.

[0362] The processing flow will be explained below.

[0363] Step 1:

[0364] The device connects to the Zoom API to capture audio data in real time during the meeting, capturing what the user is saying as digital audio data on the spot.

[0365] Step 2:

[0366] The device converts the audio data it receives into text using Zoom's live transcription feature, which sends a request to Zoom using the meeting ID to convert the audio into text.

[0367] Step 3:

[0368] The terminal transmits the text data to the server, which then uploads the text data to the server via the network for analysis.

[0369] Step 4:

[0370] The server analyzes the received text data and uses natural language processing (NLP) technology to tokenize it, which divides the speech into individual words.

[0371] Step 5:

[0372] The server checks the tokenized words against a known word list, which contains common vocabulary and industry-standard terminology, to identify unknown words or technical terms.

[0373] Step 6:

[0374] The server queries web data and past record databases to find the meaning and context of the identified unknown words. It obtains relevant information from web APIs and gathers contextual information by searching past meeting records.

[0375] Step 7:

[0376] The server formats the collected information to generate semantic and contextual search results, including details such as word definitions and historical usage examples.

[0377] Step 8:

[0378] The server sends the formatted search results to the terminal. By sending the information to the user's terminal, the user can receive the information in real time.

[0379] Step 9:

[0380] The device will display the retrieved information to the user, and will also post the search results to the Zoom chat to share the information with other participants.

[0381] Step 10:

[0382] The server records the acquired information in a spreadsheet using the Google Spreadsheet API, so that the information can be referenced after the meeting.

[0383] Examples:

[0384] For example, if a user says during a meeting, "In order to maximize the ROI of this project, we first need to set KPIs," the process would go something like this:

[0385] 1. The device receives the speech and collects audio data using the Zoom API.

[0386] 2. The device converts the voice into text information and generates text data.

[0387] 3. The device sends the generated text data to the server.

[0388] 4. The server analyzes the received text data and identifies unknown words including "ROI" and "KPI."

[0389] 5. The server searches for the meaning and context of "ROI" and "KPI" from web data and historical record data.

[0390] 6. The server generates the results "ROI: Return on Investment" and "KPI: Key Performance Indicator."

[0391] 7. The server sends the generated results to the device, which displays the information to the user and posts it to the Zoom chat.

[0392] 8. The server records the obtained information in a spreadsheet using the Google Spreadsheet API.

[0393] In this way, users can understand the meaning and context of technical terms in real time without disrupting the flow of the meeting, allowing them to participate in the meeting efficiently.

[0394] Example 1

[0395] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0396] In online meetings, when technical terms or unfamiliar words appear, participants may be unable to keep up and the meeting progress may be delayed. Furthermore, when an unfamiliar word appears during a meeting, it is difficult to search for and understand it on the spot. This reduces the efficiency of the meeting and leads to insufficient understanding among participants. Furthermore, if there is no easy way to access the information later, it becomes difficult to reuse the information.

[0397] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0398] In this invention, the server includes a means for acquiring audio data of an online conference from a user's terminal in real time and converting the audio data into text data, a means for identifying unknown words or technical terms from the text data using natural language processing technology, and a means for searching for the meaning and context of the identified words from web data or past conference records. This allows users to grasp the meaning of technical terms and unknown words in real time, deepening their understanding without disrupting the progress of the conference. Furthermore, a means for sending search results as a message to an external interface and a means for recording them in a database or spreadsheet make it easy to reuse the same information later.

[0399] A "user terminal" is an electronic device such as a computer, tablet, or smartphone used by a user participating in an online conference.

[0400] "Audio data" refers to digital audio information recorded from an online conference conversation.

[0401] "Text data" is digital data that has been converted from voice data into character information.

[0402] "Natural language processing technology" is a technology for analyzing text data to identify unknown words and technical terms, and is a series of processing techniques including tokenization and morphological analysis.

[0403] "Unknown words or technical terms" are new words that are not included in known word lists or words that are technical and not commonly understood.

[0404] "Web data" refers to digital information that is publicly available on the Internet, including information that can be obtained through web pages or APIs.

[0405] "Past meeting records" refers to audio data and text data from previous meetings, the contents of which are stored in a database.

[0406] "Searching" is the act of referencing web data or past meeting records to find the meaning or context of a specified word.

[0407] "Formatting" refers to converting search results into something that is easy for users to understand, and organizing them into an appropriate format and structure.

[0408] "Providing in real time" refers to providing information immediately without waiting while users are conducting an online conference.

[0409] "Sending to an external interface" means sending the obtained information to another system or application, such as a messaging platform.

[0410] "Recording in a database or spreadsheet" means storing search results or meeting information in a digital format for future reference.

[0411] The present invention is a system that provides real-time support for technical terms and unknown words in online meetings. The system shown below is composed of multiple elements, including a user terminal, a server, an external interface, and a database or spreadsheet.

[0412] 1. Acquiring audio data and converting it to text

[0413] When a user starts an online meeting, the user's device captures the meeting's audio data in real time. Specifically, the user's device (PC, smartphone, tablet, etc.) uses the online meeting system to collect audio data. This device then converts the audio into text data using audio data capture technology such as the Zoom API. For example, the device accesses Zoom's live transcription function and converts the captured audio data into text in real time.

[0414] 2. Sending text data

[0415] The device sends the converted text data to the server using a communication protocol such as an HTTP POST request, where it is transferred to the server and prepared for analysis.

[0416] 3. Identifying Unknown Words and Terminology

[0417] The server processes the received text data using natural language processing (NLP) techniques, such as tokenizing the text and matching it against a list of known words to identify unknown words and terminology. This NLP includes morphological analysis, tokenization, and dictionary lookup of words.

[0418] 4. Searching for meaning and context

[0419] For the identified unknown words or terminology, the server searches for meaning and context from web APIs or a database of past meeting records. For example, the server may refer to web data or previous meeting records to clarify the meaning of the identified words. The program then performs a regular web search or internal resource search to extract relevant information.

[0420] 5. Providing search results

[0421] The server formats the search results for delivery to the user. Additionally, the server delivers the formatted information to the user's device in real time. This information is then sent as a message to an external interface, such as a Zoom chat. For example, the Zoom API can be used to post a description and context of the retrieved words to the chat. The same search results can also be saved to a database or spreadsheet using the Google Spreadsheet API for later reference.

[0422] Specific examples

[0423] For example, if a user says the following during a meeting:

[0424] "To maximize the ROI of this project, we first need to set KPIs."

[0425] The device captures this speech as audio data and converts it to text using the Zoom API:

[0426] "To maximize the ROI of this project, we first need to set KPIs."

[0427] The server identifies the words "ROI" and "KPI" and searches for their meanings in web data and past meeting records. For example, the server obtains results such as "ROI: Return on Investment" and "KPI: Key Performance Indicator."

[0428] The server sends the results as a message to the Zoom chat and simultaneously records them in a Google spreadsheet, providing real-time definitions of technical terms and supporting understanding during the meeting.

[0429] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0430] Step 1:

[0431] A user starts an online meeting. The user uses the online meeting system and is issued a meeting ID. The device starts collecting audio data via the Zoom API. The input is the trigger to start the meeting, and the output is the collected audio data.

[0432] Step 2:

[0433] The device uses Zoom's live transcription feature to convert audio data into text data in real time, directly using the Zoom API. The input is the collected audio data, and the output is the converted text data.

[0434] Step 3:

[0435] The terminal sends the converted text data to the server using an HTTP POST request. The input is text data, and the output is the transfer of the text data to the server.

[0436] Step 4:

[0437] The server analyzes the received text data using NLP techniques. Specifically, it tokenizes the text data and compares it with a list of known words to identify unknown words and technical terms. The input is the transmitted text data, and the output is the identified unknown words and technical terms.

[0438] Step 5:

[0439] The server searches for the meaning and context of the identified words from web data and a database of past meeting records. It uses a web API to obtain related information and reference past records. The input is the identified word, and the output is information about the meaning and context of that word.

[0440] Step 6:

[0441] The server formats the information it receives and converts it into something that is easy for the user to understand, for example, "ROI: Return on Investment." The input is the semantic and contextual information it receives, and the output is the formatted information.

[0442] Step 7:

[0443] The server delivers the formatted information to the user's device in real time. This information is sent as a message to an external interface such as Zoom chat. Furthermore, the information is recorded in a database or spreadsheet using the Google Spreadsheet API. The input is the formatted information, and the output is sending a message to Zoom chat and recording it in the database.

[0444] Specific examples of behavior:

[0445] For example, consider the case where a user says during a meeting, "To maximize the ROI of this project, we first need to set KPIs."

[0446] Step 1: A user starts a conference and audio data collection begins.

[0447] Step 2: The device translates the speech into "To maximize the ROI of this project, we first need to set KPIs."

[0448] Step 3: The device sends this text data to the server.

[0449] Step 4: The server identifies the words "ROI" and "KPI."

[0450] Step 5: The server obtains information such as "ROI: Return on Investment" and "KPI: Key Performance Indicator" from web data and past meeting records.

[0451] Step 6: The server formats this information.

[0452] Step 7: The server sends the formatted information to the Zoom chat and records it in a Google spreadsheet.

[0453] (Application example 1)

[0454] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0455] When a large amount of technical terms and unfamiliar words are used in online meetings or virtual stores, participants and customers often cannot understand their meaning or context, resulting in a decrease in communication efficiency. This can also impair the customer's purchasing experience, so a solution is needed.

[0456] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0457] In this invention, the server includes means for converting speech data acquired in real time into text data, means for identifying unknown words or technical terms from the text data, means for searching for the meaning and context of the identified words from web data or past recorded data, means for providing the search results to the user's terminal in real time, means for providing customer support functions in a virtual store, and means for overlaying related information on an interface used by the customer. This enables users and customers to understand the meaning of technical terms and unknown words used during meetings or while shopping in real time, thereby improving communication efficiency and the purchasing experience.

[0458] The "means for converting voice data acquired in real time into text data" refers to a technical means for instantly converting the voice spoken by a user into text information.

[0459] "Means for identifying unknown words or technical terms from text data" refers to technical means for analyzing acquired character information to find words that are not generally recognized or technical terms used in a specific field.

[0460] "Means for searching for the meaning and context of identified words from web data or historical records" refers to technical means for searching for the meaning and usage examples of identified unknown words or technical terms from data on the Internet or historical records.

[0461] The "means for providing search results to a user's terminal in real time" refers to a technical means for instantly displaying the obtained search results on the device used by the user.

[0462] "Means for providing customer support functions in virtual stores" refers to technical means for providing support functions to enable customers to enjoy shopping comfortably in stores operated online.

[0463] The "means for overlaying and displaying relevant information on the interface used by the customer" refers to a technical means for overlaying and displaying relevant information on the display device used by the customer.

[0464] To implement this invention, the following system configuration is required. First, the system collects voice data uttered by the user in real time and converts it into text data. This is done using the Google Speech Recognition API and the speech_recognition library. This voice data can be obtained from various input devices used by the user, such as smartphones, smart glasses, and head-mounted displays.

[0465] Next, the server analyzes the retrieved text data to identify unknown words and terminology. This step uses natural language processing (NLP) techniques to tokenize the text data and match it with a list of known words. For each unknown word identified, the server searches for its meaning and context from web data and historical record data, and sends a query to a web API to retrieve its meaning. It also searches historical record databases for related context and previous usage examples.

[0466] Specifically, when words such as "ROI" (return on investment) or "KPI" (key performance indicator) are identified, the server retrieves their meanings from the Internet and past meeting records to add detailed context. In a virtual store, if a customer says, "I want to know what the ROI of this product is," the system retrieves the meaning of "ROI" and related information in real time and displays it as an overlay on the customer's interface.

[0467] This allows customers to understand the meaning of technical terms and unknown words used in the virtual store in real time, improving the shopping experience. The server then sends the search results in the form of messages to an external interface, which can include, for example, the virtual store's chat function. At the same time, the information is recorded in a database or spreadsheet using the Google Spreadsheet API for future reference.

[0468] Examples of prompt sentences include:

[0469] "What is ROI?"

[0470] "I want to know how this product's ROI will be affected."

[0471] It is entered in the form:

[0472] This allows you to understand the meaning and context of technical terms and unknown words in real time, significantly improving meeting efficiency and purchasing experiences.

[0473] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0474] Step 1:

[0475] The device collects the user's voice data and acquires it in real time. The device can be a smartphone, smart glasses, or a head-mounted display. The input is the user's voice data, and the output is the acquired voice data. The collected voice data is passed to the Google Speech Recognition API, where it is prepared to be converted into text data.

[0476] Step 2:

[0477] The device converts the acquired voice data into text data using the Google Speech Recognition API. This process uses the speech_recognition library. The input is voice data, and the output is the converted text data. This text data is analyzed in the next step.

[0478] Step 3:

[0479] The server receives the text data sent from the terminal and tokenizes it using natural language processing (NLP) technology. The input is the text data, and the output is a tokenized word list. This process separates each word in the text.

[0480] Step 4:

[0481] The server matches the tokenized word list against a known word list to identify unknown words or terminology. The input is the tokenized word list, and the output is a list of identified unknown words. The known word list includes common words and industry-standard words.

[0482] Step 5:

[0483] The server searches for the meanings of identified unknown words from web data and historical data. This process involves sending queries using external web APIs to retrieve information. It also searches historical databases for relevant contexts and usage examples. The input is a list of identified unknown words, and the output is the meaning and context information for each word.

[0484] Step 6:

[0485] The server formats the acquired semantic and contextual information into a message and provides it to the user's device in real time. The input is the semantic and contextual information of each word, and the output is a formatted message, which is overlaid on the device's interface.

[0486] Step 7:

[0487] The server sends the search results as a message to an external interface, for example, a chat function of a virtual store. The input is a formatted message, and the output is a message displayed in the chat interface used by the user.

[0488] Step 8:

[0489] The server simultaneously records the retrieved information in a database or spreadsheet using the Google Spreadsheet API. The input is a formatted message, and the output is a recorded database or spreadsheet. This allows for quick retrieval of the information later.

[0490] Step 9:

[0491] It understands the meaning of technical terms and unknown words in real time based on the relevant information provided by the user through the interface. The input is a formatted message, and the output is improved user comprehension, thereby improving the user's shopping experience in the virtual store.

[0492] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0493] This invention combines a real-time support system for technical terms and unknown words in online meetings with an emotion engine that recognizes the user's emotions and optimizes the provision of support information. Each element of the system is described in detail below.

[0494] First, to obtain real-time voice data from users during a meeting, the user's device collects the voice data and converts it into text data using a voice recognition function. The device connects to a voice recognition API and converts the voice data into text information in real time.

[0495] The captured text data is sent to a server, which analyzes it to identify unknown words and terminology. The server uses natural language processing (NLP) techniques to tokenize the text data and match it against a list of known words, including common and industry-standard words.

[0496] The identified unknown words are then processed by the server, which searches for their meaning and context from web data and historical records. The server collects semantic and contextual information by querying web APIs to retrieve relevant information and searching historical databases for relevant context and previous usage examples.

[0497] The search results are delivered to the user's device from the server, and are displayed in real time in a user-friendly format, including detailed information such as word definitions and past usage examples. The most important information is posted to the Zoom chat and shared with other participants.

[0498] Furthermore, the system is equipped with an emotion engine that recognizes emotions from the user's speech and facial expressions. The device inputs the user's facial expression data and voice tone into an emotion analysis module to identify the user's current emotional state. The server combines this emotional information with the importance of words to automatically adjust the priority of information provision.

[0499] A specific use case is when a user feels stressed during a meeting. For example, if the user says, "To maximize the ROI of this project, we first need to set KPIs," and an emotion indicating stress is recognized, the server identifies the words "ROI" and "KPI" and searches for their meaning and context, while recognizing the user's stress level and adjusting the priority of information provision. If an emotion indicating stress is recognized, the server provides detailed assistance information and sends additional messages to help the user relax.

[0500] Additionally, search results are recorded in a spreadsheet using the Google Sheets API, and the records are saved for future reference, allowing users to quickly search for past meeting information when needed.

[0501] In this way, users can understand the meaning and context of technical terms in real time without disrupting the flow of the meeting, while receiving optimal support according to their emotional state, significantly improving meeting efficiency and understanding.

[0502] The processing flow will be explained below.

[0503] Step 1:

[0504] The device captures audio data during the meeting in real time, connects to a speech recognition API, and collects the user's speech as digital audio data.

[0505] Step 2:

[0506] The device converts the acquired voice data into text data in real time, allowing the spoken content to be recorded as text information.

[0507] Step 3:

[0508] The terminal transmits the converted text data to the server, which then sends the text data over the network for analysis on the server side.

[0509] Step 4:

[0510] The server analyzes the received text data and uses natural language processing (NLP) technology to tokenize it, dividing the speech into words.

[0511] Step 5:

[0512] The server checks the tokenized words against a known word list, which includes common vocabulary and industry-standard terminology, to identify unknown words or technical terms.

[0513] Step 6:

[0514] The server queries web data and historical record databases to find the meaning and context of the identified unknown words, using Web APIs to retrieve relevant information and collect relevant contextual information from the historical record database.

[0515] Step 7:

[0516] The server formats the collected information and generates search results that include semantic and contextual information, optionally adding details such as word definitions and previous usage examples.

[0517] Step 8:

[0518] The server provides the results to the user's device, which displays the information to the user in real time and can also post it to the Zoom chat.

[0519] Step 9:

[0520] The device inputs the user's facial expression data and voice tone into an emotion engine to recognize the user's emotional state, which then analyzes the user's emotions in real time.

[0521] Step 10:

[0522] The server automatically adjusts the priority of information provision based on the recognized emotional state. If the user is feeling stressed, detailed assistance information and relaxation messages will be provided first.

[0523] Step 11:

[0524] The server records the search results in a Google Spreadsheet, using the Google Spreadsheet API to store the retrieved information for future reference.

[0525] Examples:

[0526] Let's take the example of a user saying during a meeting, "To maximize the ROI of this project, we first need to set KPIs."

[0527] 1. The device receives the speech and collects voice data using a speech recognition API.

[0528] 2. The device converts the voice data into text data in real time.

[0529] 3. The device sends the generated text data to the server.

[0530] 4. The server analyzes the received text data and identifies the unknown words "ROI" and "KPI."

[0531] 5. The server searches for the meaning and context of "ROI" and "KPI" from web data and historical record databases.

[0532] 6. The server generates the results "ROI: Return on Investment" and "KPI: Key Performance Indicator."

[0533] 7. The server sends the results to the user's device, which displays the information in real time and also posts it to the Zoom chat.

[0534] 8. The device inputs the user's facial expression data and voice tone into an emotion engine to analyze the user's emotional state.

[0535] 9. The server adjusts the priority of information provision based on the recognized emotional state. If the user is feeling stressed, it will provide detailed assistance information and also send a relaxation message.

[0536] 10. The server records the search results in a spreadsheet using the Google Spreadsheet API.

[0537] In this way, users can understand the meaning and context of technical terms in real time without disrupting the flow of the meeting, while receiving optimal support according to their emotional state, significantly improving meeting efficiency and understanding.

[0538] Example 2

[0539] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0540] When online meetings contain a large amount of technical terms and unfamiliar words, participants may find it difficult to understand them, reducing the efficiency of the meeting and the level of understanding of the content. Furthermore, participants' emotional states can affect their comprehension and concentration, but existing systems lack support that takes this into account. Therefore, there is a need for a system that can recognize users' emotions and provide optimal information, along with real-time support for technical terms and unfamiliar words.

[0541] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0542] In this invention, the server includes means for converting speech data acquired in real time into text data, means for identifying unknown words or technical terms from the text data, means for searching for the meaning and context of the identified words from web data or past recorded data, means for detecting the user's emotions, and means for adjusting the order in which search results are presented based on the emotions. This makes it easier for the user to understand the meaning of technical terms and unknown words in real time, and enables the user to receive information optimal for their emotional state.

[0543] "Voice data" refers to information that includes a user's speech or voice, expressed in digital form.

[0544] "Text data" is voice data converted into text information, and is data written in natural language.

[0545] "Unknown words" are words or phrases that are not included in a predefined word list and require context and semantic understanding.

[0546] "Jargon" refers to specialized words and phrases used in a particular field or industry that are necessary for understanding that field.

[0547] "Web data" is information publicly available on the Internet and is data obtained using natural language processing and other technologies.

[0548] "Historical data" refers to information collected or generated in the past or data stored in a database that is used for reference or retrieval.

[0549] "Search results" are data containing information about the meaning and context of identified words that are provided to the user.

[0550] A "terminal" is a computing device used by a user to collect voice data, display text data, and perform other processing.

[0551] A "server" is a computing device that provides services over a network, such as analyzing, retrieving, generating, and providing data.

[0552] "Emotion" refers to the user's mental state or psychological reaction, and indicates states such as stress, relaxation, and concentration.

[0553] "Means for detecting emotions" refers to technology or devices that analyze the user's voice and facial expression data to identify their emotional state.

[0554] "Means for adjusting the order in which search results are presented" refers to techniques or methods for determining the priority of information to be presented based on the user's emotional state or other factors.

[0555] The present invention provides a real-time support system for technical terms and unknown words in online meetings, and further combines it with an emotion engine that recognizes the user's emotions and optimizes the provision of support information. An embodiment of this system will be described in detail.

[0556] First, the user's device collects audio data in real time to capture the user's speech during the meeting. A microphone is connected to the device to capture what is being said during the meeting. Next, the device uses a speech recognition API (e.g., Google Speech-to-Text API) to convert the collected audio data into text data. The audio data is sent to the API, and the returned text data is obtained.

[0557] The captured text data is sent from the device to a server, where it is analyzed. The server first tokenizes the text data (for example, breaking down "To maximize the ROI of this project, we first need to set KPIs" into individual words) and identifies unknown words and technical terms by comparing them with a list of known words. This list of known words includes pre-defined common words and industry-standard words.

[0558] The server then searches for the meaning and context of the identified unknown words using a Web API (e.g., Google Knowledge Graph API), and also searches for related contexts and previous usage examples from a historical record database to gather semantic and contextual information, resulting in the meaning and usage examples of words such as "ROI" and "KPI."

[0559] The search results are delivered to the user's device in real time from the server. Specifically, detailed text including word definitions, past usage examples, and related information is sent to the user's device. Important information is shared with other participants using the chat function of the meeting app (e.g., Zoom).

[0560] The system is also equipped with an emotion engine that recognizes emotions from the user's speech and facial expression data. The device uses a camera and microphone to collect the user's facial expression data and voice tone, which are then input into an emotion analysis module. The server analyzes this data to identify the user's emotional state. For example, a deep learning model can be used to generate emotion labels such as "stress," "relaxed," and "concentrated."

[0561] The server adjusts the order of search results based on the identified emotional information and the importance of unknown words. If the user is feeling stressed, detailed explanations and relaxation messages will be provided first. In this way, users can easily understand the meaning of technical terms and unknown words in real time during meetings and receive the most appropriate information according to their emotional state.

[0562] Finally, the search results are recorded in a spreadsheet using the Google Spreadsheet API from the server, allowing users to quickly reference past meeting information and avoid having to search for the same information again if necessary.

[0563] As a concrete example, suppose a user says during a meeting, "To maximize the ROI of this project, we first need to set KPIs." The device collects this speech as voice data and uses a speech recognition API to convert it into text data: "To maximize the ROI of this project, we first need to set KPIs." This text data is sent to the server, where "ROI" and "KPI" are identified as unknown words. The server searches for their meaning and context and provides detailed information to the user's device. If the user is recognized as feeling stressed, detailed support information and a relaxation message are sent. This information is recorded in a Google spreadsheet and can be referenced later.

[0564] An example prompt might be, "To get the most out of your real-time support during the meeting, please explain specifically what your ROI and KPIs mean. Also, please elaborate on other metrics that are important to the success of this project."

[0565] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0566] Step 1: Collect speech data and send it to the speech recognition API

[0567] Audio data of users speaking during a meeting is collected in real time using a microphone connected to the user's device. The device then sends the collected audio data to a speech recognition API (e.g., Google Speech-to-Text API). The input is real-time audio data, and the output is text data returned by the speech recognition API.

[0568] Step 2: Convert the audio data to text

[0569] The device receives the text data returned from the speech recognition API. Specific operations include processing HTTP requests to the speech recognition API and responses. The input is audio data from the speech recognition API, and the output is text data.

[0570] Step 3: Send the text data to the server

[0571] The terminal sends the converted text data to the server using an HTTP POST request, where the input is the text data and the output is a confirmation of transmission to the server.

[0572] Step 4: Tokenize the text data

[0573] The server tokenizes the received text data. Specifically, it splits the text into words. The input is the text data, and the output is a tokenized word list.

[0574] Step 5: Identify unknown words and terminology

[0575] The server checks the tokenized word list against a list of known words to identify unknown words or terminology. Specific operations include database query comparisons against the known word list. The input is the tokenized word list, and the output is the unknown word list.

[0576] Step 6: Search for the meaning and context of unknown words

[0577] The server queries the identified unknown words to a Web API (e.g., Google Knowledge Graph API) and an internal database to find their meaning and context. Specific operations include generating and sending API queries and analyzing the retrieved data. The input is a list of unknown words, and the output is semantic and context information.

[0578] Step 7: Format the search results and send them to your device

[0579] The server formats the semantic and contextual information it has obtained into a form that is easy for the user to use. Specific operations include converting it to HTML or text format. The input is the semantic and contextual information, and the output is the formatted information. The server then sends the formatted information to the device. The input is the formatted information, and the output is a confirmation of transmission to the device.

[0580] Step 8: Collect emotional data and identify emotional states

[0581] The device inputs the user's facial expression data and voice tone into the emotion analysis module. Specific operations include collecting data using a camera and microphone and analyzing it using an emotion analysis algorithm. The input is facial expression data and voice tone, and the output is the user's emotional state.

[0582] Step 9: Adjust the order of information based on emotional information

[0583] The server combines the acquired emotional information with the importance information of unknown words and automatically adjusts the order in which search results are presented. Specific operations include prioritizing the results based on the emotional state and the importance of words. The input is emotional information and importance information, and the output is the adjusted order in which the results are presented.

[0584] Step 10: Send search results to an external interface and record them in a spreadsheet

[0585] The server sends the search results as a message to an external interface (e.g., the chat function of a conference app). It then records the search results in a spreadsheet using the Google Spreadsheet API. Specific operations include sending and recording data using the API. The input is the search results, and the output is confirmation of sending to the external interface and recording in the spreadsheet.

[0586] (Application example 2)

[0587] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0588] When a driver in an autonomous vehicle encounters technical terms or unfamiliar words while driving, it is difficult to provide the necessary information for driving due to a lack of meaning and context information. It is also difficult to provide information at the appropriate time based on the driver's emotional state. This can lead to a decrease in driving safety and efficiency.

[0589] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for converting voice data acquired in real time into text data, means for identifying unknown words or technical terms from the text data, means for searching for the meaning and context of the identified words from web data or past recorded data, means for providing the search results to the user's terminal in real time, means for analyzing facial expression data or voice tone to recognize the driver's emotional state, and means for adjusting the priority of information provision based on the recognized emotional state. This makes it possible to provide information at the optimal timing according to the driver's emotional state while supporting the driver's understanding of technical terms and unknown words.

[0590] "Audio data" is a digital representation of an audio signal captured in real time.

[0591] "Text data" refers to data that has been converted into character information based on voice data.

[0592] "Unknown words" refer to technical terms or new terms that are not generally understood by users.

[0593] "Terminology" refers to specialized terms used in a particular industry or field.

[0594] "Web data" is data that includes information on the Internet.

[0595] "Previously recorded data" is data that has been previously acquired or stored.

[0596] "Providing in real time" means providing information instantly, without delay.

[0597] "User terminal" refers to the device used by a User.

[0598] "Emotional state" refers to the psychological state of a user as perceived from their facial expression and tone of voice.

[0599] "Facial expression data" refers to data relating to the facial expression of the user captured by a camera or the like.

[0600] "Voice tone" refers to the voice characteristics obtained from the intonation and stress of the voice.

[0601] "Adjusting the priority of information provision" means providing information at an appropriate time depending on the importance and urgency of the information.

[0602] An "external interface" is an interface for connecting a system to external equipment or devices.

[0603] An "assist message" is a supplementary message to support or encourage the user.

[0604] A "database" is a system for systematically organizing and storing data.

[0605] A spreadsheet is a tool for managing, analyzing, and displaying data in rows and columns.

[0606] This invention is a system that provides real-time support to drivers of self-driving vehicles when they encounter technical terms or unknown words while driving. The system collects and analyzes the user's voice data and facial expression data to provide appropriate support information.

[0607] System configuration

[0608] The system consists of the following hardware and software components:

[0609] Hardware:

[0610] Microphone: Collects driver voice inside the vehicle.

[0611] Camera: Collects driver facial expression data.

[0612] On-board computer: processes data and provides information.

[0613] software:

[0614] Speech recognition API (e.g., Google Speech-to-Text API): Converts voice data into text data in real time.

[0615] Natural Language Processing (NLP) engines (e.g., spaCy, NLTK): Analyze text data and identify terminology and unknown words.

[0616] Emotion recognition module (e.g., Face API, Microsoft Azure Emotion API): Analyzes facial expression data and voice tone to recognize the driver's emotional state.

[0617] Real-time data collection and processing system: Collects voice and facial expression data and analyzes it in real time.

[0618] Program processing

[0619] The server collects voice data in real time and converts it into text using a speech recognition API. It then analyzes the text using an NLP engine to identify unknown words and technical terms. The meaning and context of the identified words are obtained from web data and historically recorded data. This meaning and context information is then provided to the user's device in real time.

[0620] Furthermore, the driver's facial expression data and voice tone captured by the camera are input into the emotion recognition module to recognize the driver's emotional state. The server adjusts the priority of information provision based on the recognized emotional state. If emotions indicating stress are recognized, detailed assistance information and relaxation messages will be provided.

[0621] For example, if a driver hears the term "ADAS (Advanced Driver Assistance Systems)" for the first time while driving, the system will respond as follows:

[0622] 1. Listen to the driver's voice and convert it into text.

[0623] 2. Identify "ADAS" using an NLP engine.

[0624] 3. Obtain the definition of "ADAS" and related information from the Web API.

[0625] 4. Cameras analyze the driver's facial expressions to identify their emotional state.

[0626] 5. Display information on the display and add assistance messages as needed.

[0627] Prompt Sentence Examples

[0628] A driver in a self-driving vehicle encounters the word "ADAS" for the first time. How does the system respond in the next step?

[0629] 1. Listen to the driver's voice and convert it into text.

[0630] 2. Identify "ADAS" using the NLP engine.

[0631] 3. Get the meaning of "ADAS" from the Web API.

[0632] 4. Analyze the driver's emotional state using a camera.

[0633] 5. Information is displayed on the screen, and if the driver is feeling stressed, a message to relax is also displayed.

[0634] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0635] Step 1:

[0636] The user's voice is collected by a microphone. The input is real-time voice data, and the output is digitized voice data. The device sends this voice data to a speech recognition API (e.g., Google Speech-to-Text API) to convert it into text information. The output is text data.

[0637] Step 2:

[0638] The device sends text data obtained from the speech recognition API to the server, which then analyzes the text data using a natural language processing (NLP) engine (e.g., spaCy, NLTK). The input is text data, and the output is a tokenized word list. The server identifies unknown words and technical terms from this list.

[0639] Step 3:

[0640] The server searches for the meaning and context of identified unknown words and technical terms from web data or historical record data. The input is a list of identified words, and the output is the meaning and context information of each word. The server uses a web API to obtain related information and searches for related context from the historical record database.

[0641] Step 4:

[0642] The camera collects the driver's facial expression data in real time. The input is real-time facial expression data, and the output is digitized facial expression data. The device sends this facial expression data to an emotion recognition module (e.g., Face API, Microsoft Azure Emotion API) for analysis. The output is the driver's emotional state.

[0643] Step 5:

[0644] The server integrates the emotional state obtained from the emotion recognition module with the semantic and contextual information of the words obtained in step 3. The input is the emotional state and semantic / contextual information, and the output is a list of information provision with adjusted priorities. The server automatically adjusts the priority of information provision according to the emotional state.

[0645] Step 6:

[0646] The server sends the prioritized information list to the terminal. The terminal displays the meaning of words and context information in real time on the display. If the driver shows signs of stress or anxiety, the terminal also displays a relaxation message. The input is the information list, and the output is the displayed information.

[0647] Prompt Sentence Examples

[0648] A driver in a self-driving vehicle encounters the word "ADAS" for the first time. How does the system respond in the next step?

[0649] 1. Listen to the driver's voice and convert it into text.

[0650] 2. Identify "ADAS" using the NLP engine.

[0651] 3. Get the meaning of "ADAS" from the Web API.

[0652] 4. Analyze the driver's emotional state using a camera.

[0653] 5. Information is displayed on the screen, and if the driver is feeling stressed, a message to relax is also displayed.

[0654] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0655] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0656] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0657] [Third embodiment]

[0658] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0659] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0660] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0661] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0662] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0663] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0664] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0665] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0666] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0667] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0668] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0669] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0670] The present invention relates to a real-time support system for technical terms and unknown words in online meetings. Each element of the system will be described in detail below.

[0671] First, to capture the audio data of users speaking during a meeting in real time, the user's device collects the audio data and converts it into text data using the Zoom API. The device accesses Zoom's live transcription function and specifies the meeting ID as an argument to obtain the content of the conversation as text information.

[0672] The captured text data is sent to a server, which analyzes it to identify unknown words and terminology. The server uses natural language processing (NLP) techniques to tokenize the text data and match it against a list of known words, including common and industry-standard words.

[0673] The identified unknown words are then processed by the server, which searches for their meaning and context from web data and past meeting records. The server queries web APIs to retrieve relevant information and searches past record databases for relevant context and previous usage examples.

[0674] Search results are provided from the server to the user's device. These results are formatted for user convenience and provided in real time. As a concrete example, consider the case where the word "ROI" is identified. The server retrieves the meaning of "ROI" (return on investment) from the web, adds detailed context based on usage examples from past meetings, and sends it to the user's device.

[0675] The server then sends the search results in the form of a message to an external interface, such as the Zoom chat, and simultaneously records this information in a database or spreadsheet using the Google Spreadsheet API, which is then saved for future reference, allowing for quick retrieval of past meeting information when needed.

[0676] As a concrete example, consider a situation where a user says during a meeting, "To maximize the ROI of this project, we first need to set KPIs." The speech is transcribed in real time, and the server identifies the words "ROI" and "KPI." The server then searches the internet and past meeting records for the meaning of these words, generates a message saying "ROI: Return on Investment, KPI: Key Performance Indicator," and sends it to the Zoom chat. This information is also recorded in a spreadsheet.

[0677] In this way, users can understand the meaning and context of technical terms in real time without disrupting the progress of the meeting, which can significantly improve the efficiency and understanding of meetings.

[0678] The processing flow will be explained below.

[0679] Step 1:

[0680] The device connects to the Zoom API to capture audio data in real time during the meeting, capturing what the user is saying as digital audio data on the spot.

[0681] Step 2:

[0682] The device converts the audio data it receives into text using Zoom's live transcription feature, which sends a request to Zoom using the meeting ID to convert the audio into text.

[0683] Step 3:

[0684] The terminal transmits the text data to the server, which then uploads the text data to the server via the network for analysis.

[0685] Step 4:

[0686] The server analyzes the received text data and uses natural language processing (NLP) technology to tokenize it, which divides the speech into individual words.

[0687] Step 5:

[0688] The server checks the tokenized words against a known word list, which contains common vocabulary and industry-standard terminology, to identify unknown words or technical terms.

[0689] Step 6:

[0690] The server queries web data and past record databases to find the meaning and context of the identified unknown words. It obtains relevant information from web APIs and gathers contextual information by searching past meeting records.

[0691] Step 7:

[0692] The server formats the collected information to generate semantic and contextual search results, including details such as word definitions and historical usage examples.

[0693] Step 8:

[0694] The server sends the formatted search results to the terminal. By sending the information to the user's terminal, the user can receive the information in real time.

[0695] Step 9:

[0696] The device will display the retrieved information to the user, and will also post the search results to the Zoom chat to share the information with other participants.

[0697] Step 10:

[0698] The server records the acquired information in a spreadsheet using the Google Spreadsheet API, so that the information can be referenced after the meeting.

[0699] Examples:

[0700] For example, if a user says during a meeting, "In order to maximize the ROI of this project, we first need to set KPIs," the process would go something like this:

[0701] 1. The device receives the speech and collects audio data using the Zoom API.

[0702] 2. The device converts the voice into text information and generates text data.

[0703] 3. The device sends the generated text data to the server.

[0704] 4. The server analyzes the received text data and identifies unknown words including "ROI" and "KPI."

[0705] 5. The server searches for the meaning and context of "ROI" and "KPI" from web data and historical record data.

[0706] 6. The server generates the results "ROI: Return on Investment" and "KPI: Key Performance Indicator."

[0707] 7. The server sends the generated results to the device, which displays the information to the user and posts it to the Zoom chat.

[0708] 8. The server records the obtained information in a spreadsheet using the Google Spreadsheet API.

[0709] In this way, users can understand the meaning and context of technical terms in real time without disrupting the flow of the meeting, allowing them to participate in the meeting efficiently.

[0710] Example 1

[0711] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0712] In online meetings, when technical terms or unfamiliar words appear, participants may be unable to keep up and the meeting progress may be delayed. Furthermore, when an unfamiliar word appears during a meeting, it is difficult to search for and understand it on the spot. This reduces the efficiency of the meeting and leads to insufficient understanding among participants. Furthermore, if there is no easy way to access the information later, it becomes difficult to reuse the information.

[0713] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0714] In this invention, the server includes a means for acquiring audio data of an online conference from a user's terminal in real time and converting the audio data into text data, a means for identifying unknown words or technical terms from the text data using natural language processing technology, and a means for searching for the meaning and context of the identified words from web data or past conference records. This allows users to grasp the meaning of technical terms and unknown words in real time, deepening their understanding without disrupting the progress of the conference. Furthermore, a means for sending search results as a message to an external interface and a means for recording them in a database or spreadsheet make it easy to reuse the same information later.

[0715] A "user terminal" is an electronic device such as a computer, tablet, or smartphone used by a user participating in an online conference.

[0716] "Audio data" refers to digital audio information recorded from an online conference conversation.

[0717] "Text data" is digital data that has been converted from voice data into character information.

[0718] "Natural language processing technology" is a technology for analyzing text data to identify unknown words and technical terms, and is a series of processing techniques including tokenization and morphological analysis.

[0719] "Unknown words or technical terms" are new words that are not included in known word lists or words that are technical and not commonly understood.

[0720] "Web data" refers to digital information that is publicly available on the Internet, including information that can be obtained through web pages or APIs.

[0721] "Past meeting records" refers to audio data and text data from previous meetings, the contents of which are stored in a database.

[0722] "Searching" is the act of referencing web data or past meeting records to find the meaning or context of a specified word.

[0723] "Formatting" refers to converting search results into something that is easy for users to understand, and organizing them into an appropriate format and structure.

[0724] "Providing in real time" refers to providing information immediately without waiting while users are conducting an online conference.

[0725] "Sending to an external interface" means sending the obtained information to another system or application, such as a messaging platform.

[0726] "Recording in a database or spreadsheet" means storing search results or meeting information in a digital format for future reference.

[0727] The present invention is a system that provides real-time support for technical terms and unknown words in online meetings. The system shown below is composed of multiple elements, including a user terminal, a server, an external interface, and a database or spreadsheet.

[0728] 1. Acquiring audio data and converting it to text

[0729] When a user starts an online meeting, the user's device captures the meeting's audio data in real time. Specifically, the user's device (PC, smartphone, tablet, etc.) uses the online meeting system to collect audio data. This device then converts the audio into text data using audio data capture technology such as the Zoom API. For example, the device accesses Zoom's live transcription function and converts the captured audio data into text in real time.

[0730] 2. Sending text data

[0731] The device sends the converted text data to the server using a communication protocol such as an HTTP POST request, where it is transferred to the server and prepared for analysis.

[0732] 3. Identifying Unknown Words and Terminology

[0733] The server processes the received text data using natural language processing (NLP) techniques, such as tokenizing the text and matching it against a list of known words to identify unknown words and terminology. This NLP includes morphological analysis, tokenization, and dictionary lookup of words.

[0734] 4. Searching for meaning and context

[0735] For the identified unknown words or terminology, the server searches for meaning and context from web APIs or a database of past meeting records. For example, the server may refer to web data or previous meeting records to clarify the meaning of the identified words. The program then performs a regular web search or internal resource search to extract relevant information.

[0736] 5. Providing search results

[0737] The server formats the search results for delivery to the user. Additionally, the server delivers the formatted information to the user's device in real time. This information is then sent as a message to an external interface, such as a Zoom chat. For example, the Zoom API can be used to post a description and context of the retrieved words to the chat. The same search results can also be saved to a database or spreadsheet using the Google Spreadsheet API for later reference.

[0738] Specific examples

[0739] For example, if a user says the following during a meeting:

[0740] "To maximize the ROI of this project, we first need to set KPIs."

[0741] The device captures this speech as audio data and converts it to text using the Zoom API:

[0742] "To maximize the ROI of this project, we first need to set KPIs."

[0743] The server identifies the words "ROI" and "KPI" and searches for their meanings in web data and past meeting records. For example, the server obtains results such as "ROI: Return on Investment" and "KPI: Key Performance Indicator."

[0744] The server sends the results as a message to the Zoom chat and simultaneously records them in a Google spreadsheet, providing real-time definitions of technical terms and supporting understanding during the meeting.

[0745] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0746] Step 1:

[0747] A user starts an online meeting. The user uses the online meeting system and is issued a meeting ID. The device starts collecting audio data via the Zoom API. The input is the trigger to start the meeting, and the output is the collected audio data.

[0748] Step 2:

[0749] The device uses Zoom's live transcription feature to convert audio data into text data in real time, directly using the Zoom API. The input is the collected audio data, and the output is the converted text data.

[0750] Step 3:

[0751] The terminal sends the converted text data to the server using an HTTP POST request. The input is text data, and the output is the transfer of the text data to the server.

[0752] Step 4:

[0753] The server analyzes the received text data using NLP techniques. Specifically, it tokenizes the text data and compares it with a list of known words to identify unknown words and technical terms. The input is the transmitted text data, and the output is the identified unknown words and technical terms.

[0754] Step 5:

[0755] The server searches for the meaning and context of the identified words from web data and a database of past meeting records. It uses a web API to obtain related information and reference past records. The input is the identified word, and the output is information about the meaning and context of that word.

[0756] Step 6:

[0757] The server formats the information it receives and converts it into something that is easy for the user to understand, for example, "ROI: Return on Investment." The input is the semantic and contextual information it receives, and the output is the formatted information.

[0758] Step 7:

[0759] The server delivers the formatted information to the user's device in real time. This information is sent as a message to an external interface such as Zoom chat. Furthermore, the information is recorded in a database or spreadsheet using the Google Spreadsheet API. The input is the formatted information, and the output is sending a message to Zoom chat and recording it in the database.

[0760] Specific examples of behavior:

[0761] For example, consider the case where a user says during a meeting, "To maximize the ROI of this project, we first need to set KPIs."

[0762] Step 1: A user starts a conference and audio data collection begins.

[0763] Step 2: The device translates the speech into "To maximize the ROI of this project, we first need to set KPIs."

[0764] Step 3: The device sends this text data to the server.

[0765] Step 4: The server identifies the words "ROI" and "KPI."

[0766] Step 5: The server obtains information such as "ROI: Return on Investment" and "KPI: Key Performance Indicator" from web data and past meeting records.

[0767] Step 6: The server formats this information.

[0768] Step 7: The server sends the formatted information to the Zoom chat and records it in a Google spreadsheet.

[0769] (Application example 1)

[0770] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0771] When a large amount of technical terms and unfamiliar words are used in online meetings or virtual stores, participants and customers often cannot understand their meaning or context, resulting in a decrease in communication efficiency. This can also impair the customer's purchasing experience, so a solution is needed.

[0772] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0773] In this invention, the server includes means for converting speech data acquired in real time into text data, means for identifying unknown words or technical terms from the text data, means for searching for the meaning and context of the identified words from web data or past recorded data, means for providing the search results to the user's terminal in real time, means for providing customer support functions in a virtual store, and means for overlaying related information on an interface used by the customer. This enables users and customers to understand the meaning of technical terms and unknown words used during meetings or while shopping in real time, thereby improving communication efficiency and the purchasing experience.

[0774] The "means for converting voice data acquired in real time into text data" refers to a technical means for instantly converting the voice spoken by a user into text information.

[0775] "Means for identifying unknown words or technical terms from text data" refers to technical means for analyzing acquired character information to find words that are not generally recognized or technical terms used in a specific field.

[0776] "Means for searching for the meaning and context of identified words from web data or historical records" refers to technical means for searching for the meaning and usage examples of identified unknown words or technical terms from data on the Internet or historical records.

[0777] The "means for providing search results to a user's terminal in real time" refers to a technical means for instantly displaying the obtained search results on the device used by the user.

[0778] "Means for providing customer support functions in virtual stores" refers to technical means for providing support functions to enable customers to enjoy shopping comfortably in stores operated online.

[0779] The "means for overlaying and displaying relevant information on the interface used by the customer" refers to a technical means for overlaying and displaying relevant information on the display device used by the customer.

[0780] To implement this invention, the following system configuration is required. First, the system collects voice data uttered by the user in real time and converts it into text data. This is done using the Google Speech Recognition API and the speech_recognition library. This voice data can be obtained from various input devices used by the user, such as smartphones, smart glasses, and head-mounted displays.

[0781] Next, the server analyzes the retrieved text data to identify unknown words and terminology. This step uses natural language processing (NLP) techniques to tokenize the text data and match it with a list of known words. For each unknown word identified, the server searches for its meaning and context from web data and historical record data, and sends a query to a web API to retrieve its meaning. It also searches historical record databases for related context and previous usage examples.

[0782] Specifically, when words such as "ROI" (return on investment) or "KPI" (key performance indicator) are identified, the server retrieves their meanings from the Internet and past meeting records to add detailed context. In a virtual store, if a customer says, "I want to know what the ROI of this product is," the system retrieves the meaning of "ROI" and related information in real time and displays it as an overlay on the customer's interface.

[0783] This allows customers to understand the meaning of technical terms and unknown words used in the virtual store in real time, improving the shopping experience. The server then sends the search results in the form of messages to an external interface, which can include, for example, the virtual store's chat function. At the same time, the information is recorded in a database or spreadsheet using the Google Spreadsheet API for future reference.

[0784] Examples of prompt sentences include:

[0785] "What is ROI?"

[0786] "I want to know how this product's ROI will be affected."

[0787] It is entered in the form:

[0788] This allows you to understand the meaning and context of technical terms and unknown words in real time, significantly improving meeting efficiency and purchasing experiences.

[0789] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0790] Step 1:

[0791] The device collects the user's voice data and acquires it in real time. The device can be a smartphone, smart glasses, or a head-mounted display. The input is the user's voice data, and the output is the acquired voice data. The collected voice data is passed to the Google Speech Recognition API, where it is prepared to be converted into text data.

[0792] Step 2:

[0793] The device converts the acquired voice data into text data using the Google Speech Recognition API. This process uses the speech_recognition library. The input is voice data, and the output is the converted text data. This text data is analyzed in the next step.

[0794] Step 3:

[0795] The server receives the text data sent from the terminal and tokenizes it using natural language processing (NLP) technology. The input is the text data, and the output is a tokenized word list. This process separates each word in the text.

[0796] Step 4:

[0797] The server matches the tokenized word list against a known word list to identify unknown words or terminology. The input is the tokenized word list, and the output is a list of identified unknown words. The known word list includes common words and industry-standard words.

[0798] Step 5:

[0799] The server searches for the meanings of identified unknown words from web data and historical data. This process involves sending queries using external web APIs to retrieve information. It also searches historical databases for relevant contexts and usage examples. The input is a list of identified unknown words, and the output is the meaning and context information for each word.

[0800] Step 6:

[0801] The server formats the acquired semantic and contextual information into a message and provides it to the user's device in real time. The input is the semantic and contextual information of each word, and the output is a formatted message, which is overlaid on the device's interface.

[0802] Step 7:

[0803] The server sends the search results as a message to an external interface, for example, a chat function of a virtual store. The input is a formatted message, and the output is a message displayed in the chat interface used by the user.

[0804] Step 8:

[0805] The server simultaneously records the retrieved information in a database or spreadsheet using the Google Spreadsheet API. The input is a formatted message, and the output is a recorded database or spreadsheet. This allows for quick retrieval of the information later.

[0806] Step 9:

[0807] It understands the meaning of technical terms and unknown words in real time based on the relevant information provided by the user through the interface. The input is a formatted message, and the output is improved user comprehension, thereby improving the user's shopping experience in the virtual store.

[0808] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0809] This invention combines a real-time support system for technical terms and unknown words in online meetings with an emotion engine that recognizes the user's emotions and optimizes the provision of support information. Each element of the system is described in detail below.

[0810] First, to obtain real-time voice data from users during a meeting, the user's device collects the voice data and converts it into text data using a voice recognition function. The device connects to a voice recognition API and converts the voice data into text information in real time.

[0811] The captured text data is sent to a server, which analyzes it to identify unknown words and terminology. The server uses natural language processing (NLP) techniques to tokenize the text data and match it against a list of known words, including common and industry-standard words.

[0812] The identified unknown words are then processed by the server, which searches for their meaning and context from web data and historical records. The server collects semantic and contextual information by querying web APIs to retrieve relevant information and searching historical databases for relevant context and previous usage examples.

[0813] The search results are delivered to the user's device from the server, and are displayed in real time in a user-friendly format, including detailed information such as word definitions and past usage examples. The most important information is posted to the Zoom chat and shared with other participants.

[0814] Furthermore, the system is equipped with an emotion engine that recognizes emotions from the user's speech and facial expressions. The device inputs the user's facial expression data and voice tone into an emotion analysis module to identify the user's current emotional state. The server combines this emotional information with the importance of words to automatically adjust the priority of information provision.

[0815] A specific use case is when a user feels stressed during a meeting. For example, if the user says, "To maximize the ROI of this project, we first need to set KPIs," and an emotion indicating stress is recognized, the server identifies the words "ROI" and "KPI" and searches for their meaning and context, while recognizing the user's stress level and adjusting the priority of information provision. If an emotion indicating stress is recognized, the server provides detailed assistance information and sends additional messages to help the user relax.

[0816] Additionally, search results are recorded in a spreadsheet using the Google Sheets API, and the records are saved for future reference, allowing users to quickly search for past meeting information when needed.

[0817] In this way, users can understand the meaning and context of technical terms in real time without disrupting the flow of the meeting, while receiving optimal support according to their emotional state, significantly improving meeting efficiency and understanding.

[0818] The processing flow will be explained below.

[0819] Step 1:

[0820] The device captures audio data during the meeting in real time, connects to a speech recognition API, and collects the user's speech as digital audio data.

[0821] Step 2:

[0822] The device converts the acquired voice data into text data in real time, allowing the spoken content to be recorded as text information.

[0823] Step 3:

[0824] The terminal transmits the converted text data to the server, which then sends the text data over the network for analysis on the server side.

[0825] Step 4:

[0826] The server analyzes the received text data and uses natural language processing (NLP) technology to tokenize it, dividing the speech into words.

[0827] Step 5:

[0828] The server checks the tokenized words against a known word list, which includes common vocabulary and industry-standard terminology, to identify unknown words or technical terms.

[0829] Step 6:

[0830] The server queries web data and historical record databases to find the meaning and context of the identified unknown words, using Web APIs to retrieve relevant information and collect relevant contextual information from the historical record database.

[0831] Step 7:

[0832] The server formats the collected information and generates search results that include semantic and contextual information, optionally adding details such as word definitions and previous usage examples.

[0833] Step 8:

[0834] The server provides the results to the user's device, which displays the information to the user in real time and can also post it to the Zoom chat.

[0835] Step 9:

[0836] The device inputs the user's facial expression data and voice tone into an emotion engine to recognize the user's emotional state, which then analyzes the user's emotions in real time.

[0837] Step 10:

[0838] The server automatically adjusts the priority of information provision based on the recognized emotional state. If the user is feeling stressed, detailed assistance information and relaxation messages will be provided first.

[0839] Step 11:

[0840] The server records the search results in a Google Spreadsheet, using the Google Spreadsheet API to store the retrieved information for future reference.

[0841] Examples:

[0842] Let's take the example of a user saying during a meeting, "To maximize the ROI of this project, we first need to set KPIs."

[0843] 1. The device receives the speech and collects voice data using a speech recognition API.

[0844] 2. The device converts the voice data into text data in real time.

[0845] 3. The device sends the generated text data to the server.

[0846] 4. The server analyzes the received text data and identifies the unknown words "ROI" and "KPI."

[0847] 5. The server searches for the meaning and context of "ROI" and "KPI" from web data and historical record databases.

[0848] 6. The server generates the results "ROI: Return on Investment" and "KPI: Key Performance Indicator."

[0849] 7. The server sends the results to the user's device, which displays the information in real time and also posts it to the Zoom chat.

[0850] 8. The device inputs the user's facial expression data and voice tone into an emotion engine to analyze the user's emotional state.

[0851] 9. The server adjusts the priority of information provision based on the recognized emotional state. If the user is feeling stressed, it will provide detailed assistance information and also send a relaxation message.

[0852] 10. The server records the search results in a spreadsheet using the Google Spreadsheet API.

[0853] In this way, users can understand the meaning and context of technical terms in real time without disrupting the flow of the meeting, while receiving optimal support according to their emotional state, significantly improving meeting efficiency and understanding.

[0854] Example 2

[0855] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0856] When online meetings contain a large amount of technical terms and unfamiliar words, participants may find it difficult to understand them, reducing the efficiency of the meeting and the level of understanding of the content. Furthermore, participants' emotional states can affect their comprehension and concentration, but existing systems lack support that takes this into account. Therefore, there is a need for a system that can recognize users' emotions and provide optimal information, along with real-time support for technical terms and unfamiliar words.

[0857] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0858] In this invention, the server includes means for converting speech data acquired in real time into text data, means for identifying unknown words or technical terms from the text data, means for searching for the meaning and context of the identified words from web data or past recorded data, means for detecting the user's emotions, and means for adjusting the order in which search results are presented based on the emotions. This makes it easier for the user to understand the meaning of technical terms and unknown words in real time, and enables the user to receive information optimal for their emotional state.

[0859] "Voice data" refers to information that includes a user's speech or voice, expressed in digital form.

[0860] "Text data" is voice data converted into text information, and is data written in natural language.

[0861] "Unknown words" are words or phrases that are not included in a predefined word list and require context and semantic understanding.

[0862] "Jargon" refers to specialized words and phrases used in a particular field or industry that are necessary for understanding that field.

[0863] "Web data" is information publicly available on the Internet and is data obtained using natural language processing and other technologies.

[0864] "Historical data" refers to information collected or generated in the past or data stored in a database that is used for reference or retrieval.

[0865] "Search results" are data containing information about the meaning and context of identified words that are provided to the user.

[0866] A "terminal" is a computing device used by a user to collect voice data, display text data, and perform other processing.

[0867] A "server" is a computing device that provides services over a network, such as analyzing, retrieving, generating, and providing data.

[0868] "Emotion" refers to the user's mental state or psychological reaction, and indicates states such as stress, relaxation, and concentration.

[0869] "Means for detecting emotions" refers to technology or devices that analyze the user's voice and facial expression data to identify their emotional state.

[0870] "Means for adjusting the order in which search results are presented" refers to techniques or methods for determining the priority of information to be presented based on the user's emotional state or other factors.

[0871] The present invention provides a real-time support system for technical terms and unknown words in online meetings, and further combines it with an emotion engine that recognizes the user's emotions and optimizes the provision of support information. An embodiment of this system will be described in detail.

[0872] First, the user's device collects audio data in real time to capture the user's speech during the meeting. A microphone is connected to the device to capture what is being said during the meeting. Next, the device uses a speech recognition API (e.g., Google Speech-to-Text API) to convert the collected audio data into text data. The audio data is sent to the API, and the returned text data is obtained.

[0873] The captured text data is sent from the device to a server, where it is analyzed. The server first tokenizes the text data (for example, breaking down "To maximize the ROI of this project, we first need to set KPIs" into individual words) and identifies unknown words and technical terms by comparing them with a list of known words. This list of known words includes pre-defined common words and industry-standard words.

[0874] The server then searches for the meaning and context of the identified unknown words using a Web API (e.g., Google Knowledge Graph API), and also searches for related contexts and previous usage examples from a historical record database to gather semantic and contextual information, resulting in the meaning and usage examples of words such as "ROI" and "KPI."

[0875] The search results are delivered to the user's device in real time from the server. Specifically, detailed text including word definitions, past usage examples, and related information is sent to the user's device. Important information is shared with other participants using the chat function of the meeting app (e.g., Zoom).

[0876] The system is also equipped with an emotion engine that recognizes emotions from the user's speech and facial expression data. The device uses a camera and microphone to collect the user's facial expression data and voice tone, which are then input into an emotion analysis module. The server analyzes this data to identify the user's emotional state. For example, a deep learning model can be used to generate emotion labels such as "stress," "relaxed," and "concentrated."

[0877] The server adjusts the order of search results based on the identified emotional information and the importance of unknown words. If the user is feeling stressed, detailed explanations and relaxation messages will be provided first. In this way, users can easily understand the meaning of technical terms and unknown words in real time during meetings and receive the most appropriate information according to their emotional state.

[0878] Finally, the search results are recorded in a spreadsheet using the Google Spreadsheet API from the server, allowing users to quickly reference past meeting information and avoid having to search for the same information again if necessary.

[0879] As a concrete example, suppose a user says during a meeting, "To maximize the ROI of this project, we first need to set KPIs." The device collects this speech as voice data and uses a speech recognition API to convert it into text data: "To maximize the ROI of this project, we first need to set KPIs." This text data is sent to the server, where "ROI" and "KPI" are identified as unknown words. The server searches for their meaning and context and provides detailed information to the user's device. If the user is recognized as feeling stressed, detailed support information and a relaxation message are sent. This information is recorded in a Google spreadsheet and can be referenced later.

[0880] An example prompt might be, "To get the most out of your real-time support during the meeting, please explain specifically what your ROI and KPIs mean. Also, please elaborate on other metrics that are important to the success of this project."

[0881] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0882] Step 1: Collect speech data and send it to the speech recognition API

[0883] Audio data of users speaking during a meeting is collected in real time using a microphone connected to the user's device. The device then sends the collected audio data to a speech recognition API (e.g., Google Speech-to-Text API). The input is real-time audio data, and the output is text data returned by the speech recognition API.

[0884] Step 2: Convert the audio data to text

[0885] The device receives the text data returned from the speech recognition API. Specific operations include processing HTTP requests to the speech recognition API and responses. The input is audio data from the speech recognition API, and the output is text data.

[0886] Step 3: Send the text data to the server

[0887] The terminal sends the converted text data to the server using an HTTP POST request, where the input is the text data and the output is a confirmation of transmission to the server.

[0888] Step 4: Tokenize the text data

[0889] The server tokenizes the received text data. Specifically, it splits the text into words. The input is the text data, and the output is a tokenized word list.

[0890] Step 5: Identify unknown words and terminology

[0891] The server checks the tokenized word list against a list of known words to identify unknown words or terminology. Specific operations include database query comparisons against the known word list. The input is the tokenized word list, and the output is the unknown word list.

[0892] Step 6: Search for the meaning and context of unknown words

[0893] The server queries the identified unknown words to a Web API (e.g., Google Knowledge Graph API) and an internal database to find their meaning and context. Specific operations include generating and sending API queries and analyzing the retrieved data. The input is a list of unknown words, and the output is semantic and context information.

[0894] Step 7: Format the search results and send them to your device

[0895] The server formats the semantic and contextual information it has obtained into a form that is easy for the user to use. Specific operations include converting it to HTML or text format. The input is the semantic and contextual information, and the output is the formatted information. The server then sends the formatted information to the device. The input is the formatted information, and the output is a confirmation of transmission to the device.

[0896] Step 8: Collect emotional data and identify emotional states

[0897] The device inputs the user's facial expression data and voice tone into the emotion analysis module. Specific operations include collecting data using a camera and microphone and analyzing it using an emotion analysis algorithm. The input is facial expression data and voice tone, and the output is the user's emotional state.

[0898] Step 9: Adjust the order of information based on emotional information

[0899] The server combines the acquired emotional information with the importance information of unknown words and automatically adjusts the order in which search results are presented. Specific operations include prioritizing the results based on the emotional state and the importance of words. The input is emotional information and importance information, and the output is the adjusted order in which the results are presented.

[0900] Step 10: Send search results to an external interface and record them in a spreadsheet

[0901] The server sends the search results as a message to an external interface (e.g., the chat function of a conference app). It then records the search results in a spreadsheet using the Google Spreadsheet API. Specific operations include sending and recording data using the API. The input is the search results, and the output is confirmation of sending to the external interface and recording in the spreadsheet.

[0902] (Application example 2)

[0903] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0904] When a driver in an autonomous vehicle encounters technical terms or unfamiliar words while driving, it is difficult to provide the necessary information for driving due to a lack of meaning and context information. It is also difficult to provide information at the appropriate time based on the driver's emotional state. This can lead to a decrease in driving safety and efficiency.

[0905] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for converting voice data acquired in real time into text data, means for identifying unknown words or technical terms from the text data, means for searching for the meaning and context of the identified words from web data or past recorded data, means for providing the search results to the user's terminal in real time, means for analyzing facial expression data or voice tone to recognize the driver's emotional state, and means for adjusting the priority of information provision based on the recognized emotional state. This makes it possible to provide information at the optimal timing according to the driver's emotional state while supporting the driver's understanding of technical terms and unknown words.

[0906] "Audio data" is a digital representation of an audio signal captured in real time.

[0907] "Text data" refers to data that has been converted into character information based on voice data.

[0908] "Unknown words" refer to technical terms or new terms that are not generally understood by users.

[0909] "Terminology" refers to specialized terms used in a particular industry or field.

[0910] "Web data" is data that includes information on the Internet.

[0911] "Previously recorded data" is data that has been previously acquired or stored.

[0912] "Providing in real time" means providing information instantly, without delay.

[0913] "User terminal" refers to the device used by a User.

[0914] "Emotional state" refers to the psychological state of a user as perceived from their facial expression and tone of voice.

[0915] "Facial expression data" refers to data relating to the facial expression of the user captured by a camera or the like.

[0916] "Voice tone" refers to the voice characteristics obtained from the intonation and stress of the voice.

[0917] "Adjusting the priority of information provision" means providing information at an appropriate time depending on the importance and urgency of the information.

[0918] An "external interface" is an interface for connecting a system to external equipment or devices.

[0919] An "assist message" is a supplementary message to support or encourage the user.

[0920] A "database" is a system for systematically organizing and storing data.

[0921] A spreadsheet is a tool for managing, analyzing, and displaying data in rows and columns.

[0922] This invention is a system that provides real-time support to drivers of self-driving vehicles when they encounter technical terms or unknown words while driving. The system collects and analyzes the user's voice data and facial expression data to provide appropriate support information.

[0923] System configuration

[0924] The system consists of the following hardware and software components:

[0925] Hardware:

[0926] Microphone: Collects driver voice inside the vehicle.

[0927] Camera: Collects driver facial expression data.

[0928] On-board computer: processes data and provides information.

[0929] software:

[0930] Speech recognition API (e.g., Google Speech-to-Text API): Converts voice data into text data in real time.

[0931] Natural Language Processing (NLP) engines (e.g., spaCy, NLTK): Analyze text data and identify terminology and unknown words.

[0932] Emotion recognition module (e.g., Face API, Microsoft Azure Emotion API): Analyzes facial expression data and voice tone to recognize the driver's emotional state.

[0933] Real-time data collection and processing system: Collects voice and facial expression data and analyzes it in real time.

[0934] Program processing

[0935] The server collects voice data in real time and converts it into text using a speech recognition API. It then analyzes the text using an NLP engine to identify unknown words and technical terms. The meaning and context of the identified words are obtained from web data and historically recorded data. This meaning and context information is then provided to the user's device in real time.

[0936] Furthermore, the driver's facial expression data and voice tone captured by the camera are input into the emotion recognition module to recognize the driver's emotional state. The server adjusts the priority of information provision based on the recognized emotional state. If emotions indicating stress are recognized, detailed assistance information and relaxation messages will be provided.

[0937] For example, if a driver hears the term "ADAS (Advanced Driver Assistance Systems)" for the first time while driving, the system will respond as follows:

[0938] 1. Listen to the driver's voice and convert it into text.

[0939] 2. Identify "ADAS" using an NLP engine.

[0940] 3. Obtain the definition of "ADAS" and related information from the Web API.

[0941] 4. Cameras analyze the driver's facial expressions to identify their emotional state.

[0942] 5. Display information on the display and add assistance messages as needed.

[0943] Prompt Sentence Examples

[0944] A driver in a self-driving vehicle encounters the word "ADAS" for the first time. How does the system respond in the next step?

[0945] 1. Listen to the driver's voice and convert it into text.

[0946] 2. Identify "ADAS" using the NLP engine.

[0947] 3. Get the meaning of "ADAS" from the Web API.

[0948] 4. Analyze the driver's emotional state using a camera.

[0949] 5. Information is displayed on the screen, and if the driver is feeling stressed, a message to relax is also displayed.

[0950] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0951] Step 1:

[0952] The user's voice is collected by a microphone. The input is real-time voice data, and the output is digitized voice data. The device sends this voice data to a speech recognition API (e.g., Google Speech-to-Text API) to convert it into text information. The output is text data.

[0953] Step 2:

[0954] The device sends text data obtained from the speech recognition API to the server, which then analyzes the text data using a natural language processing (NLP) engine (e.g., spaCy, NLTK). The input is text data, and the output is a tokenized word list. The server identifies unknown words and technical terms from this list.

[0955] Step 3:

[0956] The server searches for the meaning and context of identified unknown words and technical terms from web data or historical record data. The input is a list of identified words, and the output is the meaning and context information of each word. The server uses a web API to obtain related information and searches for related context from the historical record database.

[0957] Step 4:

[0958] The camera collects the driver's facial expression data in real time. The input is real-time facial expression data, and the output is digitized facial expression data. The device sends this facial expression data to an emotion recognition module (e.g., Face API, Microsoft Azure Emotion API) for analysis. The output is the driver's emotional state.

[0959] Step 5:

[0960] The server integrates the emotional state obtained from the emotion recognition module with the semantic and contextual information of the words obtained in step 3. The input is the emotional state and semantic / contextual information, and the output is a list of information provision with adjusted priorities. The server automatically adjusts the priority of information provision according to the emotional state.

[0961] Step 6:

[0962] The server sends the prioritized information list to the terminal. The terminal displays the meaning of words and context information in real time on the display. If the driver shows signs of stress or anxiety, the terminal also displays a relaxation message. The input is the information list, and the output is the displayed information.

[0963] Prompt Sentence Examples

[0964] A driver in a self-driving vehicle encounters the word "ADAS" for the first time. How does the system respond in the next step?

[0965] 1. Listen to the driver's voice and convert it into text.

[0966] 2. Identify "ADAS" using the NLP engine.

[0967] 3. Get the meaning of "ADAS" from the Web API.

[0968] 4. Analyze the driver's emotional state using a camera.

[0969] 5. Information is displayed on the screen, and if the driver is feeling stressed, a message to relax is also displayed.

[0970] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0971] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0972] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0973] [Fourth embodiment]

[0974] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0975] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0976] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0977] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0978] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0979] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0980] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0981] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0982] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0983] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0984] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0985] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0986] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0987] The present invention relates to a real-time support system for technical terms and unknown words in online meetings. Each element of the system will be described in detail below.

[0988] First, to capture the audio data of users speaking during a meeting in real time, the user's device collects the audio data and converts it into text data using the Zoom API. The device accesses Zoom's live transcription function and specifies the meeting ID as an argument to obtain the content of the conversation as text information.

[0989] The captured text data is sent to a server, which analyzes it to identify unknown words and terminology. The server uses natural language processing (NLP) techniques to tokenize the text data and match it against a list of known words, including common and industry-standard words.

[0990] The identified unknown words are then processed by the server, which searches for their meaning and context from web data and past meeting records. The server queries web APIs to retrieve relevant information and searches past record databases for relevant context and previous usage examples.

[0991] Search results are provided from the server to the user's device. These results are formatted for user convenience and provided in real time. As a concrete example, consider the case where the word "ROI" is identified. The server retrieves the meaning of "ROI" (return on investment) from the web, adds detailed context based on usage examples from past meetings, and sends it to the user's device.

[0992] The server then sends the search results in the form of a message to an external interface, such as the Zoom chat, and simultaneously records this information in a database or spreadsheet using the Google Spreadsheet API, which is then saved for future reference, allowing for quick retrieval of past meeting information when needed.

[0993] As a concrete example, consider a situation where a user says during a meeting, "To maximize the ROI of this project, we first need to set KPIs." The speech is transcribed in real time, and the server identifies the words "ROI" and "KPI." The server then searches the internet and past meeting records for the meaning of these words, generates a message saying "ROI: Return on Investment, KPI: Key Performance Indicator," and sends it to the Zoom chat. This information is also recorded in a spreadsheet.

[0994] In this way, users can understand the meaning and context of technical terms in real time without disrupting the progress of the meeting, which can significantly improve the efficiency and understanding of meetings.

[0995] The processing flow will be explained below.

[0996] Step 1:

[0997] The device connects to the Zoom API to capture audio data in real time during the meeting, capturing what the user is saying as digital audio data on the spot.

[0998] Step 2:

[0999] The device converts the audio data it receives into text using Zoom's live transcription feature, which sends a request to Zoom using the meeting ID to convert the audio into text.

[1000] Step 3:

[1001] The terminal transmits the text data to the server, which then uploads the text data to the server via the network for analysis.

[1002] Step 4:

[1003] The server analyzes the received text data and uses natural language processing (NLP) technology to tokenize it, which divides the speech into individual words.

[1004] Step 5:

[1005] The server checks the tokenized words against a known word list, which contains common vocabulary and industry-standard terminology, to identify unknown words or technical terms.

[1006] Step 6:

[1007] The server queries web data and past record databases to find the meaning and context of the identified unknown words. It obtains relevant information from web APIs and gathers contextual information by searching past meeting records.

[1008] Step 7:

[1009] The server formats the collected information to generate semantic and contextual search results, including details such as word definitions and historical usage examples.

[1010] Step 8:

[1011] The server sends the formatted search results to the terminal. By sending the information to the user's terminal, the user can receive the information in real time.

[1012] Step 9:

[1013] The device will display the retrieved information to the user, and will also post the search results to the Zoom chat to share the information with other participants.

[1014] Step 10:

[1015] The server records the acquired information in a spreadsheet using the Google Spreadsheet API, so that the information can be referenced after the meeting.

[1016] Examples:

[1017] For example, if a user says during a meeting, "In order to maximize the ROI of this project, we first need to set KPIs," the process would go something like this:

[1018] 1. The device receives the speech and collects audio data using the Zoom API.

[1019] 2. The device converts the voice into text information and generates text data.

[1020] 3. The device sends the generated text data to the server.

[1021] 4. The server analyzes the received text data and identifies unknown words including "ROI" and "KPI."

[1022] 5. The server searches for the meaning and context of "ROI" and "KPI" from web data and historical record data.

[1023] 6. The server generates the results "ROI: Return on Investment" and "KPI: Key Performance Indicator."

[1024] 7. The server sends the generated results to the device, which displays the information to the user and posts it to the Zoom chat.

[1025] 8. The server records the obtained information in a spreadsheet using the Google Spreadsheet API.

[1026] In this way, users can understand the meaning and context of technical terms in real time without disrupting the flow of the meeting, allowing them to participate in the meeting efficiently.

[1027] Example 1

[1028] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1029] In online meetings, when technical terms or unfamiliar words appear, participants may be unable to keep up and the meeting progress may be delayed. Furthermore, when an unfamiliar word appears during a meeting, it is difficult to search for and understand it on the spot. This reduces the efficiency of the meeting and leads to insufficient understanding among participants. Furthermore, if there is no easy way to access the information later, it becomes difficult to reuse the information.

[1030] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1031] In this invention, the server includes a means for acquiring audio data of an online conference from a user's terminal in real time and converting the audio data into text data, a means for identifying unknown words or technical terms from the text data using natural language processing technology, and a means for searching for the meaning and context of the identified words from web data or past conference records. This allows users to grasp the meaning of technical terms and unknown words in real time, deepening their understanding without disrupting the progress of the conference. Furthermore, a means for sending search results as a message to an external interface and a means for recording them in a database or spreadsheet make it easy to reuse the same information later.

[1032] A "user terminal" is an electronic device such as a computer, tablet, or smartphone used by a user participating in an online conference.

[1033] "Audio data" refers to digital audio information recorded from an online conference conversation.

[1034] "Text data" is digital data that has been converted from voice data into character information.

[1035] "Natural language processing technology" is a technology for analyzing text data to identify unknown words and technical terms, and is a series of processing techniques including tokenization and morphological analysis.

[1036] "Unknown words or technical terms" are new words that are not included in known word lists or words that are technical and not commonly understood.

[1037] "Web data" refers to digital information that is publicly available on the Internet, including information that can be obtained through web pages or APIs.

[1038] "Past meeting records" refers to audio data and text data from previous meetings, the contents of which are stored in a database.

[1039] "Searching" is the act of referencing web data or past meeting records to find the meaning or context of a specified word.

[1040] "Formatting" refers to converting search results into something that is easy for users to understand, and organizing them into an appropriate format and structure.

[1041] "Providing in real time" refers to providing information immediately without waiting while users are conducting an online conference.

[1042] "Sending to an external interface" means sending the obtained information to another system or application, such as a messaging platform.

[1043] "Recording in a database or spreadsheet" means storing search results or meeting information in a digital format for future reference.

[1044] The present invention is a system that provides real-time support for technical terms and unknown words in online meetings. The system shown below is composed of multiple elements, including a user terminal, a server, an external interface, and a database or spreadsheet.

[1045] 1. Acquiring audio data and converting it to text

[1046] When a user starts an online meeting, the user's device captures the meeting's audio data in real time. Specifically, the user's device (PC, smartphone, tablet, etc.) uses the online meeting system to collect audio data. This device then converts the audio into text data using audio data capture technology such as the Zoom API. For example, the device accesses Zoom's live transcription function and converts the captured audio data into text in real time.

[1047] 2. Sending text data

[1048] The device sends the converted text data to the server using a communication protocol such as an HTTP POST request, where it is transferred to the server and prepared for analysis.

[1049] 3. Identifying Unknown Words and Terminology

[1050] The server processes the received text data using natural language processing (NLP) techniques, such as tokenizing the text and matching it against a list of known words to identify unknown words and terminology. This NLP includes morphological analysis, tokenization, and dictionary lookup of words.

[1051] 4. Searching for meaning and context

[1052] For the identified unknown words or terminology, the server searches for meaning and context from web APIs or a database of past meeting records. For example, the server may refer to web data or previous meeting records to clarify the meaning of the identified words. The program then performs a regular web search or internal resource search to extract relevant information.

[1053] 5. Providing search results

[1054] The server formats the search results for delivery to the user. Additionally, the server delivers the formatted information to the user's device in real time. This information is then sent as a message to an external interface, such as a Zoom chat. For example, the Zoom API can be used to post a description and context of the retrieved words to the chat. The same search results can also be saved to a database or spreadsheet using the Google Spreadsheet API for later reference.

[1055] Specific examples

[1056] For example, if a user says the following during a meeting:

[1057] "To maximize the ROI of this project, we first need to set KPIs."

[1058] The device captures this speech as audio data and converts it to text using the Zoom API:

[1059] "To maximize the ROI of this project, we first need to set KPIs."

[1060] The server identifies the words "ROI" and "KPI" and searches for their meanings in web data and past meeting records. For example, the server obtains results such as "ROI: Return on Investment" and "KPI: Key Performance Indicator."

[1061] The server sends the results as a message to the Zoom chat and simultaneously records them in a Google spreadsheet, providing real-time definitions of technical terms and supporting understanding during the meeting.

[1062] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1063] Step 1:

[1064] A user starts an online meeting. The user uses the online meeting system and is issued a meeting ID. The device starts collecting audio data via the Zoom API. The input is the trigger to start the meeting, and the output is the collected audio data.

[1065] Step 2:

[1066] The device uses Zoom's live transcription feature to convert audio data into text data in real time, directly using the Zoom API. The input is the collected audio data, and the output is the converted text data.

[1067] Step 3:

[1068] The terminal sends the converted text data to the server using an HTTP POST request. The input is text data, and the output is the transfer of the text data to the server.

[1069] Step 4:

[1070] The server analyzes the received text data using NLP techniques. Specifically, it tokenizes the text data and compares it with a list of known words to identify unknown words and technical terms. The input is the transmitted text data, and the output is the identified unknown words and technical terms.

[1071] Step 5:

[1072] The server searches for the meaning and context of the identified words from web data and a database of past meeting records. It uses a web API to obtain related information and reference past records. The input is the identified word, and the output is information about the meaning and context of that word.

[1073] Step 6:

[1074] The server formats the information it receives and converts it into something that is easy for the user to understand, for example, "ROI: Return on Investment." The input is the semantic and contextual information it receives, and the output is the formatted information.

[1075] Step 7:

[1076] The server delivers the formatted information to the user's device in real time. This information is sent as a message to an external interface such as Zoom chat. Furthermore, the information is recorded in a database or spreadsheet using the Google Spreadsheet API. The input is the formatted information, and the output is sending a message to Zoom chat and recording it in the database.

[1077] Specific examples of behavior:

[1078] For example, consider the case where a user says during a meeting, "To maximize the ROI of this project, we first need to set KPIs."

[1079] Step 1: A user starts a conference and audio data collection begins.

[1080] Step 2: The device translates the speech into "To maximize the ROI of this project, we first need to set KPIs."

[1081] Step 3: The device sends this text data to the server.

[1082] Step 4: The server identifies the words "ROI" and "KPI."

[1083] Step 5: The server obtains information such as "ROI: Return on Investment" and "KPI: Key Performance Indicator" from web data and past meeting records.

[1084] Step 6: The server formats this information.

[1085] Step 7: The server sends the formatted information to the Zoom chat and records it in a Google spreadsheet.

[1086] (Application example 1)

[1087] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1088] When a large amount of technical terms and unfamiliar words are used in online meetings or virtual stores, participants and customers often cannot understand their meaning or context, resulting in a decrease in communication efficiency. This can also impair the customer's purchasing experience, so a solution is needed.

[1089] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1090] In this invention, the server includes means for converting speech data acquired in real time into text data, means for identifying unknown words or technical terms from the text data, means for searching for the meaning and context of the identified words from web data or past recorded data, means for providing the search results to the user's terminal in real time, means for providing customer support functions in a virtual store, and means for overlaying related information on an interface used by the customer. This enables users and customers to understand the meaning of technical terms and unknown words used during meetings or while shopping in real time, thereby improving communication efficiency and the purchasing experience.

[1091] The "means for converting voice data acquired in real time into text data" refers to a technical means for instantly converting the voice spoken by a user into text information.

[1092] "Means for identifying unknown words or technical terms from text data" refers to technical means for analyzing acquired character information to find words that are not generally recognized or technical terms used in a specific field.

[1093] "Means for searching for the meaning and context of identified words from web data or historical records" refers to technical means for searching for the meaning and usage examples of identified unknown words or technical terms from data on the Internet or historical records.

[1094] The "means for providing search results to a user's terminal in real time" refers to a technical means for instantly displaying the obtained search results on the device used by the user.

[1095] "Means for providing customer support functions in virtual stores" refers to technical means for providing support functions to enable customers to enjoy shopping comfortably in stores operated online.

[1096] The "means for overlaying and displaying relevant information on the interface used by the customer" refers to a technical means for overlaying and displaying relevant information on the display device used by the customer.

[1097] To implement this invention, the following system configuration is required. First, the system collects voice data uttered by the user in real time and converts it into text data. This is done using the Google Speech Recognition API and the speech_recognition library. This voice data can be obtained from various input devices used by the user, such as smartphones, smart glasses, and head-mounted displays.

[1098] Next, the server analyzes the retrieved text data to identify unknown words and terminology. This step uses natural language processing (NLP) techniques to tokenize the text data and match it with a list of known words. For each unknown word identified, the server searches for its meaning and context from web data and historical record data, and sends a query to a web API to retrieve its meaning. It also searches historical record databases for related context and previous usage examples.

[1099] Specifically, when words such as "ROI" (return on investment) or "KPI" (key performance indicator) are identified, the server retrieves their meanings from the Internet and past meeting records to add detailed context. In a virtual store, if a customer says, "I want to know what the ROI of this product is," the system retrieves the meaning of "ROI" and related information in real time and displays it as an overlay on the customer's interface.

[1100] This allows customers to understand the meaning of technical terms and unknown words used in the virtual store in real time, improving the shopping experience. The server then sends the search results in the form of messages to an external interface, which can include, for example, the virtual store's chat function. At the same time, the information is recorded in a database or spreadsheet using the Google Spreadsheet API for future reference.

[1101] Examples of prompt sentences include:

[1102] "What is ROI?"

[1103] "I want to know how this product's ROI will be affected."

[1104] It is entered in the form:

[1105] This allows you to understand the meaning and context of technical terms and unknown words in real time, significantly improving meeting efficiency and purchasing experiences.

[1106] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1107] Step 1:

[1108] The device collects the user's voice data and acquires it in real time. The device can be a smartphone, smart glasses, or a head-mounted display. The input is the user's voice data, and the output is the acquired voice data. The collected voice data is passed to the Google Speech Recognition API, where it is prepared to be converted into text data.

[1109] Step 2:

[1110] The device converts the acquired voice data into text data using the Google Speech Recognition API. This process uses the speech_recognition library. The input is voice data, and the output is the converted text data. This text data is analyzed in the next step.

[1111] Step 3:

[1112] The server receives the text data sent from the terminal and tokenizes it using natural language processing (NLP) technology. The input is the text data, and the output is a tokenized word list. This process separates each word in the text.

[1113] Step 4:

[1114] The server matches the tokenized word list against a known word list to identify unknown words or terminology. The input is the tokenized word list, and the output is a list of identified unknown words. The known word list includes common words and industry-standard words.

[1115] Step 5:

[1116] The server searches for the meanings of identified unknown words from web data and historical data. This process involves sending queries using external web APIs to retrieve information. It also searches historical databases for relevant contexts and usage examples. The input is a list of identified unknown words, and the output is the meaning and context information for each word.

[1117] Step 6:

[1118] The server formats the acquired semantic and contextual information into a message and provides it to the user's device in real time. The input is the semantic and contextual information of each word, and the output is a formatted message, which is overlaid on the device's interface.

[1119] Step 7:

[1120] The server sends the search results as a message to an external interface, for example, a chat function of a virtual store. The input is a formatted message, and the output is a message displayed in the chat interface used by the user.

[1121] Step 8:

[1122] The server simultaneously records the retrieved information in a database or spreadsheet using the Google Spreadsheet API. The input is a formatted message, and the output is a recorded database or spreadsheet. This allows for quick retrieval of the information later.

[1123] Step 9:

[1124] It understands the meaning of technical terms and unknown words in real time based on the relevant information provided by the user through the interface. The input is a formatted message, and the output is improved user comprehension, thereby improving the user's shopping experience in the virtual store.

[1125] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1126] This invention combines a real-time support system for technical terms and unknown words in online meetings with an emotion engine that recognizes the user's emotions and optimizes the provision of support information. Each element of the system is described in detail below.

[1127] First, to obtain real-time voice data from users during a meeting, the user's device collects the voice data and converts it into text data using a voice recognition function. The device connects to a voice recognition API and converts the voice data into text information in real time.

[1128] The captured text data is sent to a server, which analyzes it to identify unknown words and terminology. The server uses natural language processing (NLP) techniques to tokenize the text data and match it against a list of known words, including common and industry-standard words.

[1129] The identified unknown words are then processed by the server, which searches for their meaning and context from web data and historical records. The server collects semantic and contextual information by querying web APIs to retrieve relevant information and searching historical databases for relevant context and previous usage examples.

[1130] The search results are delivered to the user's device from the server, and are displayed in real time in a user-friendly format, including detailed information such as word definitions and past usage examples. The most important information is posted to the Zoom chat and shared with other participants.

[1131] Furthermore, the system is equipped with an emotion engine that recognizes emotions from the user's speech and facial expressions. The device inputs the user's facial expression data and voice tone into an emotion analysis module to identify the user's current emotional state. The server combines this emotional information with the importance of words to automatically adjust the priority of information provision.

[1132] A specific use case is when a user feels stressed during a meeting. For example, if the user says, "To maximize the ROI of this project, we first need to set KPIs," and an emotion indicating stress is recognized, the server identifies the words "ROI" and "KPI" and searches for their meaning and context, while recognizing the user's stress level and adjusting the priority of information provision. If an emotion indicating stress is recognized, the server provides detailed assistance information and sends additional messages to help the user relax.

[1133] Additionally, search results are recorded in a spreadsheet using the Google Sheets API, and the records are saved for future reference, allowing users to quickly search for past meeting information when needed.

[1134] In this way, users can understand the meaning and context of technical terms in real time without disrupting the flow of the meeting, while receiving optimal support according to their emotional state, significantly improving meeting efficiency and understanding.

[1135] The processing flow will be explained below.

[1136] Step 1:

[1137] The device captures audio data during the meeting in real time, connects to a speech recognition API, and collects the user's speech as digital audio data.

[1138] Step 2:

[1139] The device converts the acquired voice data into text data in real time, allowing the spoken content to be recorded as text information.

[1140] Step 3:

[1141] The terminal transmits the converted text data to the server, which then sends the text data over the network for analysis on the server side.

[1142] Step 4:

[1143] The server analyzes the received text data and uses natural language processing (NLP) technology to tokenize it, dividing the speech into words.

[1144] Step 5:

[1145] The server checks the tokenized words against a known word list, which includes common vocabulary and industry-standard terminology, to identify unknown words or technical terms.

[1146] Step 6:

[1147] The server queries web data and historical record databases to find the meaning and context of the identified unknown words, using Web APIs to retrieve relevant information and collect relevant contextual information from the historical record database.

[1148] Step 7:

[1149] The server formats the collected information and generates search results that include semantic and contextual information, optionally adding details such as word definitions and previous usage examples.

[1150] Step 8:

[1151] The server provides the results to the user's device, which displays the information to the user in real time and can also post it to the Zoom chat.

[1152] Step 9:

[1153] The device inputs the user's facial expression data and voice tone into an emotion engine to recognize the user's emotional state, which then analyzes the user's emotions in real time.

[1154] Step 10:

[1155] The server automatically adjusts the priority of information provision based on the recognized emotional state. If the user is feeling stressed, detailed assistance information and relaxation messages will be provided first.

[1156] Step 11:

[1157] The server records the search results in a Google Spreadsheet, using the Google Spreadsheet API to store the retrieved information for future reference.

[1158] Examples:

[1159] Let's take the example of a user saying during a meeting, "To maximize the ROI of this project, we first need to set KPIs."

[1160] 1. The device receives the speech and collects voice data using a speech recognition API.

[1161] 2. The device converts the voice data into text data in real time.

[1162] 3. The device sends the generated text data to the server.

[1163] 4. The server analyzes the received text data and identifies the unknown words "ROI" and "KPI."

[1164] 5. The server searches for the meaning and context of "ROI" and "KPI" from web data and historical record databases.

[1165] 6. The server generates the results "ROI: Return on Investment" and "KPI: Key Performance Indicator."

[1166] 7. The server sends the results to the user's device, which displays the information in real time and also posts it to the Zoom chat.

[1167] 8. The device inputs the user's facial expression data and voice tone into an emotion engine to analyze the user's emotional state.

[1168] 9. The server adjusts the priority of information provision based on the recognized emotional state. If the user is feeling stressed, it will provide detailed assistance information and also send a relaxation message.

[1169] 10. The server records the search results in a spreadsheet using the Google Spreadsheet API.

[1170] In this way, users can understand the meaning and context of technical terms in real time without disrupting the flow of the meeting, while receiving optimal support according to their emotional state, significantly improving meeting efficiency and understanding.

[1171] Example 2

[1172] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1173] When online meetings contain a large amount of technical terms and unfamiliar words, participants may find it difficult to understand them, reducing the efficiency of the meeting and the level of understanding of the content. Furthermore, participants' emotional states can affect their comprehension and concentration, but existing systems lack support that takes this into account. Therefore, there is a need for a system that can recognize users' emotions and provide optimal information, along with real-time support for technical terms and unfamiliar words.

[1174] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1175] In this invention, the server includes means for converting speech data acquired in real time into text data, means for identifying unknown words or technical terms from the text data, means for searching for the meaning and context of the identified words from web data or past recorded data, means for detecting the user's emotions, and means for adjusting the order in which search results are presented based on the emotions. This makes it easier for the user to understand the meaning of technical terms and unknown words in real time, and enables the user to receive information optimal for their emotional state.

[1176] "Voice data" refers to information that includes a user's speech or voice, expressed in digital form.

[1177] "Text data" is voice data converted into text information, and is data written in natural language.

[1178] "Unknown words" are words or phrases that are not included in a predefined word list and require context and semantic understanding.

[1179] "Jargon" refers to specialized words and phrases used in a particular field or industry that are necessary for understanding that field.

[1180] "Web data" is information publicly available on the Internet and is data obtained using natural language processing and other technologies.

[1181] "Historical data" refers to information collected or generated in the past or data stored in a database that is used for reference or retrieval.

[1182] "Search results" are data containing information about the meaning and context of identified words that are provided to the user.

[1183] A "terminal" is a computing device used by a user to collect voice data, display text data, and perform other processing.

[1184] A "server" is a computing device that provides services over a network, such as analyzing, retrieving, generating, and providing data.

[1185] "Emotion" refers to the user's mental state or psychological reaction, and indicates states such as stress, relaxation, and concentration.

[1186] "Means for detecting emotions" refers to technology or devices that analyze the user's voice and facial expression data to identify their emotional state.

[1187] "Means for adjusting the order in which search results are presented" refers to techniques or methods for determining the priority of information to be presented based on the user's emotional state or other factors.

[1188] The present invention provides a real-time support system for technical terms and unknown words in online meetings, and further combines it with an emotion engine that recognizes the user's emotions and optimizes the provision of support information. An embodiment of this system will be described in detail.

[1189] First, the user's device collects audio data in real time to capture the user's speech during the meeting. A microphone is connected to the device to capture what is being said during the meeting. Next, the device uses a speech recognition API (e.g., Google Speech-to-Text API) to convert the collected audio data into text data. The audio data is sent to the API, and the returned text data is obtained.

[1190] The captured text data is sent from the device to a server, where it is analyzed. The server first tokenizes the text data (for example, breaking down "To maximize the ROI of this project, we first need to set KPIs" into individual words) and identifies unknown words and technical terms by comparing them with a list of known words. This list of known words includes pre-defined common words and industry-standard words.

[1191] The server then searches for the meaning and context of the identified unknown words using a Web API (e.g., Google Knowledge Graph API), and also searches for related contexts and previous usage examples from a historical record database to gather semantic and contextual information, resulting in the meaning and usage examples of words such as "ROI" and "KPI."

[1192] The search results are delivered to the user's device in real time from the server. Specifically, detailed text including word definitions, past usage examples, and related information is sent to the user's device. Important information is shared with other participants using the chat function of the meeting app (e.g., Zoom).

[1193] The system is also equipped with an emotion engine that recognizes emotions from the user's speech and facial expression data. The device uses a camera and microphone to collect the user's facial expression data and voice tone, which are then input into an emotion analysis module. The server analyzes this data to identify the user's emotional state. For example, a deep learning model can be used to generate emotion labels such as "stress," "relaxed," and "concentrated."

[1194] The server adjusts the order of search results based on the identified emotional information and the importance of unknown words. If the user is feeling stressed, detailed explanations and relaxation messages will be provided first. In this way, users can easily understand the meaning of technical terms and unknown words in real time during meetings and receive the most appropriate information according to their emotional state.

[1195] Finally, the search results are recorded in a spreadsheet using the Google Spreadsheet API from the server, allowing users to quickly reference past meeting information and avoid having to search for the same information again if necessary.

[1196] As a concrete example, suppose a user says during a meeting, "To maximize the ROI of this project, we first need to set KPIs." The device collects this speech as voice data and uses a speech recognition API to convert it into text data: "To maximize the ROI of this project, we first need to set KPIs." This text data is sent to the server, where "ROI" and "KPI" are identified as unknown words. The server searches for their meaning and context and provides detailed information to the user's device. If the user is recognized as feeling stressed, detailed support information and a relaxation message are sent. This information is recorded in a Google spreadsheet and can be referenced later.

[1197] An example prompt might be, "To get the most out of your real-time support during the meeting, please explain specifically what your ROI and KPIs mean. Also, please elaborate on other metrics that are important to the success of this project."

[1198] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1199] Step 1: Collect speech data and send it to the speech recognition API

[1200] Audio data of users speaking during a meeting is collected in real time using a microphone connected to the user's device. The device then sends the collected audio data to a speech recognition API (e.g., Google Speech-to-Text API). The input is real-time audio data, and the output is text data returned by the speech recognition API.

[1201] Step 2: Convert the audio data to text

[1202] The device receives the text data returned from the speech recognition API. Specific operations include processing HTTP requests to the speech recognition API and responses. The input is audio data from the speech recognition API, and the output is text data.

[1203] Step 3: Send the text data to the server

[1204] The terminal sends the converted text data to the server using an HTTP POST request, where the input is the text data and the output is a confirmation of transmission to the server.

[1205] Step 4: Tokenize the text data

[1206] The server tokenizes the received text data. Specifically, it splits the text into words. The input is the text data, and the output is a tokenized word list.

[1207] Step 5: Identify unknown words and terminology

[1208] The server checks the tokenized word list against a list of known words to identify unknown words or terminology. Specific operations include database query comparisons against the known word list. The input is the tokenized word list, and the output is the unknown word list.

[1209] Step 6: Search for the meaning and context of unknown words

[1210] The server queries the identified unknown words to a Web API (e.g., Google Knowledge Graph API) and an internal database to find their meaning and context. Specific operations include generating and sending API queries and analyzing the retrieved data. The input is a list of unknown words, and the output is semantic and context information.

[1211] Step 7: Format the search results and send them to your device

[1212] The server formats the semantic and contextual information it has obtained into a form that is easy for the user to use. Specific operations include converting it to HTML or text format. The input is the semantic and contextual information, and the output is the formatted information. The server then sends the formatted information to the device. The input is the formatted information, and the output is a confirmation of transmission to the device.

[1213] Step 8: Collect emotional data and identify emotional states

[1214] The device inputs the user's facial expression data and voice tone into the emotion analysis module. Specific operations include collecting data using a camera and microphone and analyzing it using an emotion analysis algorithm. The input is facial expression data and voice tone, and the output is the user's emotional state.

[1215] Step 9: Adjust the order of information based on emotional information

[1216] The server combines the acquired emotional information with the importance information of unknown words and automatically adjusts the order in which search results are presented. Specific operations include prioritizing the results based on the emotional state and the importance of words. The input is emotional information and importance information, and the output is the adjusted order in which the results are presented.

[1217] Step 10: Send search results to an external interface and record them in a spreadsheet

[1218] The server sends the search results as a message to an external interface (e.g., the chat function of a conference app). It then records the search results in a spreadsheet using the Google Spreadsheet API. Specific operations include sending and recording data using the API. The input is the search results, and the output is confirmation of sending to the external interface and recording in the spreadsheet.

[1219] (Application example 2)

[1220] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1221] When a driver in an autonomous vehicle encounters technical terms or unfamiliar words while driving, it is difficult to provide the necessary information for driving due to a lack of meaning and context information. It is also difficult to provide information at the appropriate time based on the driver's emotional state. This can lead to a decrease in driving safety and efficiency.

[1222] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for converting voice data acquired in real time into text data, means for identifying unknown words or technical terms from the text data, means for searching for the meaning and context of the identified words from web data or past recorded data, means for providing the search results to the user's terminal in real time, means for analyzing facial expression data or voice tone to recognize the driver's emotional state, and means for adjusting the priority of information provision based on the recognized emotional state. This makes it possible to provide information at the optimal timing according to the driver's emotional state while supporting the driver's understanding of technical terms and unknown words.

[1223] "Audio data" is a digital representation of an audio signal captured in real time.

[1224] "Text data" refers to data that has been converted into character information based on voice data.

[1225] "Unknown words" refer to technical terms or new terms that are not generally understood by users.

[1226] "Terminology" refers to specialized terms used in a particular industry or field.

[1227] "Web data" is data that includes information on the Internet.

[1228] "Previously recorded data" is data that has been previously acquired or stored.

[1229] "Providing in real time" means providing information instantly, without delay.

[1230] "User terminal" refers to the device used by a User.

[1231] "Emotional state" refers to the psychological state of a user as perceived from their facial expression and tone of voice.

[1232] "Facial expression data" refers to data relating to the facial expression of the user captured by a camera or the like.

[1233] "Voice tone" refers to the voice characteristics obtained from the intonation and stress of the voice.

[1234] "Adjusting the priority of information provision" means providing information at an appropriate time depending on the importance and urgency of the information.

[1235] An "external interface" is an interface for connecting a system to external equipment or devices.

[1236] An "assist message" is a supplementary message to support or encourage the user.

[1237] A "database" is a system for systematically organizing and storing data.

[1238] A spreadsheet is a tool for managing, analyzing, and displaying data in rows and columns.

[1239] This invention is a system that provides real-time support to drivers of self-driving vehicles when they encounter technical terms or unknown words while driving. The system collects and analyzes the user's voice data and facial expression data to provide appropriate support information.

[1240] System configuration

[1241] The system consists of the following hardware and software components:

[1242] Hardware:

[1243] Microphone: Collects driver voice inside the vehicle.

[1244] Camera: Collects driver facial expression data.

[1245] On-board computer: processes data and provides information.

[1246] software:

[1247] Speech recognition API (e.g., Google Speech-to-Text API): Converts voice data into text data in real time.

[1248] Natural Language Processing (NLP) engines (e.g., spaCy, NLTK): Analyze text data and identify terminology and unknown words.

[1249] Emotion recognition module (e.g., Face API, Microsoft Azure Emotion API): Analyzes facial expression data and voice tone to recognize the driver's emotional state.

[1250] Real-time data collection and processing system: Collects voice and facial expression data and analyzes it in real time.

[1251] Program processing

[1252] The server collects voice data in real time and converts it into text using a speech recognition API. It then analyzes the text using an NLP engine to identify unknown words and technical terms. The meaning and context of the identified words are obtained from web data and historically recorded data. This meaning and context information is then provided to the user's device in real time.

[1253] Furthermore, the driver's facial expression data and voice tone captured by the camera are input into the emotion recognition module to recognize the driver's emotional state. The server adjusts the priority of information provision based on the recognized emotional state. If emotions indicating stress are recognized, detailed assistance information and relaxation messages will be provided.

[1254] For example, if a driver hears the term "ADAS (Advanced Driver Assistance Systems)" for the first time while driving, the system will respond as follows:

[1255] 1. Listen to the driver's voice and convert it into text.

[1256] 2. Identify "ADAS" using an NLP engine.

[1257] 3. Obtain the definition of "ADAS" and related information from the Web API.

[1258] 4. Cameras analyze the driver's facial expressions to identify their emotional state.

[1259] 5. Display information on the display and add assistance messages as needed.

[1260] Prompt Sentence Examples

[1261] A driver in a self-driving vehicle encounters the word "ADAS" for the first time. How does the system respond in the next step?

[1262] 1. Listen to the driver's voice and convert it into text.

[1263] 2. Identify "ADAS" using the NLP engine.

[1264] 3. Get the meaning of "ADAS" from the Web API.

[1265] 4. Analyze the driver's emotional state using a camera.

[1266] 5. Information is displayed on the screen, and if the driver is feeling stressed, a message to relax is also displayed.

[1267] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1268] Step 1:

[1269] The user's voice is collected by a microphone. The input is real-time voice data, and the output is digitized voice data. The device sends this voice data to a speech recognition API (e.g., Google Speech-to-Text API) to convert it into text information. The output is text data.

[1270] Step 2:

[1271] The device sends text data obtained from the speech recognition API to the server, which then analyzes the text data using a natural language processing (NLP) engine (e.g., spaCy, NLTK). The input is text data, and the output is a tokenized word list. The server identifies unknown words and technical terms from this list.

[1272] Step 3:

[1273] The server searches for the meaning and context of identified unknown words and technical terms from web data or historical record data. The input is a list of identified words, and the output is the meaning and context information of each word. The server uses a web API to obtain related information and searches for related context from the historical record database.

[1274] Step 4:

[1275] The camera collects the driver's facial expression data in real time. The input is real-time facial expression data, and the output is digitized facial expression data. The device sends this facial expression data to an emotion recognition module (e.g., Face API, Microsoft Azure Emotion API) for analysis. The output is the driver's emotional state.

[1276] Step 5:

[1277] The server integrates the emotional state obtained from the emotion recognition module with the semantic and contextual information of the words obtained in step 3. The input is the emotional state and semantic / contextual information, and the output is a list of information provision with adjusted priorities. The server automatically adjusts the priority of information provision according to the emotional state.

[1278] Step 6:

[1279] The server sends the prioritized information list to the terminal. The terminal displays the meaning of words and context information in real time on the display. If the driver shows signs of stress or anxiety, the terminal also displays a relaxation message. The input is the information list, and the output is the displayed information.

[1280] Prompt Sentence Examples

[1281] A driver in a self-driving vehicle encounters the word "ADAS" for the first time. How does the system respond in the next step?

[1282] 1. Listen to the driver's voice and convert it into text.

[1283] 2. Identify "ADAS" using the NLP engine.

[1284] 3. Get the meaning of "ADAS" from the Web API.

[1285] 4. Analyze the driver's emotional state using a camera.

[1286] 5. Information is displayed on the screen, and if the driver is feeling stressed, a message to relax is also displayed.

[1287] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1288] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1289] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1290] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1291] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1292] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1293] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1294] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1295] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1296] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1297] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1298] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1299] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1300] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1301] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1302] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1303] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1304] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1305] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1306] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1307] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1308] The following is further disclosed regarding the above embodiment.

[1309] (Claim 1)

[1310] A means for converting the voice data acquired in real time into text data;

[1311] means for identifying unknown words or terminology from said text data;

[1312] A means for searching the meaning and context of the identified words from web data or historical record data;

[1313] means for providing the search results to a user's terminal in real time;

[1314] A system including:

[1315] (Claim 2)

[1316] 10. The system of claim 1, further comprising: means for transmitting the search results as a message to an external interface.

[1317] (Claim 3)

[1318] 10. The system of claim 1, further comprising means for recording the search results in a database or spreadsheet.

[1319] "Example 1"

[1320] (Claim 1)

[1321] A means for acquiring audio data of an online conference from a user's terminal in real time and converting the audio data into text data;

[1322] means for identifying unknown words or technical terms from the text data using natural language processing techniques;

[1323] A means for searching the meaning and context of the identified words from web data or past meeting records;

[1324] means for formatting the search results and providing them to a user's terminal in real time;

[1325] A system including:

[1326] (Claim 2)

[1327] 10. The system of claim 1, further comprising: means for transmitting the search results as a message to an external interface.

[1328] (Claim 3)

[1329] 10. The system of claim 1, further comprising means for recording the search results in a database or spreadsheet.

[1330] "Application Example 1"

[1331] (Claim 1)

[1332] A means for converting the voice data acquired in real time into text data;

[1333] means for identifying unknown words or terminology from said text data;

[1334] A means for searching the meaning and context of the identified words from web data or historical record data;

[1335] means for providing the search results to a user's terminal in real time;

[1336] A means for providing customer support functions in a virtual store;

[1337] a means for overlaying relevant information onto a customer interface;

[1338] A system including:

[1339] (Claim 2)

[1340] 10. The system of claim 1, further comprising: means for transmitting the search results as a message to an external interface.

[1341] (Claim 3)

[1342] 10. The system of claim 1, further comprising means for recording the search results in a database or spreadsheet.

[1343] "Example 2: Combining Emotion Engines"

[1344] (Claim 1)

[1345] A means for converting the voice data acquired in real time into text data;

[1346] means for identifying unknown words or terminology from said text data;

[1347] A means for searching the meaning and context of the identified words from web data or historical record data;

[1348] means for providing the search results to a user's terminal in real time;

[1349] means for detecting a user's emotion;

[1350] means for adjusting the order in which search results are provided based on the emotion;

[1351] A system including:

[1352] (Claim 2)

[1353] 10. The system of claim 1, further comprising: means for transmitting the search results as a message to an external interface.

[1354] (Claim 3)

[1355] 10. The system of claim 1, further comprising means for recording the search results in a database or information recording system.

[1356] "Application example 2 when combining emotion engines"

[1357] (Claim 1)

[1358] A means for converting the voice data acquired in real time into text data;

[1359] means for identifying unknown words or terminology from said text data;

[1360] A means for searching the meaning and context of the identified words from web data or historical record data;

[1361] means for providing the search results to a user's terminal in real time;

[1362] means for analyzing facial expression data or vocal tone to recognize emotional states;

[1363] a means for adjusting information provision priorities based on the perceived emotional state;

[1364] A system including:

[1365] (Claim 2)

[1366] means for transmitting the search results as a message to an external interface;

[1367] The system according to claim 1 , further comprising means for additionally displaying an assist message based on the emotion recognized in the external interface.

[1368] (Claim 3)

[1369] 10. The system according to claim 1, further comprising means for recording the search results or sentiment analysis results in a database or spreadsheet. [Explanation of symbols]

[1370] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for converting the voice data acquired in real time into text data; means for identifying unknown words or terminology from said text data; A means for searching the meaning and context of the identified words from web data or historical record data; means for providing the search results to a user's terminal in real time; A system including:

2. The system of claim 1 further comprising means for transmitting the search results as a message to an external interface.

3. The system of claim 1 further comprising means for recording the search results in a database or a spreadsheet.

Images