system

Abstract

Provide a system. An input device for acquiring user voice or text data; a communication means for transmitting data acquired by the input device means to a server device; A means for preprocessing the acquired data in the server device to perform noise removal and text conversion; A means for analyzing the data converted into text and performing keyword extraction and context analysis; A means for classifying conversation content into specific scopes based on the analysis results; means for storing the classified data based on scope and sending a notification to an appropriate output device; means for receiving user feedback to train and refine said classifier; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] With the recent advancement of technology, users are now engaging in a variety of conversations and communications across a variety of devices. However, it is difficult to properly classify the content of these conversations and manage and provide information appropriate to each. In particular, there is currently no system that can automatically identify conversations based on different scopes, such as those with family, friends, or superiors at work, and provide appropriate information. Therefore, there is a need for a system that allows users to centrally manage conversations across different scopes and output appropriate information. [Means for solving the problem]

[0005] To solve the above problems, the present invention provides the following means. First, an input device means for acquiring user voice or text data is provided. Then, a communication means for transmitting the acquired data to a server device is provided. Next, a means for preprocessing the acquired data in the server device, performing noise removal and text conversion is provided. Furthermore, a means for analyzing the converted text data and performing keyword extraction and context analysis is provided. Then, a means for classifying the conversation content into specific scopes based on the analysis results is provided. Finally, a means for saving the data classified based on the scope and sending a notification to an appropriate output device is provided. Furthermore, a means for receiving user feedback and training and improving the accuracy of the classification means is provided, thereby providing a system including these means. In this way, a system is realized that can automatically analyze and classify various user conversations and provide information appropriate to each of them.

[0006] The "input device means" is a device for acquiring voice and text data from the user.

[0007] The "communication means" is a means for transmitting the acquired data to the server device.

[0008] The "server device" is a device that receives and processes data transmitted from the input device means.

[0009] "Preprocessing" refers to a process of removing noise from the acquired voice data and converting it into clear voice data.

[0010] "Noise removal" is a process for removing unnecessary noise from audio data.

[0011] "Text conversion" is the process of converting voice data into character data.

[0012] The "analysis means" is a means for analyzing keywords and context based on text data.

[0013] "Keyword extraction" is the process of extracting important words and phrases from text data.

[0014] "Contextual analysis" is the process of understanding the content and meaning of text and extracting relevant information.

[0015] "Scope" is a classification category based on the type of conversation and the people involved.

[0016] The "scope classification means" is a means for classifying the conversation contents into specific scopes based on the analysis results.

[0017] "Storage means" refers to a means for storing classified data in a database or storage device.

[0018] An "output device" is a device that displays stored information to a user or sends notifications.

[0019] "Feedback" is evaluation and correction information provided by the user to the system.

[0020] The "learning means" is a means for improving classification accuracy based on user feedback. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0022] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0023] First, the terms used in the following description will be explained.

[0024] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0025] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0026] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0027] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0028] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0029] [First embodiment]

[0030] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0031] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0032] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0033] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0034] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0035] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0036] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0037] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0038] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0039] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0040] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0041] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0042] This invention is a system that automatically identifies various conversational contents of users and classifies them into appropriate scopes to manage and provide information appropriate to each of them. A specific embodiment of this system is shown below.

[0043] System Overview

[0044] 1. Capturing User Input

[0045] The user starts talking to the terminal, which has the function of capturing the user's voice and text data.

[0046] The device uses a microphone and keyboard to capture the user's voice and text data in real time, and the captured data is immediately sent to the server.

[0047] 2. Preprocessing of audio data

[0048] The server receives the data sent from the device. Since the received voice data may contain noise, it first applies a noise reduction algorithm to generate clear voice data.

[0049] The clear audio data is then converted into text data using a speech recognition model (e.g., a speech-to-text API).

[0050] 3. Analysis using natural language processing

[0051] The server analyzes the text data, extracts keywords, and performs context analysis. This analysis uses natural language processing technology. The analysis identifies the content of the conversation.

[0052] Based on the analysis results, the server classifies the conversation content into a specific scope (e.g., family, boss, friends, etc.) using a pre-defined algorithm.

[0053] 4. Information management by scope

[0054] The server stores data classified by scope in a database or storage device, and sends different information to the appropriate output device (such as a display or notification system) based on the scope.

[0055] For example, family-related conversations will be displayed on a living room display, while work-related conversations will be sent to a PC or smartphone at work.

[0056] 5. Processing and Learning from User Feedback

[0057] Users can provide feedback on the system's output, for example, if an item is classified into an incorrect scope, the user can provide feedback to correct it.

[0058] The server receives user feedback and uses it to retrain the machine learning model, improving the system's classification accuracy.

[0059] Specific use cases

[0060] Scenario 1: Conversation with family

[0061] 1. A user is having a conversation with a family member in the living room, and the conversation is captured on the device.

[0062] 2. The device sends the audio data to the server.

[0063] 3. The server removes noise from the audio data and converts it into text.

[0064] 4. The server analyzes the text data and identifies the "family" and scope.

[0065] 5. The server displays information about the family on the living room display.

[0066] Scenario 2: A conversation with your boss

[0067] 1. A user is talking to their boss in the office, and the conversation is captured by a headset.

[0068] 2. The device sends the audio data to the server.

[0069] 3. The server removes noise from the audio data and converts it into text.

[0070] 4. The server analyzes the text data and identifies the "boss" and scope.

[0071] 5. The server notifies the work-related information to the PC or smartphone at work.

[0072] Through these processes, the content of user conversations is automatically analyzed and appropriately categorized, allowing users to seamlessly manage different types of conversations and utilize information efficiently.

[0073] The processing flow will be explained below.

[0074] Step 1:

[0075] The user begins speaking. Audio capture begins in the living room, office, or other environment.

[0076] Step 2:

[0077] The device uses a built-in microphone or an external microphone to capture the user's voice data, which is collected in real time and sent to a server (possibly temporarily stored locally).

[0078] Step 3:

[0079] The device uses a communication means to send the captured audio data to a server, which can be via an internet connection or a local network.

[0080] Step 4:

[0081] The server receives the voice data from the device. Since the received data is likely to contain noise as it is, a noise reduction algorithm is first applied to it.

[0082] Step 5:

[0083] The server applies noise reduction algorithms to convert the audio data into a clearer version. This step uses digital signal processing techniques to remove background noise and unwanted sounds.

[0084] Step 6:

[0085] The server uses a speech recognition model (e.g., Google® Speech-to-Text API) to convert the noise-removed audio data into text data, which is then converted into readable text data.

[0086] Step 7:

[0087] The server analyzes the converted text data. In this analysis step, natural language processing (NLP) techniques are used to extract keywords and perform context analysis. The meaning and importance of each part of the text data are then extracted.

[0088] Step 8:

[0089] The server classifies the text data into specific scopes based on the extracted keywords and context information, such as "family," "boss," and "friends."

[0090] Step 9:

[0091] The server stores the classified data. Based on the scope of classification, it is stored in the appropriate database or storage device. For example, data about family members is stored in the user's household database.

[0092] Step 10:

[0093] The server then sends the stored data to the appropriate output device: depending on the scope, family information might be sent to a display in the living room, and work-related information might be sent to a PC or smartphone at work.

[0094] Step 11:

[0095] The user provides feedback on the system's output, for example if a classification is incorrect or if additional information is needed, and the user provides that information to the system.

[0096] Step 12:

[0097] The server receives user feedback and retrains the machine learning model based on it. This retraining improves classification accuracy from the next time onwards. By appropriately incorporating feedback, the accuracy of the system continues to improve.

[0098] Example 1

[0099] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0100] In today's world, it is important for users to appropriately manage conversations occurring in diverse environments and quickly and accurately obtain information appropriate to each context. However, conventional systems lack the accuracy of noise reduction and text conversion of speech data, as well as the ability to analyze conversation content and classify it into appropriate scopes, making it difficult to provide information tailored to user needs. As a result, users are unable to use information efficiently, resulting in issues such as reduced work efficiency and quality of life.

[0101] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0102] In this invention, the server includes an input device for acquiring user voice or text data, a communication device for transmitting the acquired data to the server device, a preprocessing device for preprocessing the acquired data, performing noise reduction and text conversion, a device for analyzing the converted text data and performing keyword extraction and context analysis, a device for classifying the conversation content into specific scopes based on the analysis results, a device for saving the classified data based on the scopes and sending notifications to appropriate display devices, a device for receiving user feedback and training and improving the accuracy of the classification device, a device for analyzing prompts entered by the user using a generative AI model and providing optimal responses, and a device for selecting appropriate notification formats according to different scopes. This allows the system to accurately classify diverse user conversations in real time and provide information tailored to each context. As a result, users can efficiently manage and use information, improving their work efficiency and quality of life.

[0103] The "input device means" is a hardware or software system for acquiring voice and text data from a user.

[0104] "Communication means" refers to the networking protocols and techniques used to transmit data acquired by the input device means to the server device.

[0105] "Preprocessing" refers to the process of removing noise from the acquired data and converting the audio data into text data.

[0106] "Noise reduction" is a technology that removes unnecessary audio components from audio data to generate clear audio data.

[0107] "Text conversion" is the process of converting speech data into text data using natural language processing techniques.

[0108] "Keyword extraction" is a natural language processing technique that extracts important words and phrases from text data.

[0109] "Contextual analysis" is a technology that understands the context of text data and analyzes its meaning and significance.

[0110] "Scope" refers to a category or area for classifying conversation content.

[0111] A "display device" is a hardware or software system for presenting notified information to a user.

[0112] "Feedback" refers to comments and corrections provided by users regarding the system's output.

[0113] "Learning and refinement" is the process of retraining the system's classifiers and improving their performance based on user feedback.

[0114] A "generative AI model" is an artificial intelligence technology that analyzes user input data and generates optimal responses.

[0115] A "prompt sentence" is an input sentence that instructs the generative AI model on what processing to do.

[0116] "Notification format" refers to the method or means by which information is conveyed to the user.

[0117] This invention is a system that automatically identifies various conversation contents of users, classifies them into appropriate scopes, and manages and provides information corresponding to each of them. Specific embodiments for carrying out this invention will be described below.

[0118] System Configuration

[0119] 1. Capturing User Input

[0120] A user starts a conversation by turning into a device such as a smartphone or PC. This device is equipped with a microphone for voice input and a keyboard for text input.

[0121] The device captures the user's voice and text data in real time and immediately transmits it to a server. For example, the device captures voice data using a smartphone's microphone and transmits the data to the server via the Internet.

[0122] 2. Data Preprocessing

[0123] The server receives the data sent from the device. Because the audio data often contains noise, it first applies a noise reduction algorithm (e.g., Python's pydub library) to generate clear audio data.

[0124] The server then uses a speech recognition model (e.g., Google Cloud Speech-to-Text API) to convert the clear audio data into text data, which is used for further processing in the system.

[0125] 3. Analysis using natural language processing

[0126] The server analyzes the converted text data, using a Python natural language processing library (e.g., spaCy or NLTK) to extract keywords and analyze the context.

[0127] Based on this analysis, the server classifies the conversation into a specific scope (e.g., family, work, friends) using a pre-trained machine learning model.

[0128] 4. Information management by scope

[0129] The server stores the classified data in a database (e.g., MySQL (registered trademark) or MongoDB). The stored data for each scope is notified to an appropriate display device as needed.

[0130] For example, family-related conversations will be displayed on a smart display in the living room, while work-related conversations will be sent to your work computer or smartphone.

[0131] 5. Processing and Learning from User Feedback

[0132] Users can provide feedback on the system's output, for example, if a conversation is misclassified, the user can send corrective feedback.

[0133] The server receives user feedback and uses it to retrain the machine learning model, improving the system's classification accuracy.

[0134] Specific use cases

[0135] Scenario 1: Conversation with family

[0136] 1. A user says in the living room, "Let's go see a movie with the whole family today."

[0137] 2. The device's microphone captures this audio data and sends it to the server.

[0138] 3. The server removes noise and converts the audio to text using the Google Cloud Speech-to-Text API.

[0139] 4. The server analyzes the converted text, extracts the keywords "family" and "movie," and classifies them into the "family" scope based on the context.

[0140] 5. The server notifies the smart display in the living room of this information and displays the message, "Everyone is planning to watch a movie together."

[0141] Scenario 2: A conversation with your boss

[0142] 1. A user sits in the office and asks their boss, "How is this project going?"

[0143] 2. The device's headset microphone captures the audio data and sends it to the server.

[0144] 3. The server removes noise and converts the audio to text using the Google Cloud Speech-to-Text API.

[0145] 4. The server analyzes the converted text, extracts the keywords "project" and "progress," and classifies them into the "work" scope based on the context.

[0146] 5. The server sends notifications about the progress of the project to your work computer or smartphone.

[0147] Prompt Sentence Examples

[0148] "I have recorded a conversation with my family. Please analyze this conversation and categorize it into relevant scopes."

[0149] "I have recorded instructions from my supervisor. Please transcribe them into text, analyze them, and process them as work-related information."

[0150] This invention allows users to appropriately manage various conversation contents and smoothly use information. The system provides information classified into specific scopes in real time, improving the user's life and work efficiency.

[0151] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0152] Step 1:

[0153] The user starts a conversation with the device using voice or text. The input is the user's voice or text data, and the output is the captured raw voice or text data. Specifically, the user uses the microphone on their smartphone to say, "Let's go see a movie with my family today." This voice data is captured by the device's microphone.

[0154] Step 2:

[0155] The device sends the captured voice or text data to the server. The input is the raw voice or text data captured in step 1, and the output is the same data sent to the server. Specifically, the device sends this voice data to the server via Wi-Fi or mobile network.

[0156] Step 3:

[0157] The server receives the voice data sent from the terminal. The input is the raw voice data sent from the terminal, and the output is the same data received on the server. Specifically, the server receives the data using the TCP / IP protocol.

[0158] Step 4:

[0159] The server performs noise removal on the audio data received. The input is raw audio data, and the output is clear audio data with noise removed. Specifically, the Python pydub library is used to filter noise from the audio data.

[0160] Step 5:

[0161] The server converts the noise-removed audio data into text. The input is clear audio data after noise removal, and the output is text data. Specifically, the server converts the audio data into text using the Google Cloud Speech-to-Text API.

[0162] Step 6:

[0163] The server analyzes the text data and performs keyword extraction and context analysis. The input is the converted text data, and the output is the extracted keywords and analyzed context information. Specifically, it uses the Python spaCy library to extract keywords from the text and analyze the context.

[0164] Step 7:

[0165] The server classifies the conversation content into specific scopes based on the analysis results. The input is keywords and context information, and the output is data classified into scopes. For example, based on the extracted keywords "family" and "movies," the conversation content is classified into the "family" scope.

[0166] Step 8:

[0167] The server saves the data classified by scope in a database. The input is the data classified by scope, and the output is the data saved in the database. Specifically, the data is saved using MySQL or MongoDB.

[0168] Step 9:

[0169] The server sends notifications to the appropriate display devices based on the stored data. The input is the data classified into scopes, and the output is the notification sent to the display device. For example, data from the "Family" scope is notified to the smart display in the living room.

[0170] Step 10:

[0171] The user provides feedback on the system's output. The input is the user's feedback information, and the output is the feedback information sent to the server. For example, the user can provide feedback on their smartphone saying, "This classification is correct."

[0172] Step 11:

[0173] The server receives the feedback and uses it to retrain the machine learning model. The input is the feedback information from the user, and the output is an updated machine learning model. Specifically, retraining based on the feedback is performed using scikit-learn.

[0174] (Application example 1)

[0175] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0176] Autonomous vehicles are required to analyze passenger conversations in real time and provide appropriate information. However, there is no system that can effectively respond to passenger requests regarding destination changes and traffic conditions, which hinders the improvement of the driving experience.

[0177] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0178] In this invention, the server includes an input device means for acquiring user voice or text data, a communication means for transmitting the data acquired by the input device means to the server device, a means for preprocessing the acquired data in the server device and performing noise removal and text conversion, a means for analyzing the converted text data and performing keyword extraction and context analysis, a means for classifying the conversation content into specific scopes based on the analysis results, a means for saving the data classified based on the scopes and sending a notification to an appropriate output device, a means for receiving user feedback and training and improving the accuracy of the classification means, and a means for providing destination information and route change information based on the passenger conversation content. This makes it possible to analyze the passenger conversation content in real time and provide appropriate navigation information and recommended routes.

[0179] "User Voice or Text Data" means any voice or text information input by a User through conversation, which is captured and analyzed by the System.

[0180] "Input device means" refers to a device or function that acquires user voice or text data, including a microphone and keyboard.

[0181] "Communication means" refers to a function for transmitting data acquired by the input device means to a server device, including internet connection and wireless communication.

[0182] "Server device" refers to a central management system for preprocessing the acquired data and performing noise removal and text conversion.

[0183] "Noise removal and text conversion means" refers to a processing function for removing noise from audio data and converting audio to text.

[0184] "Analysis means" refers to the function for analyzing text data, extracting keywords, and analyzing context. Natural language processing technology is used for this.

[0185] "Means for classifying into scope" refers to the function of classifying conversation content into specific categories or ranges based on the analysis results.

[0186] "Output Device" means a device or function for displaying or communicating data classified based on scope, including a display or notification system.

[0187] "Means for receiving user feedback and for learning and improving accuracy" refers to a learning function for improving the accuracy of the classification means based on feedback from users.

[0188] "Means of providing destination information and route change information based on passenger conversation content" refers to the function of analyzing passenger conversation content in real time and providing appropriate navigation information and recommended routes.

[0189] A system for realizing this application example analyzes passenger conversations in real time and provides appropriate navigation information and recommended routes. An embodiment of this system will be described in detail below.

[0190] First, the server acquires data using an input device means for acquiring voice or text data from a user (passenger). This input device means may include a microphone on a smartphone or a microphone built into the vehicle. The acquired voice data is then transmitted to the server device via a communication means. The communication means may include an internet connection or wireless communication.

[0191] The server receives the transmitted voice data and first performs noise reduction processing to produce clear voice data. It then converts the voice data into text using speech recognition software (e.g., Google's speech recognition API). This text data is then analyzed using natural language processing techniques (e.g., the BERT model) to extract keywords and perform context analysis.

[0192] The server then classifies the conversation content into a specific scope (e.g., "change of destination" or "traffic information") based on the analysis results, and notifies the appropriate output device (vehicle infotainment system or smartphone display).

[0193] Additionally, the server receives user feedback to train and refine the classification method, and can retrain the machine learning model to improve future classification accuracy.

[0194] For example, if a passenger says, "I want to change my destination," the system works as follows:

[0195] 1. Voice input: "I want to change my destination."

[0196] 2. Text converted by speech recognition: "I would like to change my destination."

[0197] 3. Analysis results: destination change

[0198] 4. Notification: "Enter new destination."

[0199] Also, if a passenger says "Give me the latest traffic information," the following happens:

[0200] 1. Voice input: "Give me traffic updates"

[0201] 2. Text converted by speech recognition: "What is the latest traffic situation?"

[0202] 3. Analysis result: Traffic information request

[0203] 4. Notification: "Shows traffic updates."

[0204] The above steps make it possible to analyze passenger conversations in real time and provide appropriate navigation information and recommended routes.

[0205] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0206] Step 1:

[0207] A user starts a conversation in the car. Voice data is input. This voice data is captured using a microphone on the smartphone or a microphone built into the vehicle. Voice data is obtained as input.

[0208] Step 2:

[0209] The device transmits the captured audio data to a server in real time, either via an internet connection or wireless communication. The device receives audio data as input and transmits it to the server as output.

[0210] Step 3:

[0211] The server preprocesses the received audio data using a noise reduction algorithm, specifically applying a noise filter to generate clear audio data. The input is the captured audio data, and the output is the clear audio data with noise removed.

[0212] Step 4:

[0213] The server converts the noise-removed voice data into text data using speech recognition software (e.g., Google's speech recognition API). The input is clear voice data, and the output is the converted text data.

[0214] Step 5:

[0215] The server analyzes the converted text data and performs keyword extraction and context analysis. This process uses natural language processing techniques (e.g., the BERT model). The input is the text data, and the output is the analyzed keywords and context information.

[0216] Step 6:

[0217] The server classifies the conversation content into a specific scope based on the analysis results, and then categorizes it into categories that fit the passenger's request (e.g., "Destination Change" or "Traffic Information"). The input is the analyzed keywords and context information, and the output is the scoped information.

[0218] Step 7:

[0219] The server stores the data classified based on the scope and sends notifications to the appropriate output device (e.g., a vehicle display or a smartphone). The input is the information classified by the scope, and the output is a notification to the display or the like.

[0220] Step 8:

[0221] The user can provide feedback on the system's output. For example, if a classification is made into an incorrect scope, the user can provide feedback to correct it. The input is the user's feedback information, and the output is data to improve the accuracy of the classification algorithm.

[0222] Step 9:

[0223] The server receives user feedback and retrains the machine learning model, which can improve future classification accuracy. The input is the user feedback data, and the output is an improved classification algorithm.

[0224] The above steps make it possible to analyze passenger conversations in real time and provide appropriate navigation information and recommended routes.

[0225] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0226] This invention is a system that automatically identifies various conversational content of users and categorizes them into appropriate scopes to manage and provide information according to each. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, more advanced analysis and categorization can be achieved.

[0227] System Overview

[0228] 1. Capturing User Input

[0229] The user initiates a conversation, either by voice or text input, in a living room, office, or other environment.

[0230] The device uses a microphone and keyboard to capture the user's voice data and text data in real time and transmits it to the server.

[0231] 2. Preprocessing of audio data

[0232] The server receives the voice data sent from the device. Because the received data contains noise, it first applies a noise reduction algorithm to generate clear voice data.

[0233] 3. Speech Recognition and Text Conversion

[0234] The server converts the noise-removed speech data into text data using a speech recognition model, which transforms the speech data into readable text.

[0235] 4. Analysis using natural language processing

[0236] The server analyzes the text data, extracts keywords, and performs context analysis, thereby understanding the meaning and importance of the text data.

[0237] Based on these analysis results, the server classifies the conversation content into specific scopes (e.g., family, boss, friends, etc.).

[0238] 5. Emotion Recognition by Emotion Engine

[0239] The server inputs the text data into an emotion engine to recognize the user's emotions, which identify emotions such as joy, anger, sadness, and surprise.

[0240] The recognized emotional information influences the analysis results and scope classification, enabling more accurate classification.

[0241] 6. Information management and notification for each scope

[0242] The server stores the classified data in the appropriate database and sends notifications to the appropriate output devices based on the classified scope.

[0243] The content of the notification is adjusted based on the emotional information. For example, if the user is feeling stressed, the notification content may be simplified.

[0244] 7. Processing and Learning from User Feedback

[0245] The user provides feedback on the system's output, for example by sending feedback to the system if the classification is incorrect or if more information is needed.

[0246] The server receives user feedback and uses it to retrain the machine learning model, improving the overall classification accuracy, including the emotion engine.

[0247] Specific use cases

[0248] Scenario 1: Conversation with family

[0249] 1. A user is having a conversation with a family member in the living room, and the audio content is captured by the device.

[0250] 2. The device sends the captured audio data to the server.

[0251] 3. The server removes noise from the audio data and converts it into text.

[0252] 4. The server analyzes the text data and identifies the "family" and scope.

[0253] 5. The server uses an emotion engine to recognize the user's emotions. For example, if the user is talking happily, appropriate information will be output.

[0254] 6. The server displays information about the family on the living room display.

[0255] Scenario 2: A conversation with your boss

[0256] 1. A user is talking to their boss in the office, and the conversation is captured by a headset.

[0257] 2. The device sends the audio data to the server.

[0258] 3. The server removes noise from the audio data and converts it into text.

[0259] 4. The server analyzes the text data and identifies the "boss" and scope.

[0260] 5. The server uses an emotion engine to recognize the user's emotions. For example, if the user is feeling stressed, the notification content will be simplified.

[0261] 6. The server notifies the work-related information to the PC or smartphone at work.

[0262] This will enable advanced information management and notifications that take the user's emotions into consideration, improving the user experience.

[0263] The processing flow will be explained below.

[0264] Step 1:

[0265] The user initiates a conversation, inputting voice or text data in a specific environment, such as a living room or office.

[0266] Step 2:

[0267] The device uses a microphone and keyboard to capture the user's voice and text data in real time, for example, using the built-in microphone of a smartphone or PC.

[0268] Step 3:

[0269] The device transmits the captured voice and text data to a server via an internet connection.

[0270] Step 4:

[0271] The server preprocesses the audio data received from the device. First, it applies a noise reduction algorithm to generate clear audio data.

[0272] Step 5:

[0273] The server converts the noise-removed audio data into text data using a speech recognition model (e.g., Google Speech-to-Text API).

[0274] Step 6:

[0275] The server analyzes the text data using natural language processing (NLP) techniques, which include keyword extraction and context analysis.

[0276] Step 7:

[0277] The server classifies the analyzed text data into specific scopes (e.g., family, boss, friends, etc.) using topic modeling and machine learning algorithms.

[0278] Step 8:

[0279] The server inputs the text data into an emotion engine to recognize the user's emotions, which identify emotions such as joy, anger, sadness, surprise, etc.

[0280] Step 9:

[0281] The server reevaluates the scope classification based on the analysis results together with the emotional information, and updates the optimal scope classification, thereby achieving highly accurate classification that includes emotional information.

[0282] Step 10:

[0283] The server stores the data classified based on the scope in a database or storage device. For example, data about a family member is stored in a household database.

[0284] Step 11:

[0285] The server sends the stored data to the appropriate output device. The notification content is adjusted based on the scope classification and emotion information. For example, if the user is feeling stressed, the notification content is simplified.

[0286] Step 12:

[0287] The user provides feedback on the system's output, for example, by sending feedback on scope classification or emotion recognition.

[0288] Step 13:

[0289] The server receives user feedback and retrains the machine learning model, improving classification and emotion recognition accuracy.

[0290] Example 2

[0291] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0292] In modern society, users have diverse conversations, and there is a need for automatic identification and appropriate classification. However, conventional technologies have faced challenges, such as insufficient accuracy in classifying conversation content and difficulty in managing information taking user emotions into account. It is also not easy to improve the accuracy of the system by reflecting user feedback.

[0293] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0294] In this invention, the server includes an input device for acquiring user voice or text data, a communication device for transmitting the data acquired by the input device to the server, a device for preprocessing the acquired data in the server device, performing noise reduction and text conversion, a device for analyzing the converted text data and performing keyword extraction and context analysis, a device for classifying the conversation content into a specific scope based on the analysis results, a device for recognizing the user's emotions using an emotion engine, a device for saving the classified data based on the emotion information and scope and sending a notification to an appropriate output device, and a device for receiving user feedback and training and improving the accuracy of the classification device. This enables automatic identification and classification of diverse user conversation content with high accuracy, and further realizes information management and notification that takes user emotions into consideration. Furthermore, the accuracy of the system can be improved by reflecting user feedback.

[0295] "Input device means" refers to a device for obtaining voice or text data from a user.

[0296] The "communication means" refers to a means for transmitting data acquired by the input device means to the server device.

[0297] The "noise removal means" refers to a means for removing noise from the voice data acquired in the server device.

[0298] The "text conversion means" refers to a means that provides a function for converting noise-removed voice data into text data.

[0299] "Keyword extraction means" refers to means for extracting important keywords from text data.

[0300] The "context analysis means" refers to a means for analyzing the context of text data.

[0301] The "scope classification means" refers to a means for classifying the conversation content into a specific scope based on the analysis results.

[0302] "Emotion recognition means" refers to means for recognizing the user's emotions using an emotion engine.

[0303] "Output device" refers to a device for notifying appropriate information based on classification and emotion recognition results.

[0304] "Data storage means" refers to a means for storing classified conversation data.

[0305] "Feedback receiving means" refers to a means for receiving feedback provided by a user.

[0306] "Training refinement means" refers to means for training and refining a classifier based on received feedback.

[0307] This invention is a system that automatically identifies diverse conversational content of users and classifies it into appropriate scopes. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, more advanced analysis and classification can be achieved. Detailed embodiments of the program for this system are described below.

[0308] Hardware and software used

[0309] Input device: A device that allows a user to input voice or text data. A microphone is used for voice input, and a keyboard is used for text input.

[0310] Communication method: A network communication method for sending data captured by an input device to a server. This includes Wi-Fi and wired LAN.

[0311] Server: A central processing unit for data pre-processing, speech recognition, natural language processing analysis, and emotion recognition. Software used includes Google Speech-to-Text API, BERT model, and IBM Watson® Tone Analyzer.

[0312] Data processing and calculation

[0313] 1. Preprocessing of audio data

[0314] The server receives the audio data sent from the terminal and applies a noise reduction algorithm (e.g., Spectral Subtraction) to generate clear audio data.

[0315] 2. Speech Recognition and Text Conversion

[0316] The server uses the Google Speech-to-Text API to convert the noise-removed audio data into text data. Specifically, the audio data is converted into text that says, "I'd like to talk about today's meeting."

[0317] 3. Analysis using natural language processing

[0318] The server uses the BERT model to extract keywords from the generated text data and perform contextual analysis, thereby understanding the meaning and importance of the text data.

[0319] 4. Scope Classification

[0320] Based on the analysis results, the server classifies the text data into specific scopes such as "family," "bosses," and "friends."

[0321] 5. Emotion recognition

[0322] The server uses IBM Watson Tone Analyzer to recognize the user's emotions (happiness, anger, sadness, surprise, etc.), and based on this emotional information, the accuracy of scope classification is further improved.

[0323] 6. Information Management and Notification

[0324] The server stores the classified data in a database and sends appropriate notifications to output devices (e.g., smartphones, PCs, digital displays, etc.) The content of the notifications is adjusted taking into account emotional information.

[0325] 7. Feedback and Learning

[0326] The user provides feedback on the system's output, for example, telling the system, "This notification was inappropriate."

[0327] The server receives user feedback and updates the machine learning model, which improves the accuracy of classification and emotion recognition in future.

[0328] Specific use cases

[0329] Scenario 1: Conversation with family

[0330] 1. A user is sitting in the living room and says, "I want to watch a movie with my family today."

[0331] 2. The device captures this conversation and sends it to the server.

[0332] 3. The server removes noise from the audio data and converts it to text using the Google Speech-to-Text API.

[0333] 4. The server analyzes the text using the BERT model and identifies the scope "family".

[0334] 5. The server uses IBM Watson Tone Analyzer to recognize the user's emotion as "joy."

[0335] 6. The server displays movie information for the family on the living room display.

[0336] Scenario 2: A conversation with your boss

[0337] 1. A user is in the office and says, "I'd like to talk to my boss about the progress of the project."

[0338] 2. The device captures the conversation and sends it to the server.

[0339] 3. The server removes noise from the audio data and converts it to text using the Google Speech-to-Text API.

[0340] 4. The server analyzes the text using the BERT model and identifies the scope "boss."

[0341] 5. The server uses IBM Watson Tone Analyzer to recognize the user's emotion as "stress."

[0342] 6. The server sends simplified project progress notifications to your work PC or smartphone.

[0343] This series of processes enables the system to automatically identify diverse user conversation content and classify it into the appropriate scope. Furthermore, by recognizing the user's emotions, more accurate classification and notifications are achieved. As a result, the user experience is improved, and by appropriately incorporating feedback, the system continues to learn and improve its accuracy.

[0344] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0345] Step 1:

[0346] A user starts a conversation. A user starts a voice conversation in their living room or office. For example, they might say, "I want to watch a movie with my family today." The input is real-time voice data. Specifically, the user speaks into a microphone, and voice data is captured.

[0347] Step 2:

[0348] The device captures voice data in real time and sends it to the server. In this case, the microphone functions as an input device and sends the captured voice data to the server. The input is voice data, and the output is digital voice data sent to the server. Specifically, the device sends the data via Wi-Fi or wired LAN.

[0349] Step 3:

[0350] The server removes noise from the received audio data. It applies a noise removal algorithm (e.g., Spectral Subtraction) to generate clear audio data. The input is the audio data sent from the device, and the output is clear audio data with noise removed. Specifically, the server executes the noise removal module.

[0351] Step 4:

[0352] The server converts the noise-removed audio data into text data. The Google Speech-to-Text API is used to convert the audio data into text. The input is the noise-removed audio data, and the output is text data. Specifically, the server inputs the audio data into the Google Speech-to-Text API and stores the resulting text data.

[0353] Step 5:

[0354] The server analyzes the text data and performs keyword extraction and context analysis. It uses the BERT model to extract keywords from the text data and analyze the context. The input is the text data, and the output is the extracted keywords and context information. Specifically, the server runs the BERT model and stores the analysis results.

[0355] Step 6:

[0356] The server classifies the conversation content into a specific scope based on the analysis results. For example, it may classify the content into scopes such as "family," "boss," or "friends." The input is keywords and context information, and the output is scope information. Specifically, the server applies a classification algorithm to identify the scope.

[0357] Step 7:

[0358] The server uses an emotion engine to recognize the user's emotions. The IBM Watson Tone Analyzer is used to recognize the user's emotions from text data. The input is text data and the output is emotional information. Specifically, the server inputs text data into the IBM Watson Tone Analyzer and stores the obtained emotional information.

[0359] Step 8:

[0360] The server stores the data classified based on emotion information and scope and sends a notification to the appropriate output device. The input is emotion information and scope information, and the output is data stored in a database and a notification sent. Specific operations include the server storing the data in a database and sending a notification to an output device (e.g., smartphone, PC, digital display).

[0361] Step 9:

[0362] The user provides feedback on the system's output. For example, the user tells the system, "This notification was inappropriate." The input is the user's feedback, and the output is the feedback data sent to the server. The specific action is for the user to provide feedback in text or voice.

[0363] Step 10:

[0364] The server receives feedback from users and retrains the machine learning model based on that feedback. This improves the overall classification accuracy, including the emotion engine. The input is the feedback data, and the output is the updated model. Specifically, the server runs the retraining algorithm to update the model.

[0365] (Application example 2)

[0366] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0367] Conventional food delivery services do not provide personalized suggestions based on the user's conversation content or emotions, limiting the user experience. Furthermore, there is a need for a system that can provide optimal suggestions tailored to specific situations. This creates challenges such as a lack of improvement in user satisfaction and personalization of the service.

[0368] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0369] In this invention, the server includes an input device means for acquiring user voice or text data, a means for preprocessing the data acquired by the input device means to perform noise removal and text conversion, and a means for analyzing the converted text data to perform keyword extraction and context analysis, thereby enabling optimal food delivery service suggestions to be made in accordance with special circumstances and the user's emotions.

[0370] The "input device means" is a device for acquiring voice or text data from a user.

[0371] The "communication means" is a system for transmitting data acquired by the input device means to the server device.

[0372] The "preprocessing means" is a process for removing noise and converting the acquired data into text in the server device.

[0373] The "analysis means" is a function that analyzes the data converted into text, extracts keywords, and performs context analysis.

[0374] The "scope classification means" is a process for classifying the conversation content into a specific scope based on the analysis results.

[0375] The "food delivery proposal method" is a process for proposing appropriate food delivery services based on the classified scope.

[0376] The "emotion recognition means" is a process that recognizes the user's emotions and selects the optimal output according to the scope and emotion.

[0377] The "notification means" is a process for notifying data classified according to scope and emotion to an appropriate output device.

[0378] A "feedback mechanism" is a process that receives user feedback to train and refine the classifier.

[0379] The present invention is a system that analyzes the various conversational content and emotions of users and optimizes food delivery services accordingly. Specific embodiments for carrying out the invention are described below.

[0380] Hardware and Software Configuration

[0381] 1. Input Device Means

[0382] The user inputs voice using the smartphone's microphone, and the input voice data is sent to the server via the smartphone's built-in communication means.

[0383] 2. Means of communication

[0384] The smartphone uses Wi-Fi or mobile data to transmit the captured data to a server in real time.

[0385] 3. Pretreatment Methods

[0386] The server first denoises the captured audio data to produce clear audio, using the Python speech_recognition library and a speech recognition model from the Transformers package.

[0387] 4. Analysis method

[0388] The server converts the audio data into text data and analyzes the text data, including using natural language processing techniques to extract keywords and perform context analysis, such as using TfidfVectorizer and KMeans clustering to classify the scope of the conversation.

[0389] 5. Scope classification and proposal methods

[0390] The server classifies the user's conversation content into a specific scope (e.g., family, friends, boss) and suggests appropriate food delivery services based on the scope. This also takes into account the user's emotional information. For emotion recognition, it uses the Transformers emotion recognition model.

[0391] 6. Emotion recognition means

[0392] The server analyzes the text data to recognize the user's emotions, categorizing them as joy, surprise, anger, etc., and then suggests the most appropriate service accordingly.

[0393] 7. Means of notification

[0394] The server stores the classified data according to scope and emotion and sends notifications to the appropriate output device, which can be displayed on the smartphone display or in the notification bar.

[0395] 8. Feedback channels

[0396] The user provides feedback on the system's output, such as whether the suggestions were appropriate. The server receives this feedback and uses it to improve the accuracy of the classification and emotion recognition methods mentioned above.

[0397] Specific use cases

[0398] Example 1: When a user is having a pleasant conversation with their family

[0399] Prompt: "What are your menu recommendations?"

[0400] Output: "Here are our recommended menu items!"

[0401] Example 2: When the user is stressed

[0402] Prompt: "I'm tired."

[0403] Output: "How about some cold udon noodles today?"

[0404] This makes it possible to provide an optimal food delivery experience based on the user's conversation content and emotions. This system not only improves the user experience, but also significantly improves the accuracy and convenience of food delivery services.

[0405] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0406] Step 1:

[0407] The user inputs voice using the smartphone's microphone, and the voice data is stored in the smartphone's internal memory.

[0408] Step 2:

[0409] The smartphone uses its built-in communication means to transmit the audio data to the server in real time, using Wi-Fi or mobile data.

[0410] Step 3:

[0411] The server pre-processes the received audio data using a noise reduction algorithm, resulting in clear audio data. The denoised audio data is then input to the next step.

[0412] Step 4:

[0413] The server converts the preprocessed audio data into text using a speech recognition model, using the Python speech_recognition library and the Transformers package, and the converted text data becomes the input for the next step.

[0414] Step 5:

[0415] The server analyzes the text data using natural language processing techniques. Specifically, it uses TfidfVectorizer and KMeans clustering to extract keywords and perform context analysis, and classifies the conversation content into specific scopes. The analyzed scope information becomes the input data for the next step.

[0416] Step 6:

[0417] The server generates optimal food delivery service recommendations based on the scope. It uses an emotion recognition model to evaluate the user's emotions, such as daigo / bert-base-japanese-sentiment, and generates recommendations that incorporate the user's emotional information.

[0418] Step 7:

[0419] The server stores the data classified according to scope and emotion and sends a notification to the appropriate output device (smartphone display or notification bar). For example, messages such as "Here are our recommended menu items!" or "How about some cold udon noodles today?" are displayed.

[0420] Step 8:

[0421] The user provides feedback on the system's suggestions and outputs, which are then sent from the smartphone to the server.

[0422] Step 9:

[0423] The server uses the feedback it receives to retrain its machine learning models (classification and emotion recognition) and improve the overall accuracy of the system, resulting in better suggestions for future use.

[0424] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0425] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0426] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0427] [Second embodiment]

[0428] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0429] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0430] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0431] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0432] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0433] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0434] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0435] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0436] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0437] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0438] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0439] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0440] This invention is a system that automatically identifies various conversational contents of users and classifies them into appropriate scopes to manage and provide information appropriate to each of them. A specific embodiment of this system is shown below.

[0441] System Overview

[0442] 1. Capturing User Input

[0443] The user starts talking to the terminal, which has the function of capturing the user's voice and text data.

[0444] The device uses a microphone and keyboard to capture the user's voice and text data in real time, and the captured data is immediately sent to the server.

[0445] 2. Preprocessing of audio data

[0446] The server receives the data sent from the device. Since the received voice data may contain noise, it first applies a noise reduction algorithm to generate clear voice data.

[0447] The clear audio data is then converted into text data using a speech recognition model (e.g., a speech-to-text API).

[0448] 3. Analysis using natural language processing

[0449] The server analyzes the text data, extracts keywords, and performs context analysis. This analysis uses natural language processing technology. The analysis identifies the content of the conversation.

[0450] Based on the analysis results, the server classifies the conversation content into a specific scope (e.g., family, boss, friends, etc.) using a pre-defined algorithm.

[0451] 4. Information management by scope

[0452] The server stores data classified by scope in a database or storage device, and sends different information to the appropriate output device (such as a display or notification system) based on the scope.

[0453] For example, family-related conversations will be displayed on a living room display, while work-related conversations will be sent to a PC or smartphone at work.

[0454] 5. Processing and Learning from User Feedback

[0455] Users can provide feedback on the system's output, for example, if an item is classified into an incorrect scope, the user can provide feedback to correct it.

[0456] The server receives user feedback and uses it to retrain the machine learning model, improving the system's classification accuracy.

[0457] Specific use cases

[0458] Scenario 1: Conversation with family

[0459] 1. A user is having a conversation with a family member in the living room, and the conversation is captured on the device.

[0460] 2. The device sends the audio data to the server.

[0461] 3. The server removes noise from the audio data and converts it into text.

[0462] 4. The server analyzes the text data and identifies the "family" and scope.

[0463] 5. The server displays information about the family on the living room display.

[0464] Scenario 2: A conversation with your boss

[0465] 1. A user is talking to their boss in the office, and the conversation is captured by a headset.

[0466] 2. The device sends the audio data to the server.

[0467] 3. The server removes noise from the audio data and converts it into text.

[0468] 4. The server analyzes the text data and identifies the "boss" and scope.

[0469] 5. The server notifies the work-related information to the PC or smartphone at work.

[0470] Through these processes, the content of user conversations is automatically analyzed and appropriately categorized, allowing users to seamlessly manage different types of conversations and utilize information efficiently.

[0471] The processing flow will be explained below.

[0472] Step 1:

[0473] The user begins speaking. Audio capture begins in the living room, office, or other environment.

[0474] Step 2:

[0475] The device uses a built-in microphone or an external microphone to capture the user's voice data, which is collected in real time and sent to a server (possibly temporarily stored locally).

[0476] Step 3:

[0477] The device uses a communication means to send the captured audio data to a server, which can be via an internet connection or a local network.

[0478] Step 4:

[0479] The server receives the voice data from the device. Since the received data is likely to contain noise as it is, a noise reduction algorithm is first applied to it.

[0480] Step 5:

[0481] The server applies noise reduction algorithms to convert the audio data into a clearer version. This step uses digital signal processing techniques to remove background noise and unwanted sounds.

[0482] Step 6:

[0483] The server uses a speech recognition model (e.g., Google Speech-to-Text API) to convert the noise-removed audio data into text data, which is then converted into readable text data.

[0484] Step 7:

[0485] The server analyzes the converted text data. In this analysis step, natural language processing (NLP) techniques are used to extract keywords and perform context analysis. The meaning and importance of each part of the text data are then extracted.

[0486] Step 8:

[0487] The server classifies the text data into specific scopes based on the extracted keywords and context information, such as "family," "boss," and "friends."

[0488] Step 9:

[0489] The server stores the classified data. Based on the scope of classification, it is stored in the appropriate database or storage device. For example, data about family members is stored in the user's household database.

[0490] Step 10:

[0491] The server then sends the stored data to the appropriate output device: depending on the scope, family information might be sent to a display in the living room, and work-related information might be sent to a PC or smartphone at work.

[0492] Step 11:

[0493] The user provides feedback on the system's output, for example if a classification is incorrect or if additional information is needed, and the user provides that information to the system.

[0494] Step 12:

[0495] The server receives user feedback and retrains the machine learning model based on it. This retraining improves classification accuracy from the next time onwards. By appropriately incorporating feedback, the accuracy of the system continues to improve.

[0496] Example 1

[0497] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0498] In today's world, it is important for users to appropriately manage conversations occurring in diverse environments and quickly and accurately obtain information appropriate to each context. However, conventional systems lack the accuracy of noise reduction and text conversion of speech data, as well as the ability to analyze conversation content and classify it into appropriate scopes, making it difficult to provide information tailored to user needs. As a result, users are unable to use information efficiently, resulting in issues such as reduced work efficiency and quality of life.

[0499] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0500] In this invention, the server includes an input device for acquiring user voice or text data, a communication device for transmitting the acquired data to the server device, a preprocessing device for preprocessing the acquired data, performing noise reduction and text conversion, a device for analyzing the converted text data and performing keyword extraction and context analysis, a device for classifying the conversation content into specific scopes based on the analysis results, a device for saving the classified data based on the scopes and sending notifications to appropriate display devices, a device for receiving user feedback and training and improving the accuracy of the classification device, a device for analyzing prompts entered by the user using a generative AI model and providing optimal responses, and a device for selecting appropriate notification formats according to different scopes. This allows the system to accurately classify diverse user conversations in real time and provide information tailored to each context. As a result, users can efficiently manage and use information, improving their work efficiency and quality of life.

[0501] The "input device means" is a hardware or software system for acquiring voice and text data from a user.

[0502] "Communication means" refers to the networking protocols and techniques used to transmit data acquired by the input device means to the server device.

[0503] "Preprocessing" refers to the process of removing noise from the acquired data and converting the audio data into text data.

[0504] "Noise reduction" is a technology that removes unnecessary audio components from audio data to generate clear audio data.

[0505] "Text conversion" is the process of converting speech data into text data using natural language processing techniques.

[0506] "Keyword extraction" is a natural language processing technique that extracts important words and phrases from text data.

[0507] "Contextual analysis" is a technology that understands the context of text data and analyzes its meaning and significance.

[0508] "Scope" refers to a category or area for classifying conversation content.

[0509] A "display device" is a hardware or software system for presenting notified information to a user.

[0510] "Feedback" refers to comments and corrections provided by users regarding the system's output.

[0511] "Learning and refinement" is the process of retraining the system's classifiers and improving their performance based on user feedback.

[0512] A "generative AI model" is an artificial intelligence technology that analyzes user input data and generates optimal responses.

[0513] A "prompt sentence" is an input sentence that instructs the generative AI model on what processing to do.

[0514] "Notification format" refers to the method or means by which information is conveyed to the user.

[0515] This invention is a system that automatically identifies various conversation contents of users, classifies them into appropriate scopes, and manages and provides information corresponding to each of them. Specific embodiments for carrying out this invention will be described below.

[0516] System Configuration

[0517] 1. Capturing User Input

[0518] A user starts a conversation by turning into a device such as a smartphone or PC. This device is equipped with a microphone for voice input and a keyboard for text input.

[0519] The device captures the user's voice and text data in real time and immediately transmits it to a server. For example, the device captures voice data using a smartphone's microphone and transmits the data to the server via the Internet.

[0520] 2. Data Preprocessing

[0521] The server receives the data sent from the device. Because the audio data often contains noise, it first applies a noise reduction algorithm (e.g., Python's pydub library) to generate clear audio data.

[0522] The server then uses a speech recognition model (e.g., Google Cloud Speech-to-Text API) to convert the clear audio data into text data, which is used for further processing in the system.

[0523] 3. Analysis using natural language processing

[0524] The server analyzes the converted text data, using a Python natural language processing library (e.g., spaCy or NLTK) to extract keywords and analyze the context.

[0525] Based on this analysis, the server classifies the conversation into a specific scope (e.g., family, work, friends) using a pre-trained machine learning model.

[0526] 4. Information management by scope

[0527] The server stores the classified data in a database (e.g., MySQL or MongoDB). The stored data for each scope is notified to the appropriate display device as needed.

[0528] For example, family-related conversations will be displayed on a smart display in the living room, while work-related conversations will be sent to your work computer or smartphone.

[0529] 5. Processing and Learning from User Feedback

[0530] Users can provide feedback on the system's output, for example, if a conversation is misclassified, the user can send corrective feedback.

[0531] The server receives user feedback and uses it to retrain the machine learning model, improving the system's classification accuracy.

[0532] Specific use cases

[0533] Scenario 1: Conversation with family

[0534] 1. A user says in the living room, "Let's go see a movie with the whole family today."

[0535] 2. The device's microphone captures this audio data and sends it to the server.

[0536] 3. The server removes noise and converts the audio to text using the Google Cloud Speech-to-Text API.

[0537] 4. The server analyzes the converted text, extracts the keywords "family" and "movie," and classifies them into the "family" scope based on the context.

[0538] 5. The server notifies the smart display in the living room of this information and displays the message, "Everyone is planning to watch a movie together."

[0539] Scenario 2: A conversation with your boss

[0540] 1. A user sits in the office and asks their boss, "How is this project going?"

[0541] 2. The device's headset microphone captures the audio data and sends it to the server.

[0542] 3. The server removes noise and converts the audio to text using the Google Cloud Speech-to-Text API.

[0543] 4. The server analyzes the converted text, extracts the keywords "project" and "progress," and classifies them into the "work" scope based on the context.

[0544] 5. The server sends notifications about the progress of the project to your work computer or smartphone.

[0545] Prompt Sentence Examples

[0546] "I have recorded a conversation with my family. Please analyze this conversation and categorize it into relevant scopes."

[0547] "I have recorded instructions from my supervisor. Please transcribe them into text, analyze them, and process them as work-related information."

[0548] This invention allows users to appropriately manage various conversation contents and smoothly use information. The system provides information classified into specific scopes in real time, improving the user's life and work efficiency.

[0549] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0550] Step 1:

[0551] The user starts a conversation with the device using voice or text. The input is the user's voice or text data, and the output is the captured raw voice or text data. Specifically, the user uses the microphone on their smartphone to say, "Let's go see a movie with my family today." This voice data is captured by the device's microphone.

[0552] Step 2:

[0553] The device sends the captured voice or text data to the server. The input is the raw voice or text data captured in step 1, and the output is the same data sent to the server. Specifically, the device sends this voice data to the server via Wi-Fi or mobile network.

[0554] Step 3:

[0555] The server receives the voice data sent from the terminal. The input is the raw voice data sent from the terminal, and the output is the same data received on the server. Specifically, the server receives the data using the TCP / IP protocol.

[0556] Step 4:

[0557] The server performs noise removal on the audio data received. The input is raw audio data, and the output is clear audio data with noise removed. Specifically, the Python pydub library is used to filter noise from the audio data.

[0558] Step 5:

[0559] The server converts the noise-removed audio data into text. The input is clear audio data after noise removal, and the output is text data. Specifically, the server converts the audio data into text using the Google Cloud Speech-to-Text API.

[0560] Step 6:

[0561] The server analyzes the text data and performs keyword extraction and context analysis. The input is the converted text data, and the output is the extracted keywords and analyzed context information. Specifically, it uses the Python spaCy library to extract keywords from the text and analyze the context.

[0562] Step 7:

[0563] The server classifies the conversation content into specific scopes based on the analysis results. The input is keywords and context information, and the output is data classified into scopes. For example, based on the extracted keywords "family" and "movies," the conversation content is classified into the "family" scope.

[0564] Step 8:

[0565] The server saves the data classified by scope in a database. The input is the data classified by scope, and the output is the data saved in the database. Specifically, the data is saved using MySQL or MongoDB.

[0566] Step 9:

[0567] The server sends notifications to the appropriate display devices based on the stored data. The input is the data classified into scopes, and the output is the notification sent to the display device. For example, data from the "Family" scope is notified to the smart display in the living room.

[0568] Step 10:

[0569] The user provides feedback on the system's output. The input is the user's feedback information, and the output is the feedback information sent to the server. For example, the user can provide feedback on their smartphone saying, "This classification is correct."

[0570] Step 11:

[0571] The server receives the feedback and uses it to retrain the machine learning model. The input is the feedback information from the user, and the output is an updated machine learning model. Specifically, retraining based on the feedback is performed using scikit-learn.

[0572] (Application example 1)

[0573] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0574] Autonomous vehicles are required to analyze passenger conversations in real time and provide appropriate information. However, there is no system that can effectively respond to passenger requests regarding destination changes and traffic conditions, which hinders the improvement of the driving experience.

[0575] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0576] In this invention, the server includes an input device means for acquiring user voice or text data, a communication means for transmitting the data acquired by the input device means to the server device, a means for preprocessing the acquired data in the server device and performing noise removal and text conversion, a means for analyzing the converted text data and performing keyword extraction and context analysis, a means for classifying the conversation content into specific scopes based on the analysis results, a means for saving the data classified based on the scopes and sending a notification to an appropriate output device, a means for receiving user feedback and training and improving the accuracy of the classification means, and a means for providing destination information and route change information based on the passenger conversation content. This makes it possible to analyze the passenger conversation content in real time and provide appropriate navigation information and recommended routes.

[0577] "User Voice or Text Data" means any voice or text information input by a User through conversation, which is captured and analyzed by the System.

[0578] "Input device means" refers to a device or function that acquires user voice or text data, including a microphone and keyboard.

[0579] "Communication means" refers to a function for transmitting data acquired by the input device means to a server device, including internet connection and wireless communication.

[0580] "Server device" refers to a central management system for preprocessing the acquired data and performing noise removal and text conversion.

[0581] "Noise removal and text conversion means" refers to a processing function for removing noise from audio data and converting audio to text.

[0582] "Analysis means" refers to the function for analyzing text data, extracting keywords, and analyzing context. Natural language processing technology is used for this.

[0583] "Means for classifying into scope" refers to the function of classifying conversation content into specific categories or ranges based on the analysis results.

[0584] "Output Device" means a device or function for displaying or communicating data classified based on scope, including a display or notification system.

[0585] "Means for receiving user feedback and for learning and improving accuracy" refers to a learning function for improving the accuracy of the classification means based on feedback from users.

[0586] "Means of providing destination information and route change information based on passenger conversation content" refers to the function of analyzing passenger conversation content in real time and providing appropriate navigation information and recommended routes.

[0587] A system for realizing this application example analyzes passenger conversations in real time and provides appropriate navigation information and recommended routes. An embodiment of this system will be described in detail below.

[0588] First, the server acquires data using an input device means for acquiring voice or text data from a user (passenger). This input device means may include a microphone on a smartphone or a microphone built into the vehicle. The acquired voice data is then transmitted to the server device via a communication means. The communication means may include an internet connection or wireless communication.

[0589] The server receives the transmitted voice data and first performs noise reduction processing to produce clear voice data. It then converts the voice data into text using speech recognition software (e.g., Google's speech recognition API). This text data is then analyzed using natural language processing techniques (e.g., the BERT model) to extract keywords and perform context analysis.

[0590] The server then classifies the conversation content into a specific scope (e.g., "change of destination" or "traffic information") based on the analysis results, and notifies the appropriate output device (vehicle infotainment system or smartphone display).

[0591] Additionally, the server receives user feedback to train and refine the classification method, and can retrain the machine learning model to improve future classification accuracy.

[0592] For example, if a passenger says, "I want to change my destination," the system works as follows:

[0593] 1. Voice input: "I want to change my destination."

[0594] 2. Text converted by speech recognition: "I would like to change my destination."

[0595] 3. Analysis results: destination change

[0596] 4. Notification: "Enter new destination."

[0597] Also, if a passenger says "Give me the latest traffic information," the following happens:

[0598] 1. Voice input: "Give me traffic updates"

[0599] 2. Text converted by speech recognition: "What is the latest traffic situation?"

[0600] 3. Analysis result: Traffic information request

[0601] 4. Notification: "Shows traffic updates."

[0602] The above steps make it possible to analyze passenger conversations in real time and provide appropriate navigation information and recommended routes.

[0603] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0604] Step 1:

[0605] A user starts a conversation in the car. Voice data is input. This voice data is captured using a microphone on the smartphone or a microphone built into the vehicle. Voice data is obtained as input.

[0606] Step 2:

[0607] The device transmits the captured audio data to a server in real time, either via an internet connection or wireless communication. The device receives audio data as input and transmits it to the server as output.

[0608] Step 3:

[0609] The server preprocesses the received audio data using a noise reduction algorithm, specifically applying a noise filter to generate clear audio data. The input is the captured audio data, and the output is the clear audio data with noise removed.

[0610] Step 4:

[0611] The server converts the noise-removed voice data into text data using speech recognition software (e.g., Google's speech recognition API). The input is clear voice data, and the output is the converted text data.

[0612] Step 5:

[0613] The server analyzes the converted text data and performs keyword extraction and context analysis. This process uses natural language processing techniques (e.g., the BERT model). The input is the text data, and the output is the analyzed keywords and context information.

[0614] Step 6:

[0615] The server classifies the conversation content into a specific scope based on the analysis results, and then categorizes it into categories that fit the passenger's request (e.g., "Destination Change" or "Traffic Information"). The input is the analyzed keywords and context information, and the output is the scoped information.

[0616] Step 7:

[0617] The server stores the data classified based on the scope and sends notifications to the appropriate output device (e.g., a vehicle display or a smartphone). The input is the information classified by the scope, and the output is a notification to the display or the like.

[0618] Step 8:

[0619] The user can provide feedback on the system's output. For example, if a classification is made into an incorrect scope, the user can provide feedback to correct it. The input is the user's feedback information, and the output is data to improve the accuracy of the classification algorithm.

[0620] Step 9:

[0621] The server receives user feedback and retrains the machine learning model, which can improve future classification accuracy. The input is the user feedback data, and the output is an improved classification algorithm.

[0622] The above steps make it possible to analyze passenger conversations in real time and provide appropriate navigation information and recommended routes.

[0623] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0624] This invention is a system that automatically identifies various conversational content of users and categorizes them into appropriate scopes to manage and provide information according to each. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, more advanced analysis and categorization can be achieved.

[0625] System Overview

[0626] 1. Capturing User Input

[0627] The user initiates a conversation, either by voice or text input, in a living room, office, or other environment.

[0628] The device uses a microphone and keyboard to capture the user's voice data and text data in real time and transmits it to the server.

[0629] 2. Preprocessing of audio data

[0630] The server receives the voice data sent from the device. Because the received data contains noise, it first applies a noise reduction algorithm to generate clear voice data.

[0631] 3. Speech Recognition and Text Conversion

[0632] The server converts the noise-removed speech data into text data using a speech recognition model, which transforms the speech data into readable text.

[0633] 4. Analysis using natural language processing

[0634] The server analyzes the text data, extracts keywords, and performs context analysis, thereby understanding the meaning and importance of the text data.

[0635] Based on these analysis results, the server classifies the conversation content into specific scopes (e.g., family, boss, friends, etc.).

[0636] 5. Emotion Recognition by Emotion Engine

[0637] The server inputs the text data into an emotion engine to recognize the user's emotions, which identify emotions such as joy, anger, sadness, and surprise.

[0638] The recognized emotional information influences the analysis results and scope classification, enabling more accurate classification.

[0639] 6. Information management and notification for each scope

[0640] The server stores the classified data in the appropriate database and sends notifications to the appropriate output devices based on the classified scope.

[0641] The content of the notification is adjusted based on the emotional information. For example, if the user is feeling stressed, the notification content may be simplified.

[0642] 7. Processing and Learning from User Feedback

[0643] The user provides feedback on the system's output, for example by sending feedback to the system if the classification is incorrect or if more information is needed.

[0644] The server receives user feedback and uses it to retrain the machine learning model, improving the overall classification accuracy, including the emotion engine.

[0645] Specific use cases

[0646] Scenario 1: Conversation with family

[0647] 1. A user is having a conversation with a family member in the living room, and the audio content is captured by the device.

[0648] 2. The device sends the captured audio data to the server.

[0649] 3. The server removes noise from the audio data and converts it into text.

[0650] 4. The server analyzes the text data and identifies the "family" and scope.

[0651] 5. The server uses an emotion engine to recognize the user's emotions. For example, if the user is talking happily, appropriate information will be output.

[0652] 6. The server displays information about the family on the living room display.

[0653] Scenario 2: A conversation with your boss

[0654] 1. A user is talking to their boss in the office, and the conversation is captured by a headset.

[0655] 2. The device sends the audio data to the server.

[0656] 3. The server removes noise from the audio data and converts it into text.

[0657] 4. The server analyzes the text data and identifies the "boss" and scope.

[0658] 5. The server uses an emotion engine to recognize the user's emotions. For example, if the user is feeling stressed, the notification content will be simplified.

[0659] 6. The server notifies the work-related information to the PC or smartphone at work.

[0660] This will enable advanced information management and notifications that take the user's emotions into consideration, improving the user experience.

[0661] The processing flow will be explained below.

[0662] Step 1:

[0663] The user initiates a conversation, inputting voice or text data in a specific environment, such as a living room or office.

[0664] Step 2:

[0665] The device uses a microphone and keyboard to capture the user's voice and text data in real time, for example, using the built-in microphone of a smartphone or PC.

[0666] Step 3:

[0667] The device transmits the captured voice and text data to a server via an internet connection.

[0668] Step 4:

[0669] The server preprocesses the audio data received from the device. First, it applies a noise reduction algorithm to generate clear audio data.

[0670] Step 5:

[0671] The server converts the noise-removed audio data into text data using a speech recognition model (e.g., Google Speech-to-Text API).

[0672] Step 6:

[0673] The server analyzes the text data using natural language processing (NLP) techniques, which include keyword extraction and context analysis.

[0674] Step 7:

[0675] The server classifies the analyzed text data into specific scopes (e.g., family, boss, friends, etc.) using topic modeling and machine learning algorithms.

[0676] Step 8:

[0677] The server inputs the text data into an emotion engine to recognize the user's emotions, which identify emotions such as joy, anger, sadness, surprise, etc.

[0678] Step 9:

[0679] The server reevaluates the scope classification based on the analysis results together with the emotional information, and updates the optimal scope classification, thereby achieving highly accurate classification that includes emotional information.

[0680] Step 10:

[0681] The server stores the data classified based on the scope in a database or storage device. For example, data about a family member is stored in a household database.

[0682] Step 11:

[0683] The server sends the stored data to the appropriate output device. The notification content is adjusted based on the scope classification and emotion information. For example, if the user is feeling stressed, the notification content is simplified.

[0684] Step 12:

[0685] The user provides feedback on the system's output, for example, by sending feedback on scope classification or emotion recognition.

[0686] Step 13:

[0687] The server receives user feedback and retrains the machine learning model, improving classification and emotion recognition accuracy.

[0688] Example 2

[0689] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0690] In modern society, users have diverse conversations, and there is a need for automatic identification and appropriate classification. However, conventional technologies have faced challenges, such as insufficient accuracy in classifying conversation content and difficulty in managing information taking user emotions into account. It is also not easy to improve the accuracy of the system by reflecting user feedback.

[0691] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0692] In this invention, the server includes an input device for acquiring user voice or text data, a communication device for transmitting the data acquired by the input device to the server, a device for preprocessing the acquired data in the server device, performing noise reduction and text conversion, a device for analyzing the converted text data and performing keyword extraction and context analysis, a device for classifying the conversation content into a specific scope based on the analysis results, a device for recognizing the user's emotions using an emotion engine, a device for saving the classified data based on the emotion information and scope and sending a notification to an appropriate output device, and a device for receiving user feedback and training and improving the accuracy of the classification device. This enables automatic identification and classification of diverse user conversation content with high accuracy, and further realizes information management and notification that takes user emotions into consideration. Furthermore, the accuracy of the system can be improved by reflecting user feedback.

[0693] "Input device means" refers to a device for obtaining voice or text data from a user.

[0694] The "communication means" refers to a means for transmitting data acquired by the input device means to the server device.

[0695] The "noise removal means" refers to a means for removing noise from the voice data acquired in the server device.

[0696] The "text conversion means" refers to a means that provides a function for converting noise-removed voice data into text data.

[0697] "Keyword extraction means" refers to means for extracting important keywords from text data.

[0698] The "context analysis means" refers to a means for analyzing the context of text data.

[0699] The "scope classification means" refers to a means for classifying the conversation content into a specific scope based on the analysis results.

[0700] "Emotion recognition means" refers to means for recognizing the user's emotions using an emotion engine.

[0701] "Output device" refers to a device for notifying appropriate information based on classification and emotion recognition results.

[0702] "Data storage means" refers to a means for storing classified conversation data.

[0703] "Feedback receiving means" refers to a means for receiving feedback provided by a user.

[0704] "Training refinement means" refers to means for training and refining a classifier based on received feedback.

[0705] This invention is a system that automatically identifies diverse conversational content of users and classifies it into appropriate scopes. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, more advanced analysis and classification can be achieved. Detailed embodiments of the program for this system are described below.

[0706] Hardware and software used

[0707] Input device: A device that allows a user to input voice or text data. A microphone is used for voice input, and a keyboard is used for text input.

[0708] Communication method: A network communication method for sending data captured by an input device to a server. This includes Wi-Fi and wired LAN.

[0709] Server: A central processing unit for data pre-processing, speech recognition, natural language processing analysis, and emotion recognition. Software used includes the Google Speech-to-Text API, BERT model, and IBM Watson Tone Analyzer.

[0710] Data processing and calculation

[0711] 1. Preprocessing of audio data

[0712] The server receives the audio data sent from the terminal and applies a noise reduction algorithm (e.g., Spectral Subtraction) to generate clear audio data.

[0713] 2. Speech Recognition and Text Conversion

[0714] The server uses the Google Speech-to-Text API to convert the noise-removed audio data into text data. Specifically, the audio data is converted into text that says, "I'd like to talk about today's meeting."

[0715] 3. Analysis using natural language processing

[0716] The server uses the BERT model to extract keywords from the generated text data and perform contextual analysis, thereby understanding the meaning and importance of the text data.

[0717] 4. Scope Classification

[0718] Based on the analysis results, the server classifies the text data into specific scopes such as "family," "bosses," and "friends."

[0719] 5. Emotion recognition

[0720] The server uses IBM Watson Tone Analyzer to recognize the user's emotions (happiness, anger, sadness, surprise, etc.), and based on this emotional information, the accuracy of scope classification is further improved.

[0721] 6. Information Management and Notification

[0722] The server stores the classified data in a database and sends appropriate notifications to output devices (e.g., smartphones, PCs, digital displays, etc.) The content of the notifications is adjusted taking into account emotional information.

[0723] 7. Feedback and Learning

[0724] The user provides feedback on the system's output, for example, telling the system, "This notification was inappropriate."

[0725] The server receives user feedback and updates the machine learning model, which improves the accuracy of classification and emotion recognition in future.

[0726] Specific use cases

[0727] Scenario 1: Conversation with family

[0728] 1. A user is sitting in the living room and says, "I want to watch a movie with my family today."

[0729] 2. The device captures this conversation and sends it to the server.

[0730] 3. The server removes noise from the audio data and converts it to text using the Google Speech-to-Text API.

[0731] 4. The server analyzes the text using the BERT model and identifies the scope "family".

[0732] 5. The server uses IBM Watson Tone Analyzer to recognize the user's emotion as "joy."

[0733] 6. The server displays movie information for the family on the living room display.

[0734] Scenario 2: A conversation with your boss

[0735] 1. A user is in the office and says, "I'd like to talk to my boss about the progress of the project."

[0736] 2. The device captures the conversation and sends it to the server.

[0737] 3. The server removes noise from the audio data and converts it to text using the Google Speech-to-Text API.

[0738] 4. The server analyzes the text using the BERT model and identifies the scope "boss."

[0739] 5. The server uses IBM Watson Tone Analyzer to recognize the user's emotion as "stress."

[0740] 6. The server sends simplified project progress notifications to your work PC or smartphone.

[0741] This series of processes enables the system to automatically identify diverse user conversation content and classify it into the appropriate scope. Furthermore, by recognizing the user's emotions, more accurate classification and notifications are achieved. As a result, the user experience is improved, and by appropriately incorporating feedback, the system continues to learn and improve its accuracy.

[0742] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0743] Step 1:

[0744] A user starts a conversation. A user starts a voice conversation in their living room or office. For example, they might say, "I want to watch a movie with my family today." The input is real-time voice data. Specifically, the user speaks into a microphone, and voice data is captured.

[0745] Step 2:

[0746] The device captures voice data in real time and sends it to the server. In this case, the microphone functions as an input device and sends the captured voice data to the server. The input is voice data, and the output is digital voice data sent to the server. Specifically, the device sends the data via Wi-Fi or wired LAN.

[0747] Step 3:

[0748] The server removes noise from the received audio data. It applies a noise removal algorithm (e.g., Spectral Subtraction) to generate clear audio data. The input is the audio data sent from the device, and the output is clear audio data with noise removed. Specifically, the server executes the noise removal module.

[0749] Step 4:

[0750] The server converts the noise-removed audio data into text data. The Google Speech-to-Text API is used to convert the audio data into text. The input is the noise-removed audio data, and the output is text data. Specifically, the server inputs the audio data into the Google Speech-to-Text API and stores the resulting text data.

[0751] Step 5:

[0752] The server analyzes the text data and performs keyword extraction and context analysis. It uses the BERT model to extract keywords from the text data and analyze the context. The input is the text data, and the output is the extracted keywords and context information. Specifically, the server runs the BERT model and stores the analysis results.

[0753] Step 6:

[0754] The server classifies the conversation content into a specific scope based on the analysis results. For example, it may classify the content into scopes such as "family," "boss," or "friends." The input is keywords and context information, and the output is scope information. Specifically, the server applies a classification algorithm to identify the scope.

[0755] Step 7:

[0756] The server uses an emotion engine to recognize the user's emotions. The IBM Watson Tone Analyzer is used to recognize the user's emotions from text data. The input is text data and the output is emotional information. Specifically, the server inputs text data into the IBM Watson Tone Analyzer and stores the obtained emotional information.

[0757] Step 8:

[0758] The server stores the data classified based on emotion information and scope and sends a notification to the appropriate output device. The input is emotion information and scope information, and the output is data stored in a database and a notification sent. Specific operations include the server storing the data in a database and sending a notification to an output device (e.g., smartphone, PC, digital display).

[0759] Step 9:

[0760] The user provides feedback on the system's output. For example, the user tells the system, "This notification was inappropriate." The input is the user's feedback, and the output is the feedback data sent to the server. The specific action is for the user to provide feedback in text or voice.

[0761] Step 10:

[0762] The server receives feedback from users and retrains the machine learning model based on that feedback. This improves the overall classification accuracy, including the emotion engine. The input is the feedback data, and the output is the updated model. Specifically, the server runs the retraining algorithm to update the model.

[0763] (Application example 2)

[0764] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0765] Conventional food delivery services do not provide personalized suggestions based on the user's conversation content or emotions, limiting the user experience. Furthermore, there is a need for a system that can provide optimal suggestions tailored to specific situations. This creates challenges such as a lack of improvement in user satisfaction and personalization of the service.

[0766] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0767] In this invention, the server includes an input device means for acquiring user voice or text data, a means for preprocessing the data acquired by the input device means to perform noise removal and text conversion, and a means for analyzing the converted text data to perform keyword extraction and context analysis, thereby enabling optimal food delivery service suggestions to be made in accordance with special circumstances and the user's emotions.

[0768] The "input device means" is a device for acquiring voice or text data from a user.

[0769] The "communication means" is a system for transmitting data acquired by the input device means to the server device.

[0770] The "preprocessing means" is a process for removing noise and converting the acquired data into text in the server device.

[0771] The "analysis means" is a function that analyzes the data converted into text, extracts keywords, and performs context analysis.

[0772] The "scope classification means" is a process for classifying the conversation content into a specific scope based on the analysis results.

[0773] The "food delivery proposal method" is a process for proposing appropriate food delivery services based on the classified scope.

[0774] The "emotion recognition means" is a process that recognizes the user's emotions and selects the optimal output according to the scope and emotion.

[0775] The "notification means" is a process for notifying data classified according to scope and emotion to an appropriate output device.

[0776] A "feedback mechanism" is a process that receives user feedback to train and refine the classifier.

[0777] The present invention is a system that analyzes the various conversational content and emotions of users and optimizes food delivery services accordingly. Specific embodiments for carrying out the invention are described below.

[0778] Hardware and Software Configuration

[0779] 1. Input Device Means

[0780] The user inputs voice using the smartphone's microphone, and the input voice data is sent to the server via the smartphone's built-in communication means.

[0781] 2. Means of communication

[0782] The smartphone uses Wi-Fi or mobile data to transmit the captured data to a server in real time.

[0783] 3. Pretreatment Methods

[0784] The server first denoises the captured audio data to produce clear audio, using the Python speech_recognition library and a speech recognition model from the Transformers package.

[0785] 4. Analysis method

[0786] The server converts the audio data into text data and analyzes the text data, including using natural language processing techniques to extract keywords and perform context analysis, such as using TfidfVectorizer and KMeans clustering to classify the scope of the conversation.

[0787] 5. Scope classification and proposal methods

[0788] The server classifies the user's conversation content into a specific scope (e.g., family, friends, boss) and suggests appropriate food delivery services based on the scope. This also takes into account the user's emotional information. For emotion recognition, it uses the Transformers emotion recognition model.

[0789] 6. Emotion recognition means

[0790] The server analyzes the text data to recognize the user's emotions, categorizing them as joy, surprise, anger, etc., and then suggests the most appropriate service accordingly.

[0791] 7. Means of notification

[0792] The server stores the classified data according to scope and emotion and sends notifications to the appropriate output device, which can be displayed on the smartphone display or in the notification bar.

[0793] 8. Feedback channels

[0794] The user provides feedback on the system's output, such as whether the suggestions were appropriate. The server receives this feedback and uses it to improve the accuracy of the classification and emotion recognition methods mentioned above.

[0795] Specific use cases

[0796] Example 1: When a user is having a pleasant conversation with their family

[0797] Prompt: "What are your menu recommendations?"

[0798] Output: "Here are our recommended menu items!"

[0799] Example 2: When the user is stressed

[0800] Prompt: "I'm tired."

[0801] Output: "How about some cold udon noodles today?"

[0802] This makes it possible to provide an optimal food delivery experience based on the user's conversation content and emotions. This system not only improves the user experience, but also significantly improves the accuracy and convenience of food delivery services.

[0803] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0804] Step 1:

[0805] The user inputs voice using the smartphone's microphone, and the voice data is stored in the smartphone's internal memory.

[0806] Step 2:

[0807] The smartphone uses its built-in communication means to transmit the audio data to the server in real time, using Wi-Fi or mobile data.

[0808] Step 3:

[0809] The server pre-processes the received audio data using a noise reduction algorithm, resulting in clear audio data. The denoised audio data is then input to the next step.

[0810] Step 4:

[0811] The server converts the preprocessed audio data into text using a speech recognition model, using the Python speech_recognition library and the Transformers package, and the converted text data becomes the input for the next step.

[0812] Step 5:

[0813] The server analyzes the text data using natural language processing techniques. Specifically, it uses TfidfVectorizer and KMeans clustering to extract keywords and perform context analysis, and classifies the conversation content into specific scopes. The analyzed scope information becomes the input data for the next step.

[0814] Step 6:

[0815] The server generates optimal food delivery service recommendations based on the scope. It uses an emotion recognition model to evaluate the user's emotions, such as daigo / bert-base-japanese-sentiment, and generates recommendations that incorporate the user's emotional information.

[0816] Step 7:

[0817] The server stores the data classified according to scope and emotion and sends a notification to the appropriate output device (smartphone display or notification bar). For example, messages such as "Here are our recommended menu items!" or "How about some cold udon noodles today?" are displayed.

[0818] Step 8:

[0819] The user provides feedback on the system's suggestions and outputs, which are then sent from the smartphone to the server.

[0820] Step 9:

[0821] The server uses the feedback it receives to retrain its machine learning models (classification and emotion recognition) and improve the overall accuracy of the system, resulting in better suggestions for future use.

[0822] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0823] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0824] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0825] [Third embodiment]

[0826] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0827] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0828] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0829] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0830] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0831] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0832] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0833] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0834] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0835] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0836] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0837] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0838] This invention is a system that automatically identifies various conversational contents of users and classifies them into appropriate scopes to manage and provide information appropriate to each of them. A specific embodiment of this system is shown below.

[0839] System Overview

[0840] 1. Capturing User Input

[0841] The user starts talking to the terminal, which has the function of capturing the user's voice and text data.

[0842] The device uses a microphone and keyboard to capture the user's voice and text data in real time, and the captured data is immediately sent to the server.

[0843] 2. Preprocessing of audio data

[0844] The server receives the data sent from the device. Since the received voice data may contain noise, it first applies a noise reduction algorithm to generate clear voice data.

[0845] The clear audio data is then converted into text data using a speech recognition model (e.g., a speech-to-text API).

[0846] 3. Analysis using natural language processing

[0847] The server analyzes the text data, extracts keywords, and performs context analysis. This analysis uses natural language processing technology. The analysis identifies the content of the conversation.

[0848] Based on the analysis results, the server classifies the conversation content into a specific scope (e.g., family, boss, friends, etc.) using a pre-defined algorithm.

[0849] 4. Information management by scope

[0850] The server stores data classified by scope in a database or storage device, and sends different information to the appropriate output device (such as a display or notification system) based on the scope.

[0851] For example, family-related conversations will be displayed on a living room display, while work-related conversations will be sent to a PC or smartphone at work.

[0852] 5. Processing and Learning from User Feedback

[0853] Users can provide feedback on the system's output, for example, if an item is classified into an incorrect scope, the user can provide feedback to correct it.

[0854] The server receives user feedback and uses it to retrain the machine learning model, improving the system's classification accuracy.

[0855] Specific use cases

[0856] Scenario 1: Conversation with family

[0857] 1. A user is having a conversation with a family member in the living room, and the conversation is captured on the device.

[0858] 2. The device sends the audio data to the server.

[0859] 3. The server removes noise from the audio data and converts it into text.

[0860] 4. The server analyzes the text data and identifies the "family" and scope.

[0861] 5. The server displays information about the family on the living room display.

[0862] Scenario 2: A conversation with your boss

[0863] 1. A user is talking to their boss in the office, and the conversation is captured by a headset.

[0864] 2. The device sends the audio data to the server.

[0865] 3. The server removes noise from the audio data and converts it into text.

[0866] 4. The server analyzes the text data and identifies the "boss" and scope.

[0867] 5. The server notifies the work-related information to the PC or smartphone at work.

[0868] Through these processes, the content of user conversations is automatically analyzed and appropriately categorized, allowing users to seamlessly manage different types of conversations and utilize information efficiently.

[0869] The processing flow will be explained below.

[0870] Step 1:

[0871] The user begins speaking. Audio capture begins in the living room, office, or other environment.

[0872] Step 2:

[0873] The device uses a built-in microphone or an external microphone to capture the user's voice data, which is collected in real time and sent to a server (possibly temporarily stored locally).

[0874] Step 3:

[0875] The device uses a communication means to send the captured audio data to a server, which can be via an internet connection or a local network.

[0876] Step 4:

[0877] The server receives the voice data from the device. Since the received data is likely to contain noise as it is, a noise reduction algorithm is first applied to it.

[0878] Step 5:

[0879] The server applies noise reduction algorithms to convert the audio data into a clearer version. This step uses digital signal processing techniques to remove background noise and unwanted sounds.

[0880] Step 6:

[0881] The server uses a speech recognition model (e.g., Google Speech-to-Text API) to convert the noise-removed audio data into text data, which is then converted into readable text data.

[0882] Step 7:

[0883] The server analyzes the converted text data. In this analysis step, natural language processing (NLP) techniques are used to extract keywords and perform context analysis. The meaning and importance of each part of the text data are then extracted.

[0884] Step 8:

[0885] The server classifies the text data into specific scopes based on the extracted keywords and context information, such as "family," "boss," and "friends."

[0886] Step 9:

[0887] The server stores the classified data. Based on the scope of classification, it is stored in the appropriate database or storage device. For example, data about family members is stored in the user's household database.

[0888] Step 10:

[0889] The server then sends the stored data to the appropriate output device: depending on the scope, family information might be sent to a display in the living room, and work-related information might be sent to a PC or smartphone at work.

[0890] Step 11:

[0891] The user provides feedback on the system's output, for example if a classification is incorrect or if additional information is needed, and the user provides that information to the system.

[0892] Step 12:

[0893] The server receives user feedback and retrains the machine learning model based on it. This retraining improves classification accuracy from the next time onwards. By appropriately incorporating feedback, the accuracy of the system continues to improve.

[0894] Example 1

[0895] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0896] In today's world, it is important for users to appropriately manage conversations occurring in diverse environments and quickly and accurately obtain information appropriate to each context. However, conventional systems lack the accuracy of noise reduction and text conversion of speech data, as well as the ability to analyze conversation content and classify it into appropriate scopes, making it difficult to provide information tailored to user needs. As a result, users are unable to use information efficiently, resulting in issues such as reduced work efficiency and quality of life.

[0897] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0898] In this invention, the server includes an input device for acquiring user voice or text data, a communication device for transmitting the acquired data to the server device, a preprocessing device for preprocessing the acquired data, performing noise reduction and text conversion, a device for analyzing the converted text data and performing keyword extraction and context analysis, a device for classifying the conversation content into specific scopes based on the analysis results, a device for saving the classified data based on the scopes and sending notifications to appropriate display devices, a device for receiving user feedback and training and improving the accuracy of the classification device, a device for analyzing prompts entered by the user using a generative AI model and providing optimal responses, and a device for selecting appropriate notification formats according to different scopes. This allows the system to accurately classify diverse user conversations in real time and provide information tailored to each context. As a result, users can efficiently manage and use information, improving their work efficiency and quality of life.

[0899] The "input device means" is a hardware or software system for acquiring voice and text data from a user.

[0900] "Communication means" refers to the networking protocols and techniques used to transmit data acquired by the input device means to the server device.

[0901] "Preprocessing" refers to the process of removing noise from the acquired data and converting the audio data into text data.

[0902] "Noise reduction" is a technology that removes unnecessary audio components from audio data to generate clear audio data.

[0903] "Text conversion" is the process of converting speech data into text data using natural language processing techniques.

[0904] "Keyword extraction" is a natural language processing technique that extracts important words and phrases from text data.

[0905] "Contextual analysis" is a technology that understands the context of text data and analyzes its meaning and significance.

[0906] "Scope" refers to a category or area for classifying conversation content.

[0907] A "display device" is a hardware or software system for presenting notified information to a user.

[0908] "Feedback" refers to comments and corrections provided by users regarding the system's output.

[0909] "Learning and refinement" is the process of retraining the system's classifiers and improving their performance based on user feedback.

[0910] A "generative AI model" is an artificial intelligence technology that analyzes user input data and generates optimal responses.

[0911] A "prompt sentence" is an input sentence that instructs the generative AI model on what processing to do.

[0912] "Notification format" refers to the method or means by which information is conveyed to the user.

[0913] This invention is a system that automatically identifies various conversation contents of users, classifies them into appropriate scopes, and manages and provides information corresponding to each of them. Specific embodiments for carrying out this invention will be described below.

[0914] System Configuration

[0915] 1. Capturing User Input

[0916] A user starts a conversation by turning into a device such as a smartphone or PC. This device is equipped with a microphone for voice input and a keyboard for text input.

[0917] The device captures the user's voice and text data in real time and immediately transmits it to a server. For example, the device captures voice data using a smartphone's microphone and transmits the data to the server via the Internet.

[0918] 2. Data Preprocessing

[0919] The server receives the data sent from the device. Because the audio data often contains noise, it first applies a noise reduction algorithm (e.g., Python's pydub library) to generate clear audio data.

[0920] The server then uses a speech recognition model (e.g., Google Cloud Speech-to-Text API) to convert the clear audio data into text data, which is used for further processing in the system.

[0921] 3. Analysis using natural language processing

[0922] The server analyzes the converted text data, using a Python natural language processing library (e.g., spaCy or NLTK) to extract keywords and analyze the context.

[0923] Based on this analysis, the server classifies the conversation into a specific scope (e.g., family, work, friends) using a pre-trained machine learning model.

[0924] 4. Information management by scope

[0925] The server stores the classified data in a database (e.g., MySQL or MongoDB). The stored data for each scope is notified to the appropriate display device as needed.

[0926] For example, family-related conversations will be displayed on a smart display in the living room, while work-related conversations will be sent to your work computer or smartphone.

[0927] 5. Processing and Learning from User Feedback

[0928] Users can provide feedback on the system's output, for example, if a conversation is misclassified, the user can send corrective feedback.

[0929] The server receives user feedback and uses it to retrain the machine learning model, improving the system's classification accuracy.

[0930] Specific use cases

[0931] Scenario 1: Conversation with family

[0932] 1. A user says in the living room, "Let's go see a movie with the whole family today."

[0933] 2. The device's microphone captures this audio data and sends it to the server.

[0934] 3. The server removes noise and converts the audio to text using the Google Cloud Speech-to-Text API.

[0935] 4. The server analyzes the converted text, extracts the keywords "family" and "movie," and classifies them into the "family" scope based on the context.

[0936] 5. The server notifies the smart display in the living room of this information and displays the message, "Everyone is planning to watch a movie together."

[0937] Scenario 2: A conversation with your boss

[0938] 1. A user sits in the office and asks their boss, "How is this project going?"

[0939] 2. The device's headset microphone captures the audio data and sends it to the server.

[0940] 3. The server removes noise and converts the audio to text using the Google Cloud Speech-to-Text API.

[0941] 4. The server analyzes the converted text, extracts the keywords "project" and "progress," and classifies them into the "work" scope based on the context.

[0942] 5. The server sends notifications about the progress of the project to your work computer or smartphone.

[0943] Prompt Sentence Examples

[0944] "I have recorded a conversation with my family. Please analyze this conversation and categorize it into relevant scopes."

[0945] "I have recorded instructions from my supervisor. Please transcribe them into text, analyze them, and process them as work-related information."

[0946] This invention allows users to appropriately manage various conversation contents and smoothly use information. The system provides information classified into specific scopes in real time, improving the user's life and work efficiency.

[0947] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0948] Step 1:

[0949] The user starts a conversation with the device using voice or text. The input is the user's voice or text data, and the output is the captured raw voice or text data. Specifically, the user uses the microphone on their smartphone to say, "Let's go see a movie with my family today." This voice data is captured by the device's microphone.

[0950] Step 2:

[0951] The device sends the captured voice or text data to the server. The input is the raw voice or text data captured in step 1, and the output is the same data sent to the server. Specifically, the device sends this voice data to the server via Wi-Fi or mobile network.

[0952] Step 3:

[0953] The server receives the voice data sent from the terminal. The input is the raw voice data sent from the terminal, and the output is the same data received on the server. Specifically, the server receives the data using the TCP / IP protocol.

[0954] Step 4:

[0955] The server performs noise removal on the audio data received. The input is raw audio data, and the output is clear audio data with noise removed. Specifically, the Python pydub library is used to filter noise from the audio data.

[0956] Step 5:

[0957] The server converts the noise-removed audio data into text. The input is clear audio data after noise removal, and the output is text data. Specifically, the server converts the audio data into text using the Google Cloud Speech-to-Text API.

[0958] Step 6:

[0959] The server analyzes the text data and performs keyword extraction and context analysis. The input is the converted text data, and the output is the extracted keywords and analyzed context information. Specifically, it uses the Python spaCy library to extract keywords from the text and analyze the context.

[0960] Step 7:

[0961] The server classifies the conversation content into specific scopes based on the analysis results. The input is keywords and context information, and the output is data classified into scopes. For example, based on the extracted keywords "family" and "movies," the conversation content is classified into the "family" scope.

[0962] Step 8:

[0963] The server saves the data classified by scope in a database. The input is the data classified by scope, and the output is the data saved in the database. Specifically, the data is saved using MySQL or MongoDB.

[0964] Step 9:

[0965] The server sends notifications to the appropriate display devices based on the stored data. The input is the data classified into scopes, and the output is the notification sent to the display device. For example, data from the "Family" scope is notified to the smart display in the living room.

[0966] Step 10:

[0967] The user provides feedback on the system's output. The input is the user's feedback information, and the output is the feedback information sent to the server. For example, the user can provide feedback on their smartphone saying, "This classification is correct."

[0968] Step 11:

[0969] The server receives the feedback and uses it to retrain the machine learning model. The input is the feedback information from the user, and the output is an updated machine learning model. Specifically, retraining based on the feedback is performed using scikit-learn.

[0970] (Application example 1)

[0971] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0972] Autonomous vehicles are required to analyze passenger conversations in real time and provide appropriate information. However, there is no system that can effectively respond to passenger requests regarding destination changes and traffic conditions, which hinders the improvement of the driving experience.

[0973] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0974] In this invention, the server includes an input device means for acquiring user voice or text data, a communication means for transmitting the data acquired by the input device means to the server device, a means for preprocessing the acquired data in the server device and performing noise removal and text conversion, a means for analyzing the converted text data and performing keyword extraction and context analysis, a means for classifying the conversation content into specific scopes based on the analysis results, a means for saving the data classified based on the scopes and sending a notification to an appropriate output device, a means for receiving user feedback and training and improving the accuracy of the classification means, and a means for providing destination information and route change information based on the passenger conversation content. This makes it possible to analyze the passenger conversation content in real time and provide appropriate navigation information and recommended routes.

[0975] "User Voice or Text Data" means any voice or text information input by a User through conversation, which is captured and analyzed by the System.

[0976] "Input device means" refers to a device or function that acquires user voice or text data, including a microphone and keyboard.

[0977] "Communication means" refers to a function for transmitting data acquired by the input device means to a server device, including internet connection and wireless communication.

[0978] "Server device" refers to a central management system for preprocessing the acquired data and performing noise removal and text conversion.

[0979] "Noise removal and text conversion means" refers to a processing function for removing noise from audio data and converting audio to text.

[0980] "Analysis means" refers to the function for analyzing text data, extracting keywords, and analyzing context. Natural language processing technology is used for this.

[0981] "Means for classifying into scope" refers to the function of classifying conversation content into specific categories or ranges based on the analysis results.

[0982] "Output Device" means a device or function for displaying or communicating data classified based on scope, including a display or notification system.

[0983] "Means for receiving user feedback and for learning and improving accuracy" refers to a learning function for improving the accuracy of the classification means based on feedback from users.

[0984] "Means of providing destination information and route change information based on passenger conversation content" refers to the function of analyzing passenger conversation content in real time and providing appropriate navigation information and recommended routes.

[0985] A system for realizing this application example analyzes passenger conversations in real time and provides appropriate navigation information and recommended routes. An embodiment of this system will be described in detail below.

[0986] First, the server acquires data using an input device means for acquiring voice or text data from a user (passenger). This input device means may include a microphone on a smartphone or a microphone built into the vehicle. The acquired voice data is then transmitted to the server device via a communication means. The communication means may include an internet connection or wireless communication.

[0987] The server receives the transmitted voice data and first performs noise reduction processing to produce clear voice data. It then converts the voice data into text using speech recognition software (e.g., Google's speech recognition API). This text data is then analyzed using natural language processing techniques (e.g., the BERT model) to extract keywords and perform context analysis.

[0988] The server then classifies the conversation content into a specific scope (e.g., "change of destination" or "traffic information") based on the analysis results, and notifies the appropriate output device (vehicle infotainment system or smartphone display).

[0989] Additionally, the server receives user feedback to train and refine the classification method, and can retrain the machine learning model to improve future classification accuracy.

[0990] For example, if a passenger says, "I want to change my destination," the system works as follows:

[0991] 1. Voice input: "I want to change my destination."

[0992] 2. Text converted by speech recognition: "I would like to change my destination."

[0993] 3. Analysis results: destination change

[0994] 4. Notification: "Enter new destination."

[0995] Also, if a passenger says "Give me the latest traffic information," the following happens:

[0996] 1. Voice input: "Give me traffic updates"

[0997] 2. Text converted by speech recognition: "What is the latest traffic situation?"

[0998] 3. Analysis result: Traffic information request

[0999] 4. Notification: "Shows traffic updates."

[1000] The above steps make it possible to analyze passenger conversations in real time and provide appropriate navigation information and recommended routes.

[1001] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1002] Step 1:

[1003] A user starts a conversation in the car. Voice data is input. This voice data is captured using a microphone on the smartphone or a microphone built into the vehicle. Voice data is obtained as input.

[1004] Step 2:

[1005] The device transmits the captured audio data to a server in real time, either via an internet connection or wireless communication. The device receives audio data as input and transmits it to the server as output.

[1006] Step 3:

[1007] The server preprocesses the received audio data using a noise reduction algorithm, specifically applying a noise filter to generate clear audio data. The input is the captured audio data, and the output is the clear audio data with noise removed.

[1008] Step 4:

[1009] The server converts the noise-removed voice data into text data using speech recognition software (e.g., Google's speech recognition API). The input is clear voice data, and the output is the converted text data.

[1010] Step 5:

[1011] The server analyzes the converted text data and performs keyword extraction and context analysis. This process uses natural language processing techniques (e.g., the BERT model). The input is the text data, and the output is the analyzed keywords and context information.

[1012] Step 6:

[1013] The server classifies the conversation content into a specific scope based on the analysis results, and then categorizes it into categories that fit the passenger's request (e.g., "Destination Change" or "Traffic Information"). The input is the analyzed keywords and context information, and the output is the scoped information.

[1014] Step 7:

[1015] The server stores the data classified based on the scope and sends notifications to the appropriate output device (e.g., a vehicle display or a smartphone). The input is the information classified by the scope, and the output is a notification to the display or the like.

[1016] Step 8:

[1017] The user can provide feedback on the system's output. For example, if a classification is made into an incorrect scope, the user can provide feedback to correct it. The input is the user's feedback information, and the output is data to improve the accuracy of the classification algorithm.

[1018] Step 9:

[1019] The server receives user feedback and retrains the machine learning model, which can improve future classification accuracy. The input is the user feedback data, and the output is an improved classification algorithm.

[1020] The above steps make it possible to analyze passenger conversations in real time and provide appropriate navigation information and recommended routes.

[1021] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1022] This invention is a system that automatically identifies various conversational content of users and categorizes them into appropriate scopes to manage and provide information according to each. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, more advanced analysis and categorization can be achieved.

[1023] System Overview

[1024] 1. Capturing User Input

[1025] The user initiates a conversation, either by voice or text input, in a living room, office, or other environment.

[1026] The device uses a microphone and keyboard to capture the user's voice data and text data in real time and transmits it to the server.

[1027] 2. Preprocessing of audio data

[1028] The server receives the voice data sent from the device. Because the received data contains noise, it first applies a noise reduction algorithm to generate clear voice data.

[1029] 3. Speech Recognition and Text Conversion

[1030] The server converts the noise-removed speech data into text data using a speech recognition model, which transforms the speech data into readable text.

[1031] 4. Analysis using natural language processing

[1032] The server analyzes the text data, extracts keywords, and performs context analysis, thereby understanding the meaning and importance of the text data.

[1033] Based on these analysis results, the server classifies the conversation content into specific scopes (e.g., family, boss, friends, etc.).

[1034] 5. Emotion Recognition by Emotion Engine

[1035] The server inputs the text data into an emotion engine to recognize the user's emotions, which identify emotions such as joy, anger, sadness, and surprise.

[1036] The recognized emotional information influences the analysis results and scope classification, enabling more accurate classification.

[1037] 6. Information management and notification for each scope

[1038] The server stores the classified data in the appropriate database and sends notifications to the appropriate output devices based on the classified scope.

[1039] The content of the notification is adjusted based on the emotional information. For example, if the user is feeling stressed, the notification content may be simplified.

[1040] 7. Processing and Learning from User Feedback

[1041] The user provides feedback on the system's output, for example by sending feedback to the system if the classification is incorrect or if more information is needed.

[1042] The server receives user feedback and uses it to retrain the machine learning model, improving the overall classification accuracy, including the emotion engine.

[1043] Specific use cases

[1044] Scenario 1: Conversation with family

[1045] 1. A user is having a conversation with a family member in the living room, and the audio content is captured by the device.

[1046] 2. The device sends the captured audio data to the server.

[1047] 3. The server removes noise from the audio data and converts it into text.

[1048] 4. The server analyzes the text data and identifies the "family" and scope.

[1049] 5. The server uses an emotion engine to recognize the user's emotions. For example, if the user is talking happily, appropriate information will be output.

[1050] 6. The server displays information about the family on the living room display.

[1051] Scenario 2: A conversation with your boss

[1052] 1. A user is talking to their boss in the office, and the conversation is captured by a headset.

[1053] 2. The device sends the audio data to the server.

[1054] 3. The server removes noise from the audio data and converts it into text.

[1055] 4. The server analyzes the text data and identifies the "boss" and scope.

[1056] 5. The server uses an emotion engine to recognize the user's emotions. For example, if the user is feeling stressed, the notification content will be simplified.

[1057] 6. The server notifies the work-related information to the PC or smartphone at work.

[1058] This will enable advanced information management and notifications that take the user's emotions into consideration, improving the user experience.

[1059] The processing flow will be explained below.

[1060] Step 1:

[1061] The user initiates a conversation, inputting voice or text data in a specific environment, such as a living room or office.

[1062] Step 2:

[1063] The device uses a microphone and keyboard to capture the user's voice and text data in real time, for example, using the built-in microphone of a smartphone or PC.

[1064] Step 3:

[1065] The device transmits the captured voice and text data to a server via an internet connection.

[1066] Step 4:

[1067] The server preprocesses the audio data received from the device. First, it applies a noise reduction algorithm to generate clear audio data.

[1068] Step 5:

[1069] The server converts the noise-removed audio data into text data using a speech recognition model (e.g., Google Speech-to-Text API).

[1070] Step 6:

[1071] The server analyzes the text data using natural language processing (NLP) techniques, which include keyword extraction and context analysis.

[1072] Step 7:

[1073] The server classifies the analyzed text data into specific scopes (e.g., family, boss, friends, etc.) using topic modeling and machine learning algorithms.

[1074] Step 8:

[1075] The server inputs the text data into an emotion engine to recognize the user's emotions, which identify emotions such as joy, anger, sadness, surprise, etc.

[1076] Step 9:

[1077] The server reevaluates the scope classification based on the analysis results together with the emotional information, and updates the optimal scope classification, thereby achieving highly accurate classification that includes emotional information.

[1078] Step 10:

[1079] The server stores the data classified based on the scope in a database or storage device. For example, data about a family member is stored in a household database.

[1080] Step 11:

[1081] The server sends the stored data to the appropriate output device. The notification content is adjusted based on the scope classification and emotion information. For example, if the user is feeling stressed, the notification content is simplified.

[1082] Step 12:

[1083] The user provides feedback on the system's output, for example, by sending feedback on scope classification or emotion recognition.

[1084] Step 13:

[1085] The server receives user feedback and retrains the machine learning model, improving classification and emotion recognition accuracy.

[1086] Example 2

[1087] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1088] In modern society, users have diverse conversations, and there is a need for automatic identification and appropriate classification. However, conventional technologies have faced challenges, such as insufficient accuracy in classifying conversation content and difficulty in managing information taking user emotions into account. It is also not easy to improve the accuracy of the system by reflecting user feedback.

[1089] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1090] In this invention, the server includes an input device for acquiring user voice or text data, a communication device for transmitting the data acquired by the input device to the server, a device for preprocessing the acquired data in the server device, performing noise reduction and text conversion, a device for analyzing the converted text data and performing keyword extraction and context analysis, a device for classifying the conversation content into a specific scope based on the analysis results, a device for recognizing the user's emotions using an emotion engine, a device for saving the classified data based on the emotion information and scope and sending a notification to an appropriate output device, and a device for receiving user feedback and training and improving the accuracy of the classification device. This enables automatic identification and classification of diverse user conversation content with high accuracy, and further realizes information management and notification that takes user emotions into consideration. Furthermore, the accuracy of the system can be improved by reflecting user feedback.

[1091] "Input device means" refers to a device for obtaining voice or text data from a user.

[1092] The "communication means" refers to a means for transmitting data acquired by the input device means to the server device.

[1093] The "noise removal means" refers to a means for removing noise from the voice data acquired in the server device.

[1094] The "text conversion means" refers to a means that provides a function for converting noise-removed voice data into text data.

[1095] "Keyword extraction means" refers to means for extracting important keywords from text data.

[1096] The "context analysis means" refers to a means for analyzing the context of text data.

[1097] The "scope classification means" refers to a means for classifying the conversation content into a specific scope based on the analysis results.

[1098] "Emotion recognition means" refers to means for recognizing the user's emotions using an emotion engine.

[1099] "Output device" refers to a device for notifying appropriate information based on classification and emotion recognition results.

[1100] "Data storage means" refers to a means for storing classified conversation data.

[1101] "Feedback receiving means" refers to a means for receiving feedback provided by a user.

[1102] "Training refinement means" refers to means for training and refining a classifier based on received feedback.

[1103] This invention is a system that automatically identifies diverse conversational content of users and classifies it into appropriate scopes. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, more advanced analysis and classification can be achieved. Detailed embodiments of the program for this system are described below.

[1104] Hardware and software used

[1105] Input device: A device that allows a user to input voice or text data. A microphone is used for voice input, and a keyboard is used for text input.

[1106] Communication method: A network communication method for sending data captured by an input device to a server. This includes Wi-Fi and wired LAN.

[1107] Server: A central processing unit for data pre-processing, speech recognition, natural language processing analysis, and emotion recognition. Software used includes the Google Speech-to-Text API, BERT model, and IBM Watson Tone Analyzer.

[1108] Data processing and calculation

[1109] 1. Preprocessing of audio data

[1110] The server receives the audio data sent from the terminal and applies a noise reduction algorithm (e.g., Spectral Subtraction) to generate clear audio data.

[1111] 2. Speech Recognition and Text Conversion

[1112] The server uses the Google Speech-to-Text API to convert the noise-removed audio data into text data. Specifically, the audio data is converted into text that says, "I'd like to talk about today's meeting."

[1113] 3. Analysis using natural language processing

[1114] The server uses the BERT model to extract keywords from the generated text data and perform contextual analysis, thereby understanding the meaning and importance of the text data.

[1115] 4. Scope Classification

[1116] Based on the analysis results, the server classifies the text data into specific scopes such as "family," "bosses," and "friends."

[1117] 5. Emotion recognition

[1118] The server uses IBM Watson Tone Analyzer to recognize the user's emotions (happiness, anger, sadness, surprise, etc.), and based on this emotional information, the accuracy of scope classification is further improved.

[1119] 6. Information Management and Notification

[1120] The server stores the classified data in a database and sends appropriate notifications to output devices (e.g., smartphones, PCs, digital displays, etc.) The content of the notifications is adjusted taking into account emotional information.

[1121] 7. Feedback and Learning

[1122] The user provides feedback on the system's output, for example, telling the system, "This notification was inappropriate."

[1123] The server receives user feedback and updates the machine learning model, which improves the accuracy of classification and emotion recognition in future.

[1124] Specific use cases

[1125] Scenario 1: Conversation with family

[1126] 1. A user is sitting in the living room and says, "I want to watch a movie with my family today."

[1127] 2. The device captures this conversation and sends it to the server.

[1128] 3. The server removes noise from the audio data and converts it to text using the Google Speech-to-Text API.

[1129] 4. The server analyzes the text using the BERT model and identifies the scope "family".

[1130] 5. The server uses IBM Watson Tone Analyzer to recognize the user's emotion as "joy."

[1131] 6. The server displays movie information for the family on the living room display.

[1132] Scenario 2: A conversation with your boss

[1133] 1. A user is in the office and says, "I'd like to talk to my boss about the progress of the project."

[1134] 2. The device captures the conversation and sends it to the server.

[1135] 3. The server removes noise from the audio data and converts it to text using the Google Speech-to-Text API.

[1136] 4. The server analyzes the text using the BERT model and identifies the scope "boss."

[1137] 5. The server uses IBM Watson Tone Analyzer to recognize the user's emotion as "stress."

[1138] 6. The server sends simplified project progress notifications to your work PC or smartphone.

[1139] This series of processes enables the system to automatically identify diverse user conversation content and classify it into the appropriate scope. Furthermore, by recognizing the user's emotions, more accurate classification and notifications are achieved. As a result, the user experience is improved, and by appropriately incorporating feedback, the system continues to learn and improve its accuracy.

[1140] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1141] Step 1:

[1142] A user starts a conversation. A user starts a voice conversation in their living room or office. For example, they might say, "I want to watch a movie with my family today." The input is real-time voice data. Specifically, the user speaks into a microphone, and voice data is captured.

[1143] Step 2:

[1144] The device captures voice data in real time and sends it to the server. In this case, the microphone functions as an input device and sends the captured voice data to the server. The input is voice data, and the output is digital voice data sent to the server. Specifically, the device sends the data via Wi-Fi or wired LAN.

[1145] Step 3:

[1146] The server removes noise from the received audio data. It applies a noise removal algorithm (e.g., Spectral Subtraction) to generate clear audio data. The input is the audio data sent from the device, and the output is clear audio data with noise removed. Specifically, the server executes the noise removal module.

[1147] Step 4:

[1148] The server converts the noise-removed audio data into text data. The Google Speech-to-Text API is used to convert the audio data into text. The input is the noise-removed audio data, and the output is text data. Specifically, the server inputs the audio data into the Google Speech-to-Text API and stores the resulting text data.

[1149] Step 5:

[1150] The server analyzes the text data and performs keyword extraction and context analysis. It uses the BERT model to extract keywords from the text data and analyze the context. The input is the text data, and the output is the extracted keywords and context information. Specifically, the server runs the BERT model and stores the analysis results.

[1151] Step 6:

[1152] The server classifies the conversation content into a specific scope based on the analysis results. For example, it may classify the content into scopes such as "family," "boss," or "friends." The input is keywords and context information, and the output is scope information. Specifically, the server applies a classification algorithm to identify the scope.

[1153] Step 7:

[1154] The server uses an emotion engine to recognize the user's emotions. The IBM Watson Tone Analyzer is used to recognize the user's emotions from text data. The input is text data and the output is emotional information. Specifically, the server inputs text data into the IBM Watson Tone Analyzer and stores the obtained emotional information.

[1155] Step 8:

[1156] The server stores the data classified based on emotion information and scope and sends a notification to the appropriate output device. The input is emotion information and scope information, and the output is data stored in a database and a notification sent. Specific operations include the server storing the data in a database and sending a notification to an output device (e.g., smartphone, PC, digital display).

[1157] Step 9:

[1158] The user provides feedback on the system's output. For example, the user tells the system, "This notification was inappropriate." The input is the user's feedback, and the output is the feedback data sent to the server. The specific action is for the user to provide feedback in text or voice.

[1159] Step 10:

[1160] The server receives feedback from users and retrains the machine learning model based on that feedback. This improves the overall classification accuracy, including the emotion engine. The input is the feedback data, and the output is the updated model. Specifically, the server runs the retraining algorithm to update the model.

[1161] (Application example 2)

[1162] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1163] Conventional food delivery services do not provide personalized suggestions based on the user's conversation content or emotions, limiting the user experience. Furthermore, there is a need for a system that can provide optimal suggestions tailored to specific situations. This creates challenges such as a lack of improvement in user satisfaction and personalization of the service.

[1164] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1165] In this invention, the server includes an input device means for acquiring user voice or text data, a means for preprocessing the data acquired by the input device means to perform noise removal and text conversion, and a means for analyzing the converted text data to perform keyword extraction and context analysis, thereby enabling optimal food delivery service suggestions to be made in accordance with special circumstances and the user's emotions.

[1166] The "input device means" is a device for acquiring voice or text data from a user.

[1167] The "communication means" is a system for transmitting data acquired by the input device means to the server device.

[1168] The "preprocessing means" is a process for removing noise and converting the acquired data into text in the server device.

[1169] The "analysis means" is a function that analyzes the data converted into text, extracts keywords, and performs context analysis.

[1170] The "scope classification means" is a process for classifying the conversation content into a specific scope based on the analysis results.

[1171] The "food delivery proposal method" is a process for proposing appropriate food delivery services based on the classified scope.

[1172] The "emotion recognition means" is a process that recognizes the user's emotions and selects the optimal output according to the scope and emotion.

[1173] The "notification means" is a process for notifying data classified according to scope and emotion to an appropriate output device.

[1174] A "feedback mechanism" is a process that receives user feedback to train and refine the classifier.

[1175] The present invention is a system that analyzes the various conversational content and emotions of users and optimizes food delivery services accordingly. Specific embodiments for carrying out the invention are described below.

[1176] Hardware and Software Configuration

[1177] 1. Input Device Means

[1178] The user inputs voice using the smartphone's microphone, and the input voice data is sent to the server via the smartphone's built-in communication means.

[1179] 2. Means of communication

[1180] The smartphone uses Wi-Fi or mobile data to transmit the captured data to a server in real time.

[1181] 3. Pretreatment Methods

[1182] The server first denoises the captured audio data to produce clear audio, using the Python speech_recognition library and a speech recognition model from the Transformers package.

[1183] 4. Analysis method

[1184] The server converts the audio data into text data and analyzes the text data, including using natural language processing techniques to extract keywords and perform context analysis, such as using TfidfVectorizer and KMeans clustering to classify the scope of the conversation.

[1185] 5. Scope classification and proposal methods

[1186] The server classifies the user's conversation content into a specific scope (e.g., family, friends, boss) and suggests appropriate food delivery services based on the scope. This also takes into account the user's emotional information. For emotion recognition, it uses the Transformers emotion recognition model.

[1187] 6. Emotion recognition means

[1188] The server analyzes the text data to recognize the user's emotions, categorizing them as joy, surprise, anger, etc., and then suggests the most appropriate service accordingly.

[1189] 7. Means of notification

[1190] The server stores the classified data according to scope and emotion and sends notifications to the appropriate output device, which can be displayed on the smartphone display or in the notification bar.

[1191] 8. Feedback channels

[1192] The user provides feedback on the system's output, such as whether the suggestions were appropriate. The server receives this feedback and uses it to improve the accuracy of the classification and emotion recognition methods mentioned above.

[1193] Specific use cases

[1194] Example 1: When a user is having a pleasant conversation with their family

[1195] Prompt: "What are your menu recommendations?"

[1196] Output: "Here are our recommended menu items!"

[1197] Example 2: When the user is stressed

[1198] Prompt: "I'm tired."

[1199] Output: "How about some cold udon noodles today?"

[1200] This makes it possible to provide an optimal food delivery experience based on the user's conversation content and emotions. This system not only improves the user experience, but also significantly improves the accuracy and convenience of food delivery services.

[1201] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1202] Step 1:

[1203] The user inputs voice using the smartphone's microphone, and the voice data is stored in the smartphone's internal memory.

[1204] Step 2:

[1205] The smartphone uses its built-in communication means to transmit the audio data to the server in real time, using Wi-Fi or mobile data.

[1206] Step 3:

[1207] The server pre-processes the received audio data using a noise reduction algorithm, resulting in clear audio data. The denoised audio data is then input to the next step.

[1208] Step 4:

[1209] The server converts the preprocessed audio data into text using a speech recognition model, using the Python speech_recognition library and the Transformers package, and the converted text data becomes the input for the next step.

[1210] Step 5:

[1211] The server analyzes the text data using natural language processing techniques. Specifically, it uses TfidfVectorizer and KMeans clustering to extract keywords and perform context analysis, and classifies the conversation content into specific scopes. The analyzed scope information becomes the input data for the next step.

[1212] Step 6:

[1213] The server generates optimal food delivery service recommendations based on the scope. It uses an emotion recognition model to evaluate the user's emotions, such as daigo / bert-base-japanese-sentiment, and generates recommendations that incorporate the user's emotional information.

[1214] Step 7:

[1215] The server stores the data classified according to scope and emotion and sends a notification to the appropriate output device (smartphone display or notification bar). For example, messages such as "Here are our recommended menu items!" or "How about some cold udon noodles today?" are displayed.

[1216] Step 8:

[1217] The user provides feedback on the system's suggestions and outputs, which are then sent from the smartphone to the server.

[1218] Step 9:

[1219] The server uses the feedback it receives to retrain its machine learning models (classification and emotion recognition) and improve the overall accuracy of the system, resulting in better suggestions for future use.

[1220] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1221] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1222] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1223] [Fourth embodiment]

[1224] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1225] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1226] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1227] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1228] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1229] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1230] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1231] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1232] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1233] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1234] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1235] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1236] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1237] This invention is a system that automatically identifies various conversational contents of users and classifies them into appropriate scopes to manage and provide information appropriate to each of them. A specific embodiment of this system is shown below.

[1238] System Overview

[1239] 1. Capturing User Input

[1240] The user starts talking to the terminal, which has the function of capturing the user's voice and text data.

[1241] The device uses a microphone and keyboard to capture the user's voice and text data in real time, and the captured data is immediately sent to the server.

[1242] 2. Preprocessing of audio data

[1243] The server receives the data sent from the device. Since the received voice data may contain noise, it first applies a noise reduction algorithm to generate clear voice data.

[1244] The clear audio data is then converted into text data using a speech recognition model (e.g., a speech-to-text API).

[1245] 3. Analysis using natural language processing

[1246] The server analyzes the text data, extracts keywords, and performs context analysis. This analysis uses natural language processing technology. The analysis identifies the content of the conversation.

[1247] Based on the analysis results, the server classifies the conversation content into a specific scope (e.g., family, boss, friends, etc.) using a pre-defined algorithm.

[1248] 4. Information management by scope

[1249] The server stores data classified by scope in a database or storage device, and sends different information to the appropriate output device (such as a display or notification system) based on the scope.

[1250] For example, family-related conversations will be displayed on a living room display, while work-related conversations will be sent to a PC or smartphone at work.

[1251] 5. Processing and Learning from User Feedback

[1252] Users can provide feedback on the system's output, for example, if an item is classified into an incorrect scope, the user can provide feedback to correct it.

[1253] The server receives user feedback and uses it to retrain the machine learning model, improving the system's classification accuracy.

[1254] Specific use cases

[1255] Scenario 1: Conversation with family

[1256] 1. A user is having a conversation with a family member in the living room, and the conversation is captured on the device.

[1257] 2. The device sends the audio data to the server.

[1258] 3. The server removes noise from the audio data and converts it into text.

[1259] 4. The server analyzes the text data and identifies the "family" and scope.

[1260] 5. The server displays information about the family on the living room display.

[1261] Scenario 2: A conversation with your boss

[1262] 1. A user is talking to their boss in the office, and the conversation is captured by a headset.

[1263] 2. The device sends the audio data to the server.

[1264] 3. The server removes noise from the audio data and converts it into text.

[1265] 4. The server analyzes the text data and identifies the "boss" and scope.

[1266] 5. The server notifies the work-related information to the PC or smartphone at work.

[1267] Through these processes, the content of user conversations is automatically analyzed and appropriately categorized, allowing users to seamlessly manage different types of conversations and utilize information efficiently.

[1268] The processing flow will be explained below.

[1269] Step 1:

[1270] The user begins speaking. Audio capture begins in the living room, office, or other environment.

[1271] Step 2:

[1272] The device uses a built-in microphone or an external microphone to capture the user's voice data, which is collected in real time and sent to a server (possibly temporarily stored locally).

[1273] Step 3:

[1274] The device uses a communication means to send the captured audio data to a server, which can be via an internet connection or a local network.

[1275] Step 4:

[1276] The server receives the voice data from the device. Since the received data is likely to contain noise as it is, a noise reduction algorithm is first applied to it.

[1277] Step 5:

[1278] The server applies noise reduction algorithms to convert the audio data into a clearer version. This step uses digital signal processing techniques to remove background noise and unwanted sounds.

[1279] Step 6:

[1280] The server uses a speech recognition model (e.g., Google Speech-to-Text API) to convert the noise-removed audio data into text data, which is then converted into readable text data.

[1281] Step 7:

[1282] The server analyzes the converted text data. In this analysis step, natural language processing (NLP) techniques are used to extract keywords and perform context analysis. The meaning and importance of each part of the text data are then extracted.

[1283] Step 8:

[1284] The server classifies the text data into specific scopes based on the extracted keywords and context information, such as "family," "boss," and "friends."

[1285] Step 9:

[1286] The server stores the classified data. Based on the scope of classification, it is stored in the appropriate database or storage device. For example, data about family members is stored in the user's household database.

[1287] Step 10:

[1288] The server then sends the stored data to the appropriate output device: depending on the scope, family information might be sent to a display in the living room, and work-related information might be sent to a PC or smartphone at work.

[1289] Step 11:

[1290] The user provides feedback on the system's output, for example if a classification is incorrect or if additional information is needed, and the user provides that information to the system.

[1291] Step 12:

[1292] The server receives user feedback and retrains the machine learning model based on it. This retraining improves classification accuracy from the next time onwards. By appropriately incorporating feedback, the accuracy of the system continues to improve.

[1293] Example 1

[1294] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1295] In today's world, it is important for users to appropriately manage conversations occurring in diverse environments and quickly and accurately obtain information appropriate to each context. However, conventional systems lack the accuracy of noise reduction and text conversion of speech data, as well as the ability to analyze conversation content and classify it into appropriate scopes, making it difficult to provide information tailored to user needs. As a result, users are unable to use information efficiently, resulting in issues such as reduced work efficiency and quality of life.

[1296] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1297] In this invention, the server includes an input device for acquiring user voice or text data, a communication device for transmitting the acquired data to the server device, a preprocessing device for preprocessing the acquired data, performing noise reduction and text conversion, a device for analyzing the converted text data and performing keyword extraction and context analysis, a device for classifying the conversation content into specific scopes based on the analysis results, a device for saving the classified data based on the scopes and sending notifications to appropriate display devices, a device for receiving user feedback and training and improving the accuracy of the classification device, a device for analyzing prompts entered by the user using a generative AI model and providing optimal responses, and a device for selecting appropriate notification formats according to different scopes. This allows the system to accurately classify diverse user conversations in real time and provide information tailored to each context. As a result, users can efficiently manage and use information, improving their work efficiency and quality of life.

[1298] The "input device means" is a hardware or software system for acquiring voice and text data from a user.

[1299] "Communication means" refers to the networking protocols and techniques used to transmit data acquired by the input device means to the server device.

[1300] "Preprocessing" refers to the process of removing noise from the acquired data and converting the audio data into text data.

[1301] "Noise reduction" is a technology that removes unnecessary audio components from audio data to generate clear audio data.

[1302] "Text conversion" is the process of converting speech data into text data using natural language processing techniques.

[1303] "Keyword extraction" is a natural language processing technique that extracts important words and phrases from text data.

[1304] "Contextual analysis" is a technology that understands the context of text data and analyzes its meaning and significance.

[1305] "Scope" refers to a category or area for classifying conversation content.

[1306] A "display device" is a hardware or software system for presenting notified information to a user.

[1307] "Feedback" refers to comments and corrections provided by users regarding the system's output.

[1308] "Learning and refinement" is the process of retraining the system's classifiers and improving their performance based on user feedback.

[1309] A "generative AI model" is an artificial intelligence technology that analyzes user input data and generates optimal responses.

[1310] A "prompt sentence" is an input sentence that instructs the generative AI model on what processing to do.

[1311] "Notification format" refers to the method or means by which information is conveyed to the user.

[1312] This invention is a system that automatically identifies various conversation contents of users, classifies them into appropriate scopes, and manages and provides information corresponding to each of them. Specific embodiments for carrying out this invention will be described below.

[1313] System Configuration

[1314] 1. Capturing User Input

[1315] A user starts a conversation by turning into a device such as a smartphone or PC. This device is equipped with a microphone for voice input and a keyboard for text input.

[1316] The device captures the user's voice and text data in real time and immediately transmits it to a server. For example, the device captures voice data using a smartphone's microphone and transmits the data to the server via the Internet.

[1317] 2. Data Preprocessing

[1318] The server receives the data sent from the device. Because the audio data often contains noise, it first applies a noise reduction algorithm (e.g., Python's pydub library) to generate clear audio data.

[1319] The server then uses a speech recognition model (e.g., Google Cloud Speech-to-Text API) to convert the clear audio data into text data, which is used for further processing in the system.

[1320] 3. Analysis using natural language processing

[1321] The server analyzes the converted text data, using a Python natural language processing library (e.g., spaCy or NLTK) to extract keywords and analyze the context.

[1322] Based on this analysis, the server classifies the conversation into a specific scope (e.g., family, work, friends) using a pre-trained machine learning model.

[1323] 4. Information management by scope

[1324] The server stores the classified data in a database (e.g., MySQL or MongoDB). The stored data for each scope is notified to the appropriate display device as needed.

[1325] For example, family-related conversations will be displayed on a smart display in the living room, while work-related conversations will be sent to your work computer or smartphone.

[1326] 5. Processing and Learning from User Feedback

[1327] Users can provide feedback on the system's output, for example, if a conversation is misclassified, the user can send corrective feedback.

[1328] The server receives user feedback and uses it to retrain the machine learning model, improving the system's classification accuracy.

[1329] Specific use cases

[1330] Scenario 1: Conversation with family

[1331] 1. A user says in the living room, "Let's go see a movie with the whole family today."

[1332] 2. The device's microphone captures this audio data and sends it to the server.

[1333] 3. The server removes noise and converts the audio to text using the Google Cloud Speech-to-Text API.

[1334] 4. The server analyzes the converted text, extracts the keywords "family" and "movie," and classifies them into the "family" scope based on the context.

[1335] 5. The server notifies the smart display in the living room of this information and displays the message, "Everyone is planning to watch a movie together."

[1336] Scenario 2: A conversation with your boss

[1337] 1. A user sits in the office and asks their boss, "How is this project going?"

[1338] 2. The device's headset microphone captures the audio data and sends it to the server.

[1339] 3. The server removes noise and converts the audio to text using the Google Cloud Speech-to-Text API.

[1340] 4. The server analyzes the converted text, extracts the keywords "project" and "progress," and classifies them into the "work" scope based on the context.

[1341] 5. The server sends notifications about the progress of the project to your work computer or smartphone.

[1342] Prompt Sentence Examples

[1343] "I have recorded a conversation with my family. Please analyze this conversation and categorize it into relevant scopes."

[1344] "I have recorded instructions from my supervisor. Please transcribe them into text, analyze them, and process them as work-related information."

[1345] This invention allows users to appropriately manage various conversation contents and smoothly use information. The system provides information classified into specific scopes in real time, improving the user's life and work efficiency.

[1346] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1347] Step 1:

[1348] The user starts a conversation with the device using voice or text. The input is the user's voice or text data, and the output is the captured raw voice or text data. Specifically, the user uses the microphone on their smartphone to say, "Let's go see a movie with my family today." This voice data is captured by the device's microphone.

[1349] Step 2:

[1350] The device sends the captured voice or text data to the server. The input is the raw voice or text data captured in step 1, and the output is the same data sent to the server. Specifically, the device sends this voice data to the server via Wi-Fi or mobile network.

[1351] Step 3:

[1352] The server receives the voice data sent from the terminal. The input is the raw voice data sent from the terminal, and the output is the same data received on the server. Specifically, the server receives the data using the TCP / IP protocol.

[1353] Step 4:

[1354] The server performs noise removal on the audio data received. The input is raw audio data, and the output is clear audio data with noise removed. Specifically, the Python pydub library is used to filter noise from the audio data.

[1355] Step 5:

[1356] The server converts the noise-removed audio data into text. The input is clear audio data after noise removal, and the output is text data. Specifically, the server converts the audio data into text using the Google Cloud Speech-to-Text API.

[1357] Step 6:

[1358] The server analyzes the text data and performs keyword extraction and context analysis. The input is the converted text data, and the output is the extracted keywords and analyzed context information. Specifically, it uses the Python spaCy library to extract keywords from the text and analyze the context.

[1359] Step 7:

[1360] The server classifies the conversation content into specific scopes based on the analysis results. The input is keywords and context information, and the output is data classified into scopes. For example, based on the extracted keywords "family" and "movies," the conversation content is classified into the "family" scope.

[1361] Step 8:

[1362] The server saves the data classified by scope in a database. The input is the data classified by scope, and the output is the data saved in the database. Specifically, the data is saved using MySQL or MongoDB.

[1363] Step 9:

[1364] The server sends notifications to the appropriate display devices based on the stored data. The input is the data classified into scopes, and the output is the notification sent to the display device. For example, data from the "Family" scope is notified to the smart display in the living room.

[1365] Step 10:

[1366] The user provides feedback on the system's output. The input is the user's feedback information, and the output is the feedback information sent to the server. For example, the user can provide feedback on their smartphone saying, "This classification is correct."

[1367] Step 11:

[1368] The server receives the feedback and uses it to retrain the machine learning model. The input is the feedback information from the user, and the output is an updated machine learning model. Specifically, retraining based on the feedback is performed using scikit-learn.

[1369] (Application example 1)

[1370] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1371] Autonomous vehicles are required to analyze passenger conversations in real time and provide appropriate information. However, there is no system that can effectively respond to passenger requests regarding destination changes and traffic conditions, which hinders the improvement of the driving experience.

[1372] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1373] In this invention, the server includes an input device means for acquiring user voice or text data, a communication means for transmitting the data acquired by the input device means to the server device, a means for preprocessing the acquired data in the server device and performing noise removal and text conversion, a means for analyzing the converted text data and performing keyword extraction and context analysis, a means for classifying the conversation content into specific scopes based on the analysis results, a means for saving the data classified based on the scopes and sending a notification to an appropriate output device, a means for receiving user feedback and training and improving the accuracy of the classification means, and a means for providing destination information and route change information based on the passenger conversation content. This makes it possible to analyze the passenger conversation content in real time and provide appropriate navigation information and recommended routes.

[1374] "User Voice or Text Data" means any voice or text information input by a User through conversation, which is captured and analyzed by the System.

[1375] "Input device means" refers to a device or function that acquires user voice or text data, including a microphone and keyboard.

[1376] "Communication means" refers to a function for transmitting data acquired by the input device means to a server device, including internet connection and wireless communication.

[1377] "Server device" refers to a central management system for preprocessing the acquired data and performing noise removal and text conversion.

[1378] "Noise removal and text conversion means" refers to a processing function for removing noise from audio data and converting audio to text.

[1379] "Analysis means" refers to the function for analyzing text data, extracting keywords, and analyzing context. Natural language processing technology is used for this.

[1380] "Means for classifying into scope" refers to the function of classifying conversation content into specific categories or ranges based on the analysis results.

[1381] "Output Device" means a device or function for displaying or communicating data classified based on scope, including a display or notification system.

[1382] "Means for receiving user feedback and for learning and improving accuracy" refers to a learning function for improving the accuracy of the classification means based on feedback from users.

[1383] "Means of providing destination information and route change information based on passenger conversation content" refers to the function of analyzing passenger conversation content in real time and providing appropriate navigation information and recommended routes.

[1384] A system for realizing this application example analyzes passenger conversations in real time and provides appropriate navigation information and recommended routes. An embodiment of this system will be described in detail below.

[1385] First, the server acquires data using an input device means for acquiring voice or text data from a user (passenger). This input device means may include a microphone on a smartphone or a microphone built into the vehicle. The acquired voice data is then transmitted to the server device via a communication means. The communication means may include an internet connection or wireless communication.

[1386] The server receives the transmitted voice data and first performs noise reduction processing to produce clear voice data. It then converts the voice data into text using speech recognition software (e.g., Google's speech recognition API). This text data is then analyzed using natural language processing techniques (e.g., the BERT model) to extract keywords and perform context analysis.

[1387] The server then classifies the conversation content into a specific scope (e.g., "change of destination" or "traffic information") based on the analysis results, and notifies the appropriate output device (vehicle infotainment system or smartphone display).

[1388] Additionally, the server receives user feedback to train and refine the classification method, and can retrain the machine learning model to improve future classification accuracy.

[1389] For example, if a passenger says, "I want to change my destination," the system works as follows:

[1390] 1. Voice input: "I want to change my destination."

[1391] 2. Text converted by speech recognition: "I would like to change my destination."

[1392] 3. Analysis results: destination change

[1393] 4. Notification: "Enter new destination."

[1394] Also, if a passenger says "Give me the latest traffic information," the following happens:

[1395] 1. Voice input: "Give me traffic updates"

[1396] 2. Text converted by speech recognition: "What is the latest traffic situation?"

[1397] 3. Analysis result: Traffic information request

[1398] 4. Notification: "Shows traffic updates."

[1399] The above steps make it possible to analyze passenger conversations in real time and provide appropriate navigation information and recommended routes.

[1400] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1401] Step 1:

[1402] A user starts a conversation in the car. Voice data is input. This voice data is captured using a microphone on the smartphone or a microphone built into the vehicle. Voice data is obtained as input.

[1403] Step 2:

[1404] The device transmits the captured audio data to a server in real time, either via an internet connection or wireless communication. The device receives audio data as input and transmits it to the server as output.

[1405] Step 3:

[1406] The server preprocesses the received audio data using a noise reduction algorithm, specifically applying a noise filter to generate clear audio data. The input is the captured audio data, and the output is the clear audio data with noise removed.

[1407] Step 4:

[1408] The server converts the noise-removed voice data into text data using speech recognition software (e.g., Google's speech recognition API). The input is clear voice data, and the output is the converted text data.

[1409] Step 5:

[1410] The server analyzes the converted text data and performs keyword extraction and context analysis. This process uses natural language processing techniques (e.g., the BERT model). The input is the text data, and the output is the analyzed keywords and context information.

[1411] Step 6:

[1412] The server classifies the conversation content into a specific scope based on the analysis results, and then categorizes it into categories that fit the passenger's request (e.g., "Destination Change" or "Traffic Information"). The input is the analyzed keywords and context information, and the output is the scoped information.

[1413] Step 7:

[1414] The server stores the data classified based on the scope and sends notifications to the appropriate output device (e.g., a vehicle display or a smartphone). The input is the information classified by the scope, and the output is a notification to the display or the like.

[1415] Step 8:

[1416] The user can provide feedback on the system's output. For example, if a classification is made into an incorrect scope, the user can provide feedback to correct it. The input is the user's feedback information, and the output is data to improve the accuracy of the classification algorithm.

[1417] Step 9:

[1418] The server receives user feedback and retrains the machine learning model, which can improve future classification accuracy. The input is the user feedback data, and the output is an improved classification algorithm.

[1419] The above steps make it possible to analyze passenger conversations in real time and provide appropriate navigation information and recommended routes.

[1420] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1421] This invention is a system that automatically identifies various conversational content of users and categorizes them into appropriate scopes to manage and provide information according to each. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, more advanced analysis and categorization can be achieved.

[1422] System Overview

[1423] 1. Capturing User Input

[1424] The user initiates a conversation, either by voice or text input, in a living room, office, or other environment.

[1425] The device uses a microphone and keyboard to capture the user's voice data and text data in real time and transmits it to the server.

[1426] 2. Preprocessing of audio data

[1427] The server receives the voice data sent from the device. Because the received data contains noise, it first applies a noise reduction algorithm to generate clear voice data.

[1428] 3. Speech Recognition and Text Conversion

[1429] The server converts the noise-removed speech data into text data using a speech recognition model, which transforms the speech data into readable text.

[1430] 4. Analysis using natural language processing

[1431] The server analyzes the text data, extracts keywords, and performs context analysis, thereby understanding the meaning and importance of the text data.

[1432] Based on these analysis results, the server classifies the conversation content into specific scopes (e.g., family, boss, friends, etc.).

[1433] 5. Emotion Recognition by Emotion Engine

[1434] The server inputs the text data into an emotion engine to recognize the user's emotions, which identify emotions such as joy, anger, sadness, and surprise.

[1435] The recognized emotional information influences the analysis results and scope classification, enabling more accurate classification.

[1436] 6. Information management and notification for each scope

[1437] The server stores the classified data in the appropriate database and sends notifications to the appropriate output devices based on the classified scope.

[1438] The content of the notification is adjusted based on the emotional information. For example, if the user is feeling stressed, the notification content may be simplified.

[1439] 7. Processing and Learning from User Feedback

[1440] The user provides feedback on the system's output, for example by sending feedback to the system if the classification is incorrect or if more information is needed.

[1441] The server receives user feedback and uses it to retrain the machine learning model, improving the overall classification accuracy, including the emotion engine.

[1442] Specific use cases

[1443] Scenario 1: Conversation with family

[1444] 1. A user is having a conversation with a family member in the living room, and the audio content is captured by the device.

[1445] 2. The device sends the captured audio data to the server.

[1446] 3. The server removes noise from the audio data and converts it into text.

[1447] 4. The server analyzes the text data and identifies the "family" and scope.

[1448] 5. The server uses an emotion engine to recognize the user's emotions. For example, if the user is talking happily, appropriate information will be output.

[1449] 6. The server displays information about the family on the living room display.

[1450] Scenario 2: A conversation with your boss

[1451] 1. A user is talking to their boss in the office, and the conversation is captured by a headset.

[1452] 2. The device sends the audio data to the server.

[1453] 3. The server removes noise from the audio data and converts it into text.

[1454] 4. The server analyzes the text data and identifies the "boss" and scope.

[1455] 5. The server uses an emotion engine to recognize the user's emotions. For example, if the user is feeling stressed, the notification content will be simplified.

[1456] 6. The server notifies the work-related information to the PC or smartphone at work.

[1457] This will enable advanced information management and notifications that take the user's emotions into consideration, improving the user experience.

[1458] The processing flow will be explained below.

[1459] Step 1:

[1460] The user initiates a conversation, inputting voice or text data in a specific environment, such as a living room or office.

[1461] Step 2:

[1462] The device uses a microphone and keyboard to capture the user's voice and text data in real time, for example, using the built-in microphone of a smartphone or PC.

[1463] Step 3:

[1464] The device transmits the captured voice and text data to a server via an internet connection.

[1465] Step 4:

[1466] The server preprocesses the audio data received from the device. First, it applies a noise reduction algorithm to generate clear audio data.

[1467] Step 5:

[1468] The server converts the noise-removed audio data into text data using a speech recognition model (e.g., Google Speech-to-Text API).

[1469] Step 6:

[1470] The server analyzes the text data using natural language processing (NLP) techniques, which include keyword extraction and context analysis.

[1471] Step 7:

[1472] The server classifies the analyzed text data into specific scopes (e.g., family, boss, friends, etc.) using topic modeling and machine learning algorithms.

[1473] Step 8:

[1474] The server inputs the text data into an emotion engine to recognize the user's emotions, which identify emotions such as joy, anger, sadness, surprise, etc.

[1475] Step 9:

[1476] The server reevaluates the scope classification based on the analysis results together with the emotional information, and updates the optimal scope classification, thereby achieving highly accurate classification that includes emotional information.

[1477] Step 10:

[1478] The server stores the data classified based on the scope in a database or storage device. For example, data about a family member is stored in a household database.

[1479] Step 11:

[1480] The server sends the stored data to the appropriate output device. The notification content is adjusted based on the scope classification and emotion information. For example, if the user is feeling stressed, the notification content is simplified.

[1481] Step 12:

[1482] The user provides feedback on the system's output, for example, by sending feedback on scope classification or emotion recognition.

[1483] Step 13:

[1484] The server receives user feedback and retrains the machine learning model, improving classification and emotion recognition accuracy.

[1485] Example 2

[1486] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1487] In modern society, users have diverse conversations, and there is a need for automatic identification and appropriate classification. However, conventional technologies have faced challenges, such as insufficient accuracy in classifying conversation content and difficulty in managing information taking user emotions into account. It is also not easy to improve the accuracy of the system by reflecting user feedback.

[1488] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1489] In this invention, the server includes an input device for acquiring user voice or text data, a communication device for transmitting the data acquired by the input device to the server, a device for preprocessing the acquired data in the server device, performing noise reduction and text conversion, a device for analyzing the converted text data and performing keyword extraction and context analysis, a device for classifying the conversation content into a specific scope based on the analysis results, a device for recognizing the user's emotions using an emotion engine, a device for saving the classified data based on the emotion information and scope and sending a notification to an appropriate output device, and a device for receiving user feedback and training and improving the accuracy of the classification device. This enables automatic identification and classification of diverse user conversation content with high accuracy, and further realizes information management and notification that takes user emotions into consideration. Furthermore, the accuracy of the system can be improved by reflecting user feedback.

[1490] "Input device means" refers to a device for obtaining voice or text data from a user.

[1491] The "communication means" refers to a means for transmitting data acquired by the input device means to the server device.

[1492] The "noise removal means" refers to a means for removing noise from the voice data acquired in the server device.

[1493] The "text conversion means" refers to a means that provides a function for converting noise-removed voice data into text data.

[1494] "Keyword extraction means" refers to means for extracting important keywords from text data.

[1495] The "context analysis means" refers to a means for analyzing the context of text data.

[1496] The "scope classification means" refers to a means for classifying the conversation content into a specific scope based on the analysis results.

[1497] "Emotion recognition means" refers to means for recognizing the user's emotions using an emotion engine.

[1498] "Output device" refers to a device for notifying appropriate information based on classification and emotion recognition results.

[1499] "Data storage means" refers to a means for storing classified conversation data.

[1500] "Feedback receiving means" refers to a means for receiving feedback provided by a user.

[1501] "Training refinement means" refers to means for training and refining a classifier based on received feedback.

[1502] This invention is a system that automatically identifies diverse conversational content of users and classifies it into appropriate scopes. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, more advanced analysis and classification can be achieved. Detailed embodiments of the program for this system are described below.

[1503] Hardware and software used

[1504] Input device: A device that allows a user to input voice or text data. A microphone is used for voice input, and a keyboard is used for text input.

[1505] Communication method: A network communication method for sending data captured by an input device to a server. This includes Wi-Fi and wired LAN.

[1506] Server: A central processing unit for data pre-processing, speech recognition, natural language processing analysis, and emotion recognition. Software used includes the Google Speech-to-Text API, BERT model, and IBM Watson Tone Analyzer.

[1507] Data processing and calculation

[1508] 1. Preprocessing of audio data

[1509] The server receives the audio data sent from the terminal and applies a noise reduction algorithm (e.g., Spectral Subtraction) to generate clear audio data.

[1510] 2. Speech Recognition and Text Conversion

[1511] The server uses the Google Speech-to-Text API to convert the noise-removed audio data into text data. Specifically, the audio data is converted into text that says, "I'd like to talk about today's meeting."

[1512] 3. Analysis using natural language processing

[1513] The server uses the BERT model to extract keywords from the generated text data and perform contextual analysis, thereby understanding the meaning and importance of the text data.

[1514] 4. Scope Classification

[1515] Based on the analysis results, the server classifies the text data into specific scopes such as "family," "bosses," and "friends."

[1516] 5. Emotion recognition

[1517] The server uses IBM Watson Tone Analyzer to recognize the user's emotions (happiness, anger, sadness, surprise, etc.), and based on this emotional information, the accuracy of scope classification is further improved.

[1518] 6. Information Management and Notification

[1519] The server stores the classified data in a database and sends appropriate notifications to output devices (e.g., smartphones, PCs, digital displays, etc.) The content of the notifications is adjusted taking into account emotional information.

[1520] 7. Feedback and Learning

[1521] The user provides feedback on the system's output, for example, telling the system, "This notification was inappropriate."

[1522] The server receives user feedback and updates the machine learning model, which improves the accuracy of classification and emotion recognition in future.

[1523] Specific use cases

[1524] Scenario 1: Conversation with family

[1525] 1. A user is sitting in the living room and says, "I want to watch a movie with my family today."

[1526] 2. The device captures this conversation and sends it to the server.

[1527] 3. The server removes noise from the audio data and converts it to text using the Google Speech-to-Text API.

[1528] 4. The server analyzes the text using the BERT model and identifies the scope "family".

[1529] 5. The server uses IBM Watson Tone Analyzer to recognize the user's emotion as "joy."

[1530] 6. The server displays movie information for the family on the living room display.

[1531] Scenario 2: A conversation with your boss

[1532] 1. A user is in the office and says, "I'd like to talk to my boss about the progress of the project."

[1533] 2. The device captures the conversation and sends it to the server.

[1534] 3. The server removes noise from the audio data and converts it to text using the Google Speech-to-Text API.

[1535] 4. The server analyzes the text using the BERT model and identifies the scope "boss."

[1536] 5. The server uses IBM Watson Tone Analyzer to recognize the user's emotion as "stress."

[1537] 6. The server sends simplified project progress notifications to your work PC or smartphone.

[1538] This series of processes enables the system to automatically identify diverse user conversation content and classify it into the appropriate scope. Furthermore, by recognizing the user's emotions, more accurate classification and notifications are achieved. As a result, the user experience is improved, and by appropriately incorporating feedback, the system continues to learn and improve its accuracy.

[1539] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1540] Step 1:

[1541] A user starts a conversation. A user starts a voice conversation in their living room or office. For example, they might say, "I want to watch a movie with my family today." The input is real-time voice data. Specifically, the user speaks into a microphone, and voice data is captured.

[1542] Step 2:

[1543] The device captures voice data in real time and sends it to the server. In this case, the microphone functions as an input device and sends the captured voice data to the server. The input is voice data, and the output is digital voice data sent to the server. Specifically, the device sends the data via Wi-Fi or wired LAN.

[1544] Step 3:

[1545] The server removes noise from the received audio data. It applies a noise removal algorithm (e.g., Spectral Subtraction) to generate clear audio data. The input is the audio data sent from the device, and the output is clear audio data with noise removed. Specifically, the server executes the noise removal module.

[1546] Step 4:

[1547] The server converts the noise-removed audio data into text data. The Google Speech-to-Text API is used to convert the audio data into text. The input is the noise-removed audio data, and the output is text data. Specifically, the server inputs the audio data into the Google Speech-to-Text API and stores the resulting text data.

[1548] Step 5:

[1549] The server analyzes the text data and performs keyword extraction and context analysis. It uses the BERT model to extract keywords from the text data and analyze the context. The input is the text data, and the output is the extracted keywords and context information. Specifically, the server runs the BERT model and stores the analysis results.

[1550] Step 6:

[1551] The server classifies the conversation content into a specific scope based on the analysis results. For example, it may classify the content into scopes such as "family," "boss," or "friends." The input is keywords and context information, and the output is scope information. Specifically, the server applies a classification algorithm to identify the scope.

[1552] Step 7:

[1553] The server uses an emotion engine to recognize the user's emotions. The IBM Watson Tone Analyzer is used to recognize the user's emotions from text data. The input is text data and the output is emotional information. Specifically, the server inputs text data into the IBM Watson Tone Analyzer and stores the obtained emotional information.

[1554] Step 8:

[1555] The server stores the data classified based on emotion information and scope and sends a notification to the appropriate output device. The input is emotion information and scope information, and the output is data stored in a database and a notification sent. Specific operations include the server storing the data in a database and sending a notification to an output device (e.g., smartphone, PC, digital display).

[1556] Step 9:

[1557] The user provides feedback on the system's output. For example, the user tells the system, "This notification was inappropriate." The input is the user's feedback, and the output is the feedback data sent to the server. The specific action is for the user to provide feedback in text or voice.

[1558] Step 10:

[1559] The server receives feedback from users and retrains the machine learning model based on that feedback. This improves the overall classification accuracy, including the emotion engine. The input is the feedback data, and the output is the updated model. Specifically, the server runs the retraining algorithm to update the model.

[1560] (Application example 2)

[1561] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1562] Conventional food delivery services do not provide personalized suggestions based on the user's conversation content or emotions, limiting the user experience. Furthermore, there is a need for a system that can provide optimal suggestions tailored to specific situations. This creates challenges such as a lack of improvement in user satisfaction and personalization of the service.

[1563] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1564] In this invention, the server includes an input device means for acquiring user voice or text data, a means for preprocessing the data acquired by the input device means to perform noise removal and text conversion, and a means for analyzing the converted text data to perform keyword extraction and context analysis, thereby enabling optimal food delivery service suggestions to be made in accordance with special circumstances and the user's emotions.

[1565] The "input device means" is a device for acquiring voice or text data from a user.

[1566] The "communication means" is a system for transmitting data acquired by the input device means to the server device.

[1567] The "preprocessing means" is a process for removing noise and converting the acquired data into text in the server device.

[1568] The "analysis means" is a function that analyzes the data converted into text, extracts keywords, and performs context analysis.

[1569] The "scope classification means" is a process for classifying the conversation content into a specific scope based on the analysis results.

[1570] The "food delivery proposal method" is a process for proposing appropriate food delivery services based on the classified scope.

[1571] The "emotion recognition means" is a process that recognizes the user's emotions and selects the optimal output according to the scope and emotion.

[1572] The "notification means" is a process for notifying data classified according to scope and emotion to an appropriate output device.

[1573] A "feedback mechanism" is a process that receives user feedback to train and refine the classifier.

[1574] The present invention is a system that analyzes the various conversational content and emotions of users and optimizes food delivery services accordingly. Specific embodiments for carrying out the invention are described below.

[1575] Hardware and Software Configuration

[1576] 1. Input Device Means

[1577] The user inputs voice using the smartphone's microphone, and the input voice data is sent to the server via the smartphone's built-in communication means.

[1578] 2. Means of communication

[1579] The smartphone uses Wi-Fi or mobile data to transmit the captured data to a server in real time.

[1580] 3. Pretreatment Methods

[1581] The server first denoises the captured audio data to produce clear audio, using the Python speech_recognition library and a speech recognition model from the Transformers package.

[1582] 4. Analysis method

[1583] The server converts the audio data into text data and analyzes the text data, including using natural language processing techniques to extract keywords and perform context analysis, such as using TfidfVectorizer and KMeans clustering to classify the scope of the conversation.

[1584] 5. Scope classification and proposal methods

[1585] The server classifies the user's conversation content into a specific scope (e.g., family, friends, boss) and suggests appropriate food delivery services based on the scope. This also takes into account the user's emotional information. For emotion recognition, it uses the Transformers emotion recognition model.

[1586] 6. Emotion recognition means

[1587] The server analyzes the text data to recognize the user's emotions, categorizing them as joy, surprise, anger, etc., and then suggests the most appropriate service accordingly.

[1588] 7. Means of notification

[1589] The server stores the classified data according to scope and emotion and sends notifications to the appropriate output device, which can be displayed on the smartphone display or in the notification bar.

[1590] 8. Feedback channels

[1591] The user provides feedback on the system's output, such as whether the suggestions were appropriate. The server receives this feedback and uses it to improve the accuracy of the classification and emotion recognition methods mentioned above.

[1592] Specific use cases

[1593] Example 1: When a user is having a pleasant conversation with their family

[1594] Prompt: "What are your menu recommendations?"

[1595] Output: "Here are our recommended menu items!"

[1596] Example 2: When the user is stressed

[1597] Prompt: "I'm tired."

[1598] Output: "How about some cold udon noodles today?"

[1599] This makes it possible to provide an optimal food delivery experience based on the user's conversation content and emotions. This system not only improves the user experience, but also significantly improves the accuracy and convenience of food delivery services.

[1600] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1601] Step 1:

[1602] The user inputs voice using the smartphone's microphone, and the voice data is stored in the smartphone's internal memory.

[1603] Step 2:

[1604] The smartphone uses its built-in communication means to transmit the audio data to the server in real time, using Wi-Fi or mobile data.

[1605] Step 3:

[1606] The server pre-processes the received audio data using a noise reduction algorithm, resulting in clear audio data. The denoised audio data is then input to the next step.

[1607] Step 4:

[1608] The server converts the preprocessed audio data into text using a speech recognition model, using the Python speech_recognition library and the Transformers package, and the converted text data becomes the input for the next step.

[1609] Step 5:

[1610] The server analyzes the text data using natural language processing techniques. Specifically, it uses TfidfVectorizer and KMeans clustering to extract keywords and perform context analysis, and classifies the conversation content into specific scopes. The analyzed scope information becomes the input data for the next step.

[1611] Step 6:

[1612] The server generates optimal food delivery service recommendations based on the scope. It uses an emotion recognition model to evaluate the user's emotions, such as daigo / bert-base-japanese-sentiment, and generates recommendations that incorporate the user's emotional information.

[1613] Step 7:

[1614] The server stores the data classified according to scope and emotion and sends a notification to the appropriate output device (smartphone display or notification bar). For example, messages such as "Here are our recommended menu items!" or "How about some cold udon noodles today?" are displayed.

[1615] Step 8:

[1616] The user provides feedback on the system's suggestions and outputs, which are then sent from the smartphone to the server.

[1617] Step 9:

[1618] The server uses the feedback it receives to retrain its machine learning models (classification and emotion recognition) and improve the overall accuracy of the system, resulting in better suggestions for future use.

[1619] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1620] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1621] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1622] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1623] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1624] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1625] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1626] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1627] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1628] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1629] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1630] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1631] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1632] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1633] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1634] The hardware resource for executing a specific process can be any of the following types of processors: A processor, for example, is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1635] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1636] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1637] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1638] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1639] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1640] The following is further disclosed regarding the above embodiment.

[1641] (Claim 1)

[1642] an input device means for acquiring user voice or text data;

[1643] a communication means for transmitting data acquired by the input device means to a server device;

[1644] A means for preprocessing the acquired data in the server device to perform noise removal and text conversion;

[1645] A means for analyzing the data converted into text and performing keyword extraction and context analysis;

[1646] A means for classifying conversation content into specific scopes based on the analysis results;

[1647] means for storing the classified data based on scope and sending a notification to an appropriate output device;

[1648] means for receiving user feedback to train and refine said classifier;

[1649] A system including:

[1650] (Claim 2)

[1651] 2. The system according to claim 1, wherein the analyzing means extracts keywords and contexts using natural language processing techniques.

[1652] (Claim 3)

[1653] 10. The system of claim 1, wherein the output device displays or notifies different information to the user in different environments.

[1654] "Example 1"

[1655] (Claim 1)

[1656] an input device means for acquiring user voice or text data;

[1657] a communication means for transmitting data acquired by the input device means to a server device;

[1658] A means for preprocessing the acquired data in the server device to perform noise removal and text conversion;

[1659] A means for analyzing the data converted into text and performing keyword extraction and context analysis;

[1660] A means for classifying conversation content into specific scopes based on the analysis results;

[1661] means for storing the classified data based on scope and sending notifications to appropriate display devices;

[1662] means for receiving user feedback to train and refine said classifier;

[1663] a means for analyzing a user input prompt using a generative AI model to provide an optimal response;

[1664] A means to select the appropriate notification format for different scopes;

[1665] A system including:

[1666] (Claim 2)

[1667] 2. The system according to claim 1, wherein the analyzing means extracts keywords and contexts using natural language processing techniques.

[1668] (Claim 3)

[1669] 2. The system according to claim 1, wherein the display device displays or notifies different information depending on different usage environments of the user.

[1670] "Application Example 1"

[1671] (Claim 1)

[1672] an input device means for acquiring user voice or text data;

[1673] a communication means for transmitting data acquired by the input device means to a server device;

[1674] A means for preprocessing the acquired data in the server device to perform noise removal and text conversion;

[1675] A means for analyzing the data converted into text and performing keyword extraction and context analysis;

[1676] A means for classifying conversation content into specific scopes based on the analysis results;

[1677] means for storing the classified data based on scope and sending a notification to an appropriate output device;

[1678] means for receiving user feedback to train and refine said classifier;

[1679] a means for providing destination information and route change information based on the passenger's conversation;

[1680] A system including:

[1681] (Claim 2)

[1682] 2. The system according to claim 1, wherein the analyzing means extracts keywords and contexts using natural language processing techniques.

[1683] (Claim 3)

[1684] 10. The system of claim 1, wherein said output device provides navigation information and recommended routes based on different passenger requirements.

[1685] "Example 2: Combining Emotion Engines"

[1686] (Claim 1)

[1687] an input device means for acquiring user voice or text data;

[1688] a communication means for transmitting data acquired by the input device means to a server device;

[1689] A means for preprocessing the acquired data in the server device to perform noise removal and text conversion;

[1690] A means for analyzing the data converted into text and performing keyword extraction and context analysis;

[1691] A means for classifying conversation content into specific scopes based on the analysis results;

[1692] means for recognizing a user's emotion using an emotion engine;

[1693] means for storing the classified data based on the emotion information and scope and sending a notification to an appropriate output device;

[1694] means for receiving user feedback to train and refine said classifier;

[1695] A system including:

[1696] (Claim 2)

[1697] 2. The system according to claim 1, wherein the analyzing means extracts keywords and contexts using natural language processing techniques.

[1698] (Claim 3)

[1699] 2. The system of claim 1, wherein the output device displays or notifies different information in different environments of the user, and the notification content is adjusted based on emotional information.

[1700] "Application example 2 when combining emotion engines"

[1701] (Claim 1)

[1702] an input device means for acquiring user voice or text data;

[1703] a communication means for transmitting data acquired by the input device means to a server device;

[1704] A means for preprocessing the acquired data in the server device to perform noise removal and text conversion;

[1705] A means for analyzing the data converted into text and performing keyword extraction and context analysis;

[1706] A means for classifying conversation content into specific scopes based on the analysis results;

[1707] A means for suggesting an appropriate food delivery service based on the classified scope;

[1708] A means for recognizing a user's emotion and selecting an optimal output according to the scope and emotion;

[1709] means for storing the classified data according to scope and emotion and sending a notification to an appropriate output device;

[1710] means for receiving user feedback to train and refine said classifier;

[1711] A system including:

[1712] (Claim 2)

[1713] 2. The system according to claim 1, wherein the analyzing means extracts keywords and contexts using natural language processing techniques.

[1714] (Claim 3)

[1715] 10. The system of claim 1, wherein the output device displays or notifies different food delivery information based on different emotional states and conversation scopes of the user. [Explanation of symbols]

[1716] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. an input device means for acquiring user voice or text data; a communication means for transmitting data acquired by the input device means to a server device; A means for preprocessing the acquired data in the server device to perform noise removal and text conversion; A means for analyzing the data converted into text and performing keyword extraction and context analysis; A means for classifying conversation content into specific scopes based on the analysis results; means for storing the classified data based on scope and sending a notification to an appropriate output device; means for receiving user feedback to train and refine said classifier; A system including:

2. 2. The system of claim 1, wherein said analyzing means extracts keywords and contexts using natural language processing techniques.

3. 2. The system of claim 1, wherein said output device displays or notifies different information to a user in different environments.

Images