System

Abstract

A system is provided.SOLUTION: A system comprising: means for processing user input using a generative model; means for receiving speech input from a user and converting the speech input to string data; means for sending the string data to the generative model and obtaining a generated response; and means for converting the obtained response to speech and presenting the speech to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional car navigation systems have basic communication functions such as music playback and phone calls, but they lack interactive features, preventing users from fully utilizing the system while driving. Furthermore, they often lack adequate support when users ask abstract questions or perform complex operations. Furthermore, when a vehicle malfunction occurs, there is a lack of means to quickly provide users with specific troubleshooting methods. It is necessary to solve these issues and improve the convenience of car navigation systems. [Means for solving the problem]

[0005] The present invention provides a system that includes a means for processing user input using a generative model, a means for receiving voice input from the user and converting it into text data, a means for sending the text data to the generative model and obtaining a generated response, and a means for converting the obtained response into text and presenting it to the user. The generative model also includes a means for monitoring the vehicle's condition and, if an abnormality is detected, presenting specific countermeasures. Furthermore, the system includes a means for presenting multiple appropriate options in response to the user's abstract request. This allows the user to receive interactive and prompt support while driving, dramatically improving the convenience of car navigation systems.

[0006] A "generative model" is an artificial intelligence technique that uses natural language processing to generate appropriate responses to user input.

[0007] "User input" refers to voice instructions or questions that a user provides to a car navigation system.

[0008] "String data" refers to data that has been converted from user input into text format.

[0009] "Voice input" refers to voice information spoken by a user to a car navigation system.

[0010] "Means for converting to speech" refers to speech synthesis technology for generating speech from text.

[0011] "Means for monitoring the condition of the vehicle" refers to a system that collects data from various sensors inside the vehicle and checks for any abnormalities or problems.

[0012] "Means for detecting abnormalities" refers to technology that analyzes data obtained from vehicles and detects abnormal conditions or malfunctions.

[0013] "Means for presenting specific countermeasures" refers to a function that provides specific actions and procedures that users should take in response to detected abnormalities.

[0014] "Means for presenting multiple appropriate options" refers to the function of generating and presenting multiple answers or options when a user makes an abstract request. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The present invention relates to a next-generation car navigation system that processes user input using a generative model. The following embodiments will explain how to specifically implement the system of the present invention.

[0037] 1. A way to capture user voice input

[0038] When a user speaks to the car navigation system, the device uses a microphone to record the voice. Here, voice input is acquired by the user speaking instructions, questions, requests, etc. to the car navigation system.

[0039] 2. A method for converting audio into text data

[0040] The device uses a voice recognition function to convert the recorded voice data into text data, using natural language processing technology to convert the spoken voice into appropriate text data.

[0041] 3. A means of sending user input to the generative model

[0042] The device sends the converted text data to the server. The server then inputs the received text data into a generative model to generate an appropriate response. A generative model is an artificial intelligence technology that uses natural language processing to generate responses to user input.

[0043] 4. How to retrieve the generated response

[0044] The server retrieves the response generated by the generative model and sends the corresponding text data back to the device, including the information requested by the user and notifications about the vehicle's status.

[0045] 5. A means of converting the response into speech and presenting it to the user

[0046] The device converts the text data received from the server into speech using a speech synthesis engine, and presents the converted speech response to the user, conveying information in a form that is easy for the user to understand.

[0047] Example: Navigation request

[0048] When a user speaks to a car navigation system saying, "Tell me where the nearest gas station is," the device's microphone records the speech and converts it into text data saying, "Tell me where the nearest gas station is" using speech recognition. This data is sent to the server, which processes the generative model to generate a response saying, "The nearest gas station is ____. It will take ____ minutes to get there," and sends it back to the device. The device then converts this response into speech and provides the user with audible guidance saying, "The nearest gas station is ____. It will take ____ minutes to get there."

[0049] Example: Notification of vehicle abnormality and instructions on how to deal with it

[0050] When the vehicle detects an abnormality, such as a notification that "tire pressure is low," the device sends that information to the server. The server uses the generative model to generate specific instructions, such as "Go to the nearest service station to check your tire pressure. You can check it by following the steps below...," and sends them back to the device. The device then converts these instructions into voice and provides the user with quick and specific instructions on how to deal with the problem.

[0051] In this way, the car navigation system of the present invention can provide interactive and advanced support compared to conventional systems, and can significantly improve user convenience.

[0052] The processing flow will be explained below.

[0053] Step 1:

[0054] The user speaks to the car navigation system. The user issues a voice request, such as "Tell me where the nearest gas station is."

[0055] Step 2:

[0056] The device records the user's voice. The device's microphone picks up what the user says and stores it as digital audio data.

[0057] Step 3:

[0058] The device converts the voice data into text data. The device uses a speech recognition engine to convert the recorded voice data into text. This text is in the format "Where is the nearest gas station?"

[0059] Step 4:

[0060] The terminal sends the string data to the server. The terminal then sends the converted text data to the server as an HTTP POST request.

[0061] Step 5:

[0062] The server receives user input. The server receives the string data sent from the device and prepares it to be passed to the generative model.

[0063] Step 6:

[0064] The server inputs user input into the generative model. The server inputs user text data into the generative model (e.g., a natural language processing model) and instructs it to generate an appropriate response.

[0065] Step 7:

[0066] The server receives the generated response. The generative model generates a response such as "The nearest gas station is ____. It will take ____ minutes to get there," and the server receives this.

[0067] Step 8:

[0068] The server sends a response back to the device. The server sends the generated text response back to the device in JSON format.

[0069] Step 9:

[0070] The device receives the response. The device receives the text response sent by the server and prepares to pass it to the speech synthesis engine.

[0071] Step 10:

[0072] The device converts the text response into speech. The device uses a speech synthesis engine to convert the text data into speech data.

[0073] Step 11:

[0074] The device presents a voice response to the user. The device plays the generated voice data through a speaker, informing the user, "The nearest gas station is ____. It will take ____ minutes to get there."

[0075] Example 1

[0076] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0077] Conventional car navigation systems have limitations in their ability to provide immediate responses to user voice input, making it difficult to quickly provide specific solutions to complex requests or vehicle abnormalities. In particular, when a user makes a request through voice input, it is important to accurately understand the request and provide appropriate information and instructions. However, existing technologies lack sufficient interactivity, resulting in a loss of user convenience.

[0078] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0079] In this invention, the server includes means for acquiring user input as voice, means for converting the acquired voice into character string data, means for transmitting the converted character string data to the generative model, means for acquiring a response generated by the generative model, and means for converting the acquired response into voice and presenting it to the user. This makes it possible to provide a response to a request made by the user through voice input with high accuracy and speed. It is also possible to detect vehicle abnormalities in real time, generate specific countermeasures, and provide guidance to the user. This allows the user to operate the vehicle safely and smoothly, improving convenience.

[0080] "Means for capturing user input as audio" refers to devices and techniques for recording a user's spoken instructions or questions.

[0081] "Means for converting acquired audio into character string data" refers to the function of converting recorded audio data into text data using natural language processing technology.

[0082] "Means for sending converted string data to a generative model" refers to a function for sending text data via a network to a generative model on a server or cloud.

[0083] "Means for obtaining responses generated by a generative model" refers to the function of receiving response data provided by a generative AI model and storing it within the system.

[0084] "Means for converting the acquired response into speech and presenting it to the user" refers to a function for converting a response acquired as text data into speech using speech synthesis technology and allowing the user to hear it.

[0085] "Means for detecting vehicle abnormalities" refers to the function of using sensors and monitoring systems inside the vehicle to detect whether there are any abnormalities in each part or operating condition of the vehicle.

[0086] "Means for presenting specific countermeasures" refers to a function that generates specific actions and procedures that the user should take in response to a detected vehicle abnormality and provides that information to the user.

[0087] "Means for transmitting to a cloud server" refers to a communication function for transmitting data to a remote server via the Internet.

[0088] MODE FOR CARRYING OUT THE INVENTION

[0089] The present invention relates to a next-generation car navigation system that processes user input using a generative model. Specific embodiments of the present invention are described below.

[0090] 1. Voice input acquisition

[0091] When a user issues voice commands to the car navigation system, the device records the voice using its built-in microphone. The recorded voice data is temporarily stored in the device's memory. This step uses a standard microphone and voice recording software.

[0092] example:

[0093] When a user says, "Tell me where the nearest gas station is," the audio is recorded.

[0094] 2. Convert audio to text data

[0095] The recorded voice data is converted into text data using natural language processing technology. Specifically, speech recognition software is used. In this example, the "Google (registered trademark) Cloud Speech-to-Text" API is used.

[0096] example:

[0097] The recorded voice saying "Tell me where the nearest gas station is" is converted into text data saying "Tell me where the nearest gas station is."

[0098] 3. Send user input to the generative model

[0099] The converted text data is sent from the device to a cloud server. The cloud server receives this text data and inputs it into a generative AI model. This generative AI model generates an appropriate response based on the user input. Data is sent and received using standard cloud communication technologies.

[0100] example:

[0101] The data sent from the device, "Tell me where the nearest gas station is," reaches the cloud server and is input as a prompt into the generative AI model.

[0102] 4. Retrieving the generated response

[0103] The generative AI model generates a response to user input, and the data is received by the cloud server. This data is then sent back to the device and stored locally. Examples of generative AI models include "OpenAI (registered trademark) GPT-4 (registered trademark)."

[0104] example:

[0105] The generative AI model generates a response such as, "The nearest gas station is XX. It will take XX minutes to get there," and sends it to the device via the cloud server.

[0106] 5. Convert the response into audio and present it

[0107] The device converts the text data received from the server into speech using a speech synthesis engine, such as the Amazon Polly API, and presents the speech data to the user through the speaker.

[0108] example:

[0109] The device converts the received text data, such as "The nearest gas station is XX. It will take XX minutes to get there," into speech and plays it back to the user.

[0110] Specific examples

[0111] For navigation requests

[0112] User: "Where's the nearest gas station?"

[0113] Device: Records voice and uses voice recognition technology to convert it into text data such as "Tell me where the nearest gas station is."

[0114] Server: Input the text data into the generative AI model and obtain the generated response, "The nearest gas station is XX. It will take XX minutes to arrive."

[0115] Terminal: Converts the response into speech and presents it to the user.

[0116] Notification of vehicle abnormality and instructions on how to deal with it

[0117] Vehicle: Detects abnormalities such as low tire pressure.

[0118] Terminal: Sends abnormality information to the cloud server.

[0119] Server: Using a generative AI model, it generates a response such as, "Go to the nearest service station to check your tire pressure. You can do this by following these steps..."

[0120] Terminal: Converts the generated response into speech and presents it to the user.

[0121] In this way, the car navigation system of the present invention can significantly improve user convenience by responding quickly and accurately to user voice inputs and dealing appropriately with vehicle abnormalities.

[0122] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0123] Step 1: User speaks

[0124] Specific explanation and operation

[0125] The user issues commands or questions to the car navigation system, for example, "Tell me where the nearest gas station is."

[0126] Input: User's voice

[0127] Output: Recorded audio data (e.g. audio_input.wav)

[0128] The user speaks to the car navigation system, and the voice is recorded through a microphone.

[0129] Step 2: The device converts the audio into text data

[0130] Specific explanation and operation

[0131] The device converts the recorded voice data into text data using natural language processing technology.

[0132] Software used: Speech recognition software (e.g., "Google Cloud Speech-to-Text")

[0133] Input: Recorded audio data (audio_input.wav)

[0134] Output: Text data "Where is the nearest gas station?"

[0135] Processing: The device sends the audio_input.wav file to the speech recognition API and receives the text data.

[0136] Step 3: The device sends the string data to the generative model

[0137] Specific explanation and operation

[0138] The device sends the converted text data to a cloud server, which then inputs the text data into a generative AI model.

[0139] Input: Text data "Tell me the nearest gas station"

[0140] Output: The prompt sentence that is input to the generative model

[0141] Processing: The device sends text data to the server via an HTTP request, and the server receives the data and passes it to the generative AI model.

[0142] Step 4: The server gets the generated response

[0143] Specific explanation and operation

[0144] The generative AI model generates an appropriate response based on the user's prompt. The server retrieves the generated response data and sends it back to the device.

[0145] Software used: Generative AI models (e.g., "OpenAI GPT-4")

[0146] Input: Prompt "Where is the nearest gas station?"

[0147] Output: Text data "The nearest gas station is ____. It will take ____ minutes to get there."

[0148] Processing: The server receives the response from the generative AI model and sends the data to the device.

[0149] Step 5: The device converts the response into speech and presents it to the user.

[0150] Specific explanation and operation

[0151] The terminal converts the received text data into speech using a speech synthesis engine and presents it to the user.

[0152] Software used: Speech synthesis engine (e.g., "Amazon Polly")

[0153] Input: Text data: "The nearest gas station is ____. It will take ____ minutes to get there."

[0154] Output: Speech data: "The nearest gas station is ____. It will take ____ minutes to get there."

[0155] Processing: The device sends the text data to the speech synthesis API, and the generated speech data is played on the speaker.

[0156] This series of processing steps allows the user to operate the car navigation system through voice and receive prompt and accurate responses.

[0157] (Application example 1)

[0158] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0159] Conventional car navigation systems have limited ability to respond to user instructions and questions, resulting in a lack of interactivity. Furthermore, they lack the ability to provide specific solutions when vehicle abnormalities are detected. Furthermore, they lack the functionality to present multiple options in response to abstract user requests. To address these issues, the development of more advanced, user-friendly navigation systems is needed.

[0160] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0161] In this invention, the server includes means for collecting user input via voice and converting it into text data, means for sending the collected text data to a generative model and obtaining a text response returned from the generative model, means for converting the obtained text response into voice and presenting it to the user, and means for generating responses to user instructions and questions using the generative model and providing vehicle status confirmation, route guidance, and real-time traffic information. This allows the user to enjoy advanced and interactive navigation, as well as to obtain quick and specific countermeasures in the event of a vehicle abnormality, and further allows the user to receive multiple options even for abstract requests.

[0162] "User input" refers to information such as voice instructions or questions that a user gives to the system.

[0163] "Text data" refers to data that has been analyzed and converted into a character string format from voice input.

[0164] A "generative model" is an artificial intelligence technique used to generate appropriate responses to user input.

[0165] A "server" is a computer system that operates a generative model and generates and provides responses to user input.

[0166] "Vehicle status check" refers to monitoring and evaluating the current condition of a vehicle based on data obtained from various sensors and systems on the vehicle.

[0167] "Route guidance" refers to calculating and presenting the optimal route for a user to reach their destination.

[0168] "Real-time traffic information" means providing information on current road conditions, traffic volume, etc. immediately.

[0169] The "voice collection means" refers to a device or technology that acquires the user's voice using a microphone or the like.

[0170] "Text conversion means" refers to software or algorithms for converting acquired voice data into character string data.

[0171] A "response acquisition means" refers to a communication means or protocol for receiving text responses generated from a generative model.

[0172] "Speech conversion means" refers to technology or devices for converting text responses from a generative model into speech data.

[0173] An "abstract request" is a vague question or instruction that does not include specific information or instructions from the user.

[0174] This invention relates to a next-generation car navigation system that processes user input using generative models. When a user issues a command or question by voice, the system recognizes the voice, generates a response using a generative model, and provides it to the user as voice.

[0175] The system has the following configuration:

[0176] Hardware and software configuration

[0177] 1. Voice collection means: The user's voice is acquired using a microphone in the vehicle.

[0178] 2. Text conversion method: Use speech recognition software (e.g., speech_recognition library) to convert the captured audio into text data.

[0179] 3. Means for transmitting text data: A communication protocol is used to transmit text data to the generative model.

[0180] 4. Response generator: A generative model (e.g., the transformers library) responsible for natural language processing generates an appropriate text response.

[0181] 5. Speech conversion means: Generates voice data using a speech synthesis engine (e.g., pyttsx3 library) that converts the generated text response into speech and presents it to the user.

[0182] Program processing

[0183] The server constructs a system including the following means.

[0184] 1. Collecting user voice input: The device captures the user's voice through the microphone and saves it as voice data.

[0185] 2. Speech-to-text conversion: The device's speech recognition software converts this speech data into text data.

[0186] 3. Sending data to the generative model: The string data is sent to the server, which then inputs the data into the generative model.

[0187] 4. Obtaining and sending the response: The server obtains the response generated by the generative model and sends the text response back to the device.

[0188] 5. Generating and presenting a voice response: The device converts the text data into voice data using a voice synthesis engine and presents it to the user.

[0189] For example, if a user says "Where is the nearest restaurant?" while in a vehicle, the system recognizes the speech and converts it into text data. The converted string data is sent to the server, and the generative model generates a response such as "The nearest restaurant is ____. Travel time is ____ minutes." This generated text response is sent back from the server to the device, where it is converted into speech by a speech synthesis engine and presented to the user.

[0190] For example, the prompt text might look like this:

[0191] Prompt statement:

[0192] User: Where is the nearest restaurant?

[0193] Generative AI model input sentence: Where is the nearest restaurant?

[0194] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0195] Step 1:

[0196] Acquiring voice input

[0197] The device uses a microphone to collect the user's voice. When the user speaks in the vehicle, "Where is the nearest restaurant?", this voice is captured by the microphone as input data. The input is the user's voice data, and this voice data is passed to the device.

[0198] Step 2:

[0199] Converting audio data to text

[0200] The device processes the input voice data using voice recognition software (for example, the speech_recognition library) and converts the voice into text data. This voice recognition process converts the voice data into text data such as "Where is the nearest restaurant?" The output is string data.

[0201] Step 3:

[0202] Sending text data

[0203] The terminal sends the converted text data to the server. In order for the terminal to send the text data "Where is the nearest restaurant?" to the server, it transfers the data using a communication protocol. The input is text data, which is sent to the server as output.

[0204] Step 4:

[0205] Generating a response

[0206] The server inputs the received text data into a generative model to generate an appropriate response. The generative model (e.g., the transformers library) generates the response "The nearest restaurant is ____. Travel time is ____ minutes" based on the input text data "Where is the nearest restaurant?". The output is the generated text response.

[0207] Step 5:

[0208] Sending a Response

[0209] The server sends the text response generated from the generative model to the terminal. The server then sends the generated text response, "The nearest restaurant is ____. Travel time is ____ minutes," to the terminal using a communication protocol. The input is the generated text response, which is sent to the terminal as output.

[0210] Step 6:

[0211] Generate a voice response

[0212] The device converts the text response received from the server into voice data using a speech synthesis engine (for example, the pyttsx3 library). The speech synthesis engine converts the text data, such as "The nearest restaurant is ____. Travel time is ____ minutes," into voice data and outputs it to the user through the speaker. The input is text data, and the output is voice data.

[0213] Step 7:

[0214] Voice response presentation

[0215] The device presents the generated voice data to the user through a speaker. The user receives a voice prompt saying, "The nearest restaurant is ____. Travel time is ____ minutes." The input is the voice data, and the output is the voice prompt presented to the user.

[0216] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0217] The present invention relates to an advanced car navigation system that combines a generative model and an emotion engine. The following embodiments will explain how to specifically implement the system of the present invention.

[0218] 1. A way to capture user voice input

[0219] When a user speaks to the car navigation system, the device uses a microphone to record the voice. Here, voice input is acquired by the user speaking instructions, questions, requests, etc. to the car navigation system.

[0220] 2. A method for converting audio into text data

[0221] The device uses a voice recognition function to convert the recorded voice data into text data, using natural language processing technology to convert the spoken voice into appropriate text data.

[0222] 3. A method for recognizing user emotions using an emotion engine

[0223] The device or server analyzes the text data obtained from the speech recognition and the speech itself to identify the user's emotions. The emotion engine uses algorithms to determine the user's emotional state, such as whether they are stressed or relaxed.

[0224] 4. A means of sending user input to the generative model

[0225] The device sends the converted text data and the recognized emotion data to the server, which then inputs the received data into a generative model to generate an appropriate response.

[0226] 5. How to obtain the generated response

[0227] The server retrieves the response generated by the generative model and sends the corresponding text data back to the device. This response can include information requested by the user or notifications about the vehicle's status. The content and tone of the response can also be adjusted based on emotion data.

[0228] 6. A means of converting the response into speech and presenting it to the user

[0229] The device converts the text data received from the server into speech using a speech synthesis engine, and presents the converted speech response to the user, conveying information in a form that is easy for the user to understand.

[0230] Example: Navigation request

[0231] When a user speaks to a car navigation system saying, "Tell me where the nearest gas station is," the device's microphone records the speech and converts it into text data saying, "Tell me where the nearest gas station is" using speech recognition. This data is sent to the server, which processes the generative model to generate a response saying, "The nearest gas station is ____. It will take ____ minutes to get there," and sends it back to the device. The device then converts this response into speech and provides the user with audible guidance saying, "The nearest gas station is ____. It will take ____ minutes to get there."

[0232] Example: Notification of vehicle abnormality and instructions on how to deal with it

[0233] When the vehicle detects an abnormality, such as a notification that "tire pressure is low," the device sends that information to the server. The server uses the generative model to generate specific instructions, such as "Go to the nearest service station to check your tire pressure. You can check it by following the steps below...," and sends them back to the device. The device then converts these instructions into voice and provides the user with quick and specific instructions on how to deal with the problem.

[0234] Example: Responding according to the user's emotions

[0235] The emotion engine recognizes when a user is tired or stressed. For example, if a user says, "The roads are crowded and I'm frustrated," the device records the audio and the emotion engine recognizes the emotion of frustration. The server uses the generative model to generate an encouraging response such as, "Don't worry. We're almost there. Relax and drive safely." The device then converts this response into voice and delivers it to the user in a gentle tone.

[0236] In this way, the car navigation system of the present invention provides more interactive and advanced support than conventional systems, significantly improving user convenience and satisfaction. The introduction of an emotion engine enables flexible responses according to the user's mental state, providing a more intimate and secure driving environment.

[0237] The processing flow will be explained below.

[0238] Step 1:

[0239] The user speaks to the car navigation system. The user issues a voice request, such as "Tell me where the nearest gas station is."

[0240] Step 2:

[0241] The device records the user's voice. The device's microphone picks up what the user says and stores it as digital audio data.

[0242] Step 3:

[0243] The device converts the voice data into text data. The device uses a speech recognition engine to convert the recorded voice data into text. This text is in the format "Where is the nearest gas station?"

[0244] Step 4:

[0245] The device passes the voice data to the emotion engine, which then inputs the recorded voice data into the emotion engine and analyzes the user's emotional state.

[0246] Step 5:

[0247] The emotion engine recognizes emotions. The emotion engine analyzes the voice or text data and identifies emotions such as whether the user is stressed or relaxed.

[0248] Step 6:

[0249] The device sends text data and emotion data to the server, which then sends the data to the server as an HTTP POST request.

[0250] Step 7:

[0251] The server receives user input and emotion data. The server receives the text data and emotion data sent from the device and prepares them to be passed to the generative model.

[0252] Step 8:

[0253] The server provides user input and emotion data to the generative model, and instructs it to generate an appropriate response.

[0254] Step 9:

[0255] The generative model generates a response based on user input and emotion. Based on the identified emotion, the generative model generates a response with appropriate tone and content. For example, "Please stay calm. The nearest gas station is ____."

[0256] Step 10:

[0257] The server receives the generated response. The server receives the response generated by the generative model and prepares to send it back to the device.

[0258] Step 11:

[0259] The server sends a response back to the device. The server sends the generated text response back to the device in JSON format.

[0260] Step 12:

[0261] The device receives the response. The device receives the text response sent by the server and prepares to pass it to the speech synthesis engine.

[0262] Step 13:

[0263] The device converts the text response into speech. The device uses a speech synthesis engine to convert the text data into speech data.

[0264] Step 14:

[0265] The device presents a voice response to the user. The device plays the generated voice data through a speaker, guiding the user in a tone that takes into consideration the user's emotions, such as "The nearest gas station is ____. It will take ____ minutes to get there."

[0266] Example 2

[0267] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0268] Conventional car navigation systems can only provide guidance and information based on simple voice input from the user, and have the problem of being unable to flexibly respond to the user's emotional state or vehicle abnormalities. Furthermore, they lack the functionality to present appropriate options in response to abstract user requests. This can cause stress for users and reduces satisfaction.

[0269] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0270] In this invention, the server includes means for using a generative model to process user input, means for receiving voice input from the user and converting it into character string data, means for recognizing the user's emotion using an emotion engine, means for adjusting the content and tone of a response based on the user's emotion, means for sending the character string data to the generative model and obtaining a generated response, and means for converting the obtained response into voice and presenting it to the user. This makes it possible to provide flexible responses according to the user's emotional state, suggest specific ways to deal with vehicle abnormalities, and even provide multiple appropriate options for abstract requests.

[0271] A "generative model" is a program with an algorithm that generates appropriate responses based on user input information.

[0272] An "emotion engine" is software or algorithms that analyze voice and text data to identify a user's emotional state.

[0273] "Voice input" refers to the voice that a user speaks to a device, which is acquired as data via a microphone.

[0274] "Character string data" refers to data that has been converted from voice input into text format using voice recognition technology.

[0275] A "speech synthesis engine" is software or algorithms that convert text data so that it can be played as speech.

[0276] A "server" is a central computing device that processes data sent by users and generates the necessary responses.

[0277] A "terminal" is a device that is directly used by a user or a device with a user interface, which receives voice input and outputs voice.

[0278] This invention shows how to specifically implement an advanced car navigation system that combines a generative model and an emotion engine. The components of the system and their operation are described in detail below.

[0279] 1. A way to capture user voice input

[0280] When a user speaks to the car navigation system in the car, the device records the voice using a microphone built into the car. Voice input is acquired when the user speaks instructions, questions, requests, etc. to the car navigation system. For example, if a user says, "Tell me where the nearest gas station is," the voice is recorded. A built-in microphone is required as hardware.

[0281] 2. A method for converting audio into text data

[0282] The device converts the recorded voice data into text data using speech recognition software (e.g., Google Cloud Speech-to-Text). Natural language processing technology is used to convert the spoken voice into text data. During this conversion process, the voice waveform is analyzed to generate the appropriate text data. For example, the speech "Tell me where the nearest gas station is" is converted into text data "Tell me where the nearest gas station is."

[0283] 3. A method for recognizing user emotions using an emotion engine

[0284] The terminal or server analyzes the character string data obtained from the speech recognition and the speech itself, and identifies the user's emotion using an emotion engine (e.g., IBM Watson (registered trademark) Tone Analyzer). The emotion engine uses an algorithm to determine the user's emotional state, such as whether they are stressed or relaxed. For example, if the user says, "The roads are crowded and I'm getting annoyed," the emotion engine will recognize the emotion as "annoyed."

[0285] 4. A means of sending user input to the generative model

[0286] The device sends the converted text data and the recognized emotion data to the server. The server inputs this data into a generative AI model (e.g., OpenAI GPT-4) to generate an appropriate response. For example, in response to a request such as "Tell me where the nearest gas station is," the generative AI model generates a response such as "The nearest gas station is ____. It will take ____ minutes to get there."

[0287] 5. How to obtain the generated response

[0288] The server receives the response generated by the generative model and sends it back to the device as text data. This response includes the information requested by the user and notifications about the vehicle's status. It is also possible to adjust the content and tone of the response based on emotional data. Specifically, the server generates a response such as "It's okay. We'll arrive at our destination soon. Relax and drive safely," and sends it to the device.

[0289] 6. A means of converting the response into speech and presenting it to the user

[0290] The device converts the text data received from the server into speech using a speech synthesis engine (e.g., Amazon Polly). The speech synthesis engine converts the input text into a speech signal and outputs it from the speaker. This process allows the user to hear the generated response as audio. For example, the user may hear voice guidance such as, "The nearest gas station is ____. It will take ____ minutes to arrive."

[0291] Specific examples

[0292] 1. For navigation requests:

[0293] When a user speaks to a car navigation system saying, "Tell me where the nearest gas station is," the device's microphone records the speech and converts it into text data such as "Tell me where the nearest gas station is" using speech recognition. This data is sent to the server, which processes it using a generative AI model to generate a response such as "The nearest gas station is ____. It will take ____ minutes to get there," which is sent back to the device. The device converts this response into speech and provides the user with audible guidance such as, "The nearest gas station is ____. It will take ____ minutes to get there."

[0294] 2. Vehicle abnormality notification and instructions:

[0295] When the vehicle detects an abnormality, such as a notification that "tire pressure is low," the device sends that information to the server. The server uses the generative model to generate specific instructions, such as "Go to the nearest service station to check your tire pressure. You can check it by following the steps below...," and sends them back to the device. The device then converts these instructions into voice and provides the user with quick and specific instructions on how to deal with the problem.

[0296] 3. Responding according to the user's emotions:

[0297] The emotion engine recognizes when a user is tired or stressed. For example, if a user says, "The roads are crowded and I'm frustrated," the device records the audio and the emotion engine recognizes the emotion of frustration. The server uses the generative model to generate an encouraging response: "Don't worry. We're almost there. Relax and drive safely." The device then converts this response into speech and delivers it to the user in a gentle tone.

[0298] In this way, the car navigation system of the present invention provides more interactive and advanced support than conventional systems, significantly improving user convenience and satisfaction. The introduction of an emotion engine enables flexible responses according to the user's mental state, providing a more intimate and secure driving environment.

[0299] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0300] Step 1: Capture the user's voice input

[0301] When a user speaks to the car navigation system in the car, the device uses a built-in microphone to record the voice. The input is the user's natural voice, which is captured by the device as digital voice data. For example, if a user says, "Tell me where the nearest gas station is," that voice is recorded. The data output is digital voice data.

[0302] Step 2: Convert audio to text

[0303] The device converts the recorded voice data into text data using voice recognition software (e.g., Google Cloud Speech-to-Text). The input voice data is analyzed and converted into text data using its waveform information. This process transforms the voice signal into text data such as "Tell me where the nearest gas station is." The output data is text data corresponding to the voice data.

[0304] Step 3: Recognize user emotions using the emotion engine

[0305] The device or server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to analyze the text data and audio data obtained from speech recognition. The input is text data or raw audio data, and the emotion engine determines the emotional state. For example, if a user says, "The roads are crowded and I'm getting annoyed," the emotion engine analyzes the text data and audio data and recognizes the emotion of annoyance. The output data is the identified emotional information.

[0306] Step 4: Sending user input to the generative model

[0307] The device sends the converted text data and the recognized emotion data to the server. The input is text data and emotion data, which are sent to the server. The server inputs this data into a generative AI model (e.g., OpenAI GPT-4) and uses it as a prompt to generate an appropriate response to the user's request. For example, based on the request "Tell me where the nearest gas station is," the response generated is "The nearest gas station is XX. It will take XX minutes to get there." The output data is the text data of the generated response.

[0308] Step 5: Get the generated response

[0309] The server receives the response generated by the generative AI model and returns it to the device as text data. The input is the string data generated by the generative AI model, and an appropriate response is constructed based on this. The output data is the response string data. For example, the server might send a response to the device saying, "The nearest gas station is ____. It will take ____ minutes to arrive."

[0310] Step 6: Convert the response to speech and present it to the user

[0311] The device converts the text data received from the server into speech using a speech synthesis engine (e.g., Amazon Polly). The input is the text data of the response sent from the server, which the speech synthesis engine converts into a voice signal through a speech synthesis process. The device outputs this voice signal through a speaker and presents it to the user. The output data is a response in voice format. For example, the user is told aloud, "The nearest gas station is XX. It will take XX minutes to get there."

[0312] As described above, the car navigation system of the present invention realizes a series of processes that start with user voice input, combines a generative AI model and an emotion engine to provide advanced responses, and finally conveys information to the user through voice synthesis.

[0313] (Application example 2)

[0314] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0315] Conventional car navigation systems and shopping assistant applications simply convert user voice input into text and generate responses, but do not provide responses that take into account the user's mental state or emotions. As a result, it is difficult to provide an appropriate response that takes into account the user's emotional state, even when the user is confused, confused, or anxious. Furthermore, there has been no system that can recognize the user's emotions and provide appropriate product recommendations or advice in the shopping experience in a physical store.

[0316] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for processing user input using a generative model, means for receiving voice input from the user and converting it into character string data, means for recognizing the user's emotions, means for adjusting the generated response according to the user's emotional state, and means for acquiring the generated response, converting it into voice, and presenting it to the user. This makes it possible to recognize the user's emotions and generate and present flexible and appropriate responses accordingly. As a result, it is possible to improve user convenience and satisfaction and provide a more friendly and reassuring user experience.

[0317] A "generative model" is an algorithm that automatically generates appropriate responses and information based on user input data.

[0318] "User input" means data provided by a user for processing by the system, including voice input and text input.

[0319] "Voice input" is data provided by a user by speaking to the system through a microphone.

[0320] "Character string data" is data that is generated by analyzing voice input using voice recognition technology and converting it into text format.

[0321] The "means for recognizing emotions" is an algorithm for analyzing character string data or the voice itself to identify the user's emotional state.

[0322] A "means for tailoring a response" is a mechanism for modifying the generated response in response to the user's emotional state to provide a more appropriate, situation-specific response.

[0323] The "means for obtaining a response" is the process by which the system receives the response data generated by the generative model.

[0324] The "means for converting into voice" is a technology for converting a response in text format into voice data and presenting it in a form that the user can hear.

[0325] "User's emotional state" refers to the mental state or emotion the user is feeling (e.g., stressed, relaxed, confused, etc.).

[0326] The present invention relates to an advanced shopping assistant system that combines a generative model and an emotion recognition engine. Specific implementation methods of the invention will be described below with reference to the following embodiments.

[0327] Hardware Configuration

[0328] The main hardware components of this system are a smartphone and an internet connection. The smartphone is equipped with a microphone and a speaker, enabling voice interaction with the user.

[0329] Software Configuration

[0330] The software includes the following main components:

[0331] Speech recognition software: Google Speech-to-Text API

[0332] Emotion recognition engine: Transformer-based sentiment analysis model (e.g., RoBERTa)

[0333] Generative model: GPT-2

[0334] Speech synthesis software: AWS(R) Polly

[0335] Communication library: A library for HTTP communication (e.g., requests)

[0336] Data processing and calculation

[0337] Acquiring and converting voice input

[0338] When a user speaks into a smartphone, the microphone picks up the audio, which is then converted into text data using the Google Speech-to-Text API.

[0339] emotion recognition

[0340] The converted text data is then used to recognize emotional states using a Transformer-based sentiment analysis model, which identifies the user's current mental state, such as stressed, relaxed, or confused.

[0341] Response Generation

[0342] The text data and the recognized emotion data are input into a generative model, which uses GPT-2 to generate an appropriate response to the user's input. The content and tone of the response are adjusted based on the emotion data.

[0343] Generate and present a voice response

[0344] The generated response is converted into speech using AWS Polly and presented to the user through the smartphone speaker.

[0345] Specific examples

[0346] Take the example of a user searching for a product in a physical store. The user speaks to their smartphone, saying, "Tell me more about this product." The smartphone's microphone picks up the audio, and the Google Speech-to-Text API converts the audio data into text. The converted text data is analyzed by a sentiment analysis engine to identify the user's questions and interests. Next, the generative model GPT-2 creates a response based on the user's request, such as, "This product is high quality and has the following features. We also recommend this new product." Finally, AWS Polly synthesizes the created response into speech and presents it to the user through the speaker.

[0347] Prompt Sentence Examples

[0348] "Tell me about the features of this product"

[0349] "Please tell me what products you recommend."

[0350] "What are the most popular items on the menu?"

[0351] As described above, the shopping assistant system of the present invention can provide a more personal shopping experience by providing flexible responses based on the user's voice input and emotional state.

[0352] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0353] Step 1: Getting voice input

[0354] The user speaks into the smartphone, and the smartphone's microphone captures the voice input. The specific input is the voice data that the user speaks into the microphone, which is output as voice data to be used in subsequent processing steps.

[0355] Step 2: Converting audio data to string data

[0356] The device converts the acquired voice data into text data using the Google Speech-to-Text API. The input is the acquired voice data, and the converted text data is output.

[0357] Step 3: Emotion Recognition

[0358] The device inputs the converted text data into an emotion recognition engine, which analyzes the text data to obtain emotion data. The input is text data, and the output is the user's emotional state (e.g., stressed, relaxed, confused).

[0359] Step 4: Input to the generative model

[0360] The device inputs text data and emotion data into a generative model (GPT-2) to generate an appropriate response. The input is text data and emotion data, and the generated response text is output.

[0361] Step 5: Adjust the response

[0362] The device adjusts the response text generated by the generative model based on the emotion data. The input is the generated response text and emotion data, and the device outputs the response text adjusted to match the user's emotional state.

[0363] Step 6: Transcribing responses

[0364] The device converts the tailored response text into speech data using AWS Polly. The input is the tailored response text, and the converted speech data is output.

[0365] Step 7: Present the voice response

[0366] The terminal presents the converted voice data to the user through the smartphone speaker. The input is the converted voice data, and the voice output to be presented to the user is obtained.

[0367] Through the above processing steps, an appropriate response that takes into account the user's emotions in response to the user's voice input is generated and presented in the form of voice.

[0368] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0369] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0370] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0371] [Second embodiment]

[0372] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0373] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0374] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0375] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0376] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0377] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0378] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0379] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0380] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0381] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0382] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0383] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0384] The present invention relates to a next-generation car navigation system that processes user input using a generative model. The following embodiments will explain how to specifically implement the system of the present invention.

[0385] 1. A way to capture user voice input

[0386] When a user speaks to the car navigation system, the device uses a microphone to record the voice. Here, voice input is acquired by the user speaking instructions, questions, requests, etc. to the car navigation system.

[0387] 2. A method for converting audio into text data

[0388] The device uses a voice recognition function to convert the recorded voice data into text data, using natural language processing technology to convert the spoken voice into appropriate text data.

[0389] 3. A means of sending user input to the generative model

[0390] The device sends the converted text data to the server. The server then inputs the received text data into a generative model to generate an appropriate response. A generative model is an artificial intelligence technology that uses natural language processing to generate responses to user input.

[0391] 4. How to retrieve the generated response

[0392] The server retrieves the response generated by the generative model and sends the corresponding text data back to the device, including the information requested by the user and notifications about the vehicle's status.

[0393] 5. A means of converting the response into speech and presenting it to the user

[0394] The device converts the text data received from the server into speech using a speech synthesis engine, and presents the converted speech response to the user, conveying information in a form that is easy for the user to understand.

[0395] Example: Navigation request

[0396] When a user speaks to a car navigation system saying, "Tell me where the nearest gas station is," the device's microphone records the speech and converts it into text data saying, "Tell me where the nearest gas station is" using speech recognition. This data is sent to the server, which processes the generative model to generate a response saying, "The nearest gas station is ____. It will take ____ minutes to get there," and sends it back to the device. The device then converts this response into speech and provides the user with audible guidance saying, "The nearest gas station is ____. It will take ____ minutes to get there."

[0397] Example: Notification of vehicle abnormality and instructions on how to deal with it

[0398] When the vehicle detects an abnormality, such as a notification that "tire pressure is low," the device sends that information to the server. The server uses the generative model to generate specific instructions, such as "Go to the nearest service station to check your tire pressure. You can check it by following the steps below...," and sends them back to the device. The device then converts these instructions into voice and provides the user with quick and specific instructions on how to deal with the problem.

[0399] In this way, the car navigation system of the present invention can provide interactive and advanced support compared to conventional systems, and can significantly improve user convenience.

[0400] The processing flow will be explained below.

[0401] Step 1:

[0402] The user speaks to the car navigation system. The user issues a voice request, such as "Tell me where the nearest gas station is."

[0403] Step 2:

[0404] The device records the user's voice. The device's microphone picks up what the user says and stores it as digital audio data.

[0405] Step 3:

[0406] The device converts the voice data into text data. The device uses a speech recognition engine to convert the recorded voice data into text. This text is in the format "Where is the nearest gas station?"

[0407] Step 4:

[0408] The terminal sends the string data to the server. The terminal then sends the converted text data to the server as an HTTP POST request.

[0409] Step 5:

[0410] The server receives user input. The server receives the string data sent from the device and prepares it to be passed to the generative model.

[0411] Step 6:

[0412] The server inputs user input into the generative model. The server inputs user text data into the generative model (e.g., a natural language processing model) and instructs it to generate an appropriate response.

[0413] Step 7:

[0414] The server receives the generated response. The generative model generates a response such as "The nearest gas station is ____. It will take ____ minutes to get there," and the server receives this.

[0415] Step 8:

[0416] The server sends a response back to the device. The server sends the generated text response back to the device in JSON format.

[0417] Step 9:

[0418] The device receives the response. The device receives the text response sent by the server and prepares to pass it to the speech synthesis engine.

[0419] Step 10:

[0420] The device converts the text response into speech. The device uses a speech synthesis engine to convert the text data into speech data.

[0421] Step 11:

[0422] The device presents a voice response to the user. The device plays the generated voice data through a speaker, informing the user, "The nearest gas station is ____. It will take ____ minutes to get there."

[0423] Example 1

[0424] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0425] Conventional car navigation systems have limitations in their ability to provide immediate responses to user voice input, making it difficult to quickly provide specific solutions to complex requests or vehicle abnormalities. In particular, when a user makes a request through voice input, it is important to accurately understand the request and provide appropriate information and instructions. However, existing technologies lack sufficient interactivity, resulting in a loss of user convenience.

[0426] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0427] In this invention, the server includes means for acquiring user input as voice, means for converting the acquired voice into character string data, means for transmitting the converted character string data to the generative model, means for acquiring a response generated by the generative model, and means for converting the acquired response into voice and presenting it to the user. This makes it possible to provide a response to a request made by the user through voice input with high accuracy and speed. It is also possible to detect vehicle abnormalities in real time, generate specific countermeasures, and provide guidance to the user. This allows the user to operate the vehicle safely and smoothly, improving convenience.

[0428] "Means for capturing user input as audio" refers to devices and techniques for recording a user's spoken instructions or questions.

[0429] "Means for converting acquired audio into character string data" refers to the function of converting recorded audio data into text data using natural language processing technology.

[0430] "Means for sending converted string data to a generative model" refers to a function for sending text data via a network to a generative model on a server or cloud.

[0431] "Means for obtaining responses generated by a generative model" refers to the function of receiving response data provided by a generative AI model and storing it within the system.

[0432] "Means for converting the acquired response into speech and presenting it to the user" refers to a function for converting a response acquired as text data into speech using speech synthesis technology and allowing the user to hear it.

[0433] "Means for detecting vehicle abnormalities" refers to the function of using sensors and monitoring systems inside the vehicle to detect whether there are any abnormalities in each part or operating condition of the vehicle.

[0434] "Means for presenting specific countermeasures" refers to a function that generates specific actions and procedures that the user should take in response to a detected vehicle abnormality and provides that information to the user.

[0435] "Means for transmitting to a cloud server" refers to a communication function for transmitting data to a remote server via the Internet.

[0436] MODE FOR CARRYING OUT THE INVENTION

[0437] The present invention relates to a next-generation car navigation system that processes user input using a generative model. Specific embodiments of the present invention are described below.

[0438] 1. Voice input acquisition

[0439] When a user issues voice commands to the car navigation system, the device records the voice using its built-in microphone. The recorded voice data is temporarily stored in the device's memory. This step uses a standard microphone and voice recording software.

[0440] example:

[0441] When a user says, "Tell me where the nearest gas station is," the audio is recorded.

[0442] 2. Convert audio to text data

[0443] The recorded voice data is converted into text data using natural language processing technology. Specifically, speech recognition software is used. In this example, the Google Cloud Speech-to-Text API is used.

[0444] example:

[0445] The recorded voice saying "Tell me where the nearest gas station is" is converted into text data saying "Tell me where the nearest gas station is."

[0446] 3. Send user input to the generative model

[0447] The converted text data is sent from the device to a cloud server. The cloud server receives this text data and inputs it into a generative AI model. This generative AI model generates an appropriate response based on the user input. Data is sent and received using standard cloud communication technologies.

[0448] example:

[0449] The data sent from the device, "Tell me where the nearest gas station is," reaches the cloud server and is input as a prompt into the generative AI model.

[0450] 4. Retrieving the generated response

[0451] A generative AI model generates a response to user input, and the data is received by a cloud server. This data is then sent back to the device and stored locally. An example of a generative AI model is "OpenAI GPT-4."

[0452] example:

[0453] The generative AI model generates a response such as, "The nearest gas station is XX. It will take XX minutes to get there," and sends it to the device via the cloud server.

[0454] 5. Convert the response into audio and present it

[0455] The device converts the text data received from the server into speech using a speech synthesis engine, such as the Amazon Polly API, and presents the speech data to the user through the speaker.

[0456] example:

[0457] The device converts the received text data, such as "The nearest gas station is XX. It will take XX minutes to get there," into speech and plays it back to the user.

[0458] Specific examples

[0459] For navigation requests

[0460] User: "Where's the nearest gas station?"

[0461] Device: Records voice and uses voice recognition technology to convert it into text data such as "Tell me where the nearest gas station is."

[0462] Server: Input the text data into the generative AI model and obtain the generated response, "The nearest gas station is XX. It will take XX minutes to arrive."

[0463] Terminal: Converts the response into speech and presents it to the user.

[0464] Notification of vehicle abnormality and instructions on how to deal with it

[0465] Vehicle: Detects abnormalities such as low tire pressure.

[0466] Terminal: Sends abnormality information to the cloud server.

[0467] Server: Using a generative AI model, it generates a response such as, "Go to the nearest service station to check your tire pressure. You can do this by following these steps..."

[0468] Terminal: Converts the generated response into speech and presents it to the user.

[0469] In this way, the car navigation system of the present invention can significantly improve user convenience by responding quickly and accurately to user voice inputs and dealing appropriately with vehicle abnormalities.

[0470] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0471] Step 1: User speaks

[0472] Specific explanation and operation

[0473] The user issues commands or questions to the car navigation system, for example, "Tell me where the nearest gas station is."

[0474] Input: User's voice

[0475] Output: Recorded audio data (e.g. audio_input.wav)

[0476] The user speaks to the car navigation system, and the voice is recorded through a microphone.

[0477] Step 2: The device converts the audio into text data

[0478] Specific explanation and operation

[0479] The device converts the recorded voice data into text data using natural language processing technology.

[0480] Software used: Speech recognition software (e.g., "Google Cloud Speech-to-Text")

[0481] Input: Recorded audio data (audio_input.wav)

[0482] Output: Text data "Where is the nearest gas station?"

[0483] Processing: The device sends the audio_input.wav file to the speech recognition API and receives the text data.

[0484] Step 3: The device sends the string data to the generative model

[0485] Specific explanation and operation

[0486] The device sends the converted text data to a cloud server, which then inputs the text data into a generative AI model.

[0487] Input: Text data "Tell me the nearest gas station"

[0488] Output: The prompt sentence that is input to the generative model

[0489] Processing: The device sends text data to the server via an HTTP request, and the server receives the data and passes it to the generative AI model.

[0490] Step 4: The server gets the generated response

[0491] Specific explanation and operation

[0492] The generative AI model generates an appropriate response based on the user's prompt. The server retrieves the generated response data and sends it back to the device.

[0493] Software used: Generative AI models (e.g., "OpenAI GPT-4")

[0494] Input: Prompt "Where is the nearest gas station?"

[0495] Output: Text data "The nearest gas station is ____. It will take ____ minutes to get there."

[0496] Processing: The server receives the response from the generative AI model and sends the data to the device.

[0497] Step 5: The device converts the response into speech and presents it to the user.

[0498] Specific explanation and operation

[0499] The terminal converts the received text data into speech using a speech synthesis engine and presents it to the user.

[0500] Software used: Speech synthesis engine (e.g., "Amazon Polly")

[0501] Input: Text data: "The nearest gas station is ____. It will take ____ minutes to get there."

[0502] Output: Speech data: "The nearest gas station is ____. It will take ____ minutes to get there."

[0503] Processing: The device sends the text data to the speech synthesis API, and the generated speech data is played on the speaker.

[0504] This series of processing steps allows the user to operate the car navigation system through voice and receive prompt and accurate responses.

[0505] (Application example 1)

[0506] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0507] Conventional car navigation systems have limited ability to respond to user instructions and questions, resulting in a lack of interactivity. Furthermore, they lack the ability to provide specific solutions when vehicle abnormalities are detected. Furthermore, they lack the functionality to present multiple options in response to abstract user requests. To address these issues, the development of more advanced, user-friendly navigation systems is needed.

[0508] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0509] In this invention, the server includes means for collecting user input via voice and converting it into text data, means for sending the collected text data to a generative model and obtaining a text response returned from the generative model, means for converting the obtained text response into voice and presenting it to the user, and means for generating responses to user instructions and questions using the generative model and providing vehicle status confirmation, route guidance, and real-time traffic information. This allows the user to enjoy advanced and interactive navigation, as well as to obtain quick and specific countermeasures in the event of a vehicle abnormality, and further allows the user to receive multiple options even for abstract requests.

[0510] "User input" refers to information such as voice instructions or questions that a user gives to the system.

[0511] "Text data" refers to data that has been analyzed and converted into a character string format from voice input.

[0512] A "generative model" is an artificial intelligence technique used to generate appropriate responses to user input.

[0513] A "server" is a computer system that operates a generative model and generates and provides responses to user input.

[0514] "Vehicle status check" refers to monitoring and evaluating the current condition of a vehicle based on data obtained from various sensors and systems on the vehicle.

[0515] "Route guidance" refers to calculating and presenting the optimal route for a user to reach their destination.

[0516] "Real-time traffic information" means providing information on current road conditions, traffic volume, etc. immediately.

[0517] The "voice collection means" refers to a device or technology that acquires the user's voice using a microphone or the like.

[0518] "Text conversion means" refers to software or algorithms for converting acquired voice data into character string data.

[0519] A "response acquisition means" refers to a communication means or protocol for receiving text responses generated from a generative model.

[0520] "Speech conversion means" refers to technology or devices for converting text responses from a generative model into speech data.

[0521] An "abstract request" is a vague question or instruction that does not include specific information or instructions from the user.

[0522] This invention relates to a next-generation car navigation system that processes user input using generative models. When a user issues a command or question by voice, the system recognizes the voice, generates a response using a generative model, and provides it to the user as voice.

[0523] The system has the following configuration:

[0524] Hardware and software configuration

[0525] 1. Voice collection means: The user's voice is acquired using a microphone in the vehicle.

[0526] 2. Text conversion method: Use speech recognition software (e.g., speech_recognition library) to convert the captured audio into text data.

[0527] 3. Means for transmitting text data: A communication protocol is used to transmit text data to the generative model.

[0528] 4. Response generator: A generative model (e.g., the transformers library) responsible for natural language processing generates an appropriate text response.

[0529] 5. Speech conversion means: Generates voice data using a speech synthesis engine (e.g., pyttsx3 library) that converts the generated text response into speech and presents it to the user.

[0530] Program processing

[0531] The server constructs a system including the following means.

[0532] 1. Collecting user voice input: The device captures the user's voice through the microphone and saves it as voice data.

[0533] 2. Speech-to-text conversion: The device's speech recognition software converts this speech data into text data.

[0534] 3. Sending data to the generative model: The string data is sent to the server, which then inputs the data into the generative model.

[0535] 4. Obtaining and sending the response: The server obtains the response generated by the generative model and sends the text response back to the device.

[0536] 5. Generating and presenting a voice response: The device converts the text data into voice data using a voice synthesis engine and presents it to the user.

[0537] For example, if a user says "Where is the nearest restaurant?" while in a vehicle, the system recognizes the speech and converts it into text data. The converted string data is sent to the server, and the generative model generates a response such as "The nearest restaurant is ____. Travel time is ____ minutes." This generated text response is sent back from the server to the device, where it is converted into speech by a speech synthesis engine and presented to the user.

[0538] For example, the prompt text might look like this:

[0539] Prompt statement:

[0540] User: Where is the nearest restaurant?

[0541] Generative AI model input sentence: Where is the nearest restaurant?

[0542] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0543] Step 1:

[0544] Acquiring voice input

[0545] The device uses a microphone to collect the user's voice. When the user speaks in the vehicle, "Where is the nearest restaurant?", this voice is captured by the microphone as input data. The input is the user's voice data, and this voice data is passed to the device.

[0546] Step 2:

[0547] Converting audio data to text

[0548] The device processes the input voice data using voice recognition software (for example, the speech_recognition library) and converts the voice into text data. This voice recognition process converts the voice data into text data such as "Where is the nearest restaurant?" The output is string data.

[0549] Step 3:

[0550] Sending text data

[0551] The terminal sends the converted text data to the server. In order for the terminal to send the text data "Where is the nearest restaurant?" to the server, it transfers the data using a communication protocol. The input is text data, which is sent to the server as output.

[0552] Step 4:

[0553] Generating a response

[0554] The server inputs the received text data into a generative model to generate an appropriate response. The generative model (e.g., the transformers library) generates the response "The nearest restaurant is ____. Travel time is ____ minutes" based on the input text data "Where is the nearest restaurant?". The output is the generated text response.

[0555] Step 5:

[0556] Sending a Response

[0557] The server sends the text response generated from the generative model to the terminal. The server then sends the generated text response, "The nearest restaurant is ____. Travel time is ____ minutes," to the terminal using a communication protocol. The input is the generated text response, which is sent to the terminal as output.

[0558] Step 6:

[0559] Generate a voice response

[0560] The device converts the text response received from the server into voice data using a speech synthesis engine (for example, the pyttsx3 library). The speech synthesis engine converts the text data, such as "The nearest restaurant is ____. Travel time is ____ minutes," into voice data and outputs it to the user through the speaker. The input is text data, and the output is voice data.

[0561] Step 7:

[0562] Voice response presentation

[0563] The device presents the generated voice data to the user through a speaker. The user receives a voice prompt saying, "The nearest restaurant is ____. Travel time is ____ minutes." The input is the voice data, and the output is the voice prompt presented to the user.

[0564] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0565] The present invention relates to an advanced car navigation system that combines a generative model and an emotion engine. The following embodiments will explain how to specifically implement the system of the present invention.

[0566] 1. A way to capture user voice input

[0567] When a user speaks to the car navigation system, the device uses a microphone to record the voice. Here, voice input is acquired by the user speaking instructions, questions, requests, etc. to the car navigation system.

[0568] 2. A method for converting audio into text data

[0569] The device uses a voice recognition function to convert the recorded voice data into text data, using natural language processing technology to convert the spoken voice into appropriate text data.

[0570] 3. A method for recognizing user emotions using an emotion engine

[0571] The device or server analyzes the text data obtained from the speech recognition and the speech itself to identify the user's emotions. The emotion engine uses algorithms to determine the user's emotional state, such as whether they are stressed or relaxed.

[0572] 4. A means of sending user input to the generative model

[0573] The device sends the converted text data and the recognized emotion data to the server, which then inputs the received data into a generative model to generate an appropriate response.

[0574] 5. How to obtain the generated response

[0575] The server retrieves the response generated by the generative model and sends the corresponding text data back to the device. This response can include information requested by the user or notifications about the vehicle's status. The content and tone of the response can also be adjusted based on emotion data.

[0576] 6. A means of converting the response into speech and presenting it to the user

[0577] The device converts the text data received from the server into speech using a speech synthesis engine, and presents the converted speech response to the user, conveying information in a form that is easy for the user to understand.

[0578] Example: Navigation request

[0579] When a user speaks to a car navigation system saying, "Tell me where the nearest gas station is," the device's microphone records the speech and converts it into text data saying, "Tell me where the nearest gas station is" using speech recognition. This data is sent to the server, which processes the generative model to generate a response saying, "The nearest gas station is ____. It will take ____ minutes to get there," and sends it back to the device. The device then converts this response into speech and provides the user with audible guidance saying, "The nearest gas station is ____. It will take ____ minutes to get there."

[0580] Example: Notification of vehicle abnormality and instructions on how to deal with it

[0581] When the vehicle detects an abnormality, such as a notification that "tire pressure is low," the device sends that information to the server. The server uses the generative model to generate specific instructions, such as "Go to the nearest service station to check your tire pressure. You can check it by following the steps below...," and sends them back to the device. The device then converts these instructions into voice and provides the user with quick and specific instructions on how to deal with the problem.

[0582] Example: Responding according to the user's emotions

[0583] The emotion engine recognizes when a user is tired or stressed. For example, if a user says, "The roads are crowded and I'm frustrated," the device records the audio and the emotion engine recognizes the emotion of frustration. The server uses the generative model to generate an encouraging response such as, "Don't worry. We're almost there. Relax and drive safely." The device then converts this response into voice and delivers it to the user in a gentle tone.

[0584] In this way, the car navigation system of the present invention provides more interactive and advanced support than conventional systems, significantly improving user convenience and satisfaction. The introduction of an emotion engine enables flexible responses according to the user's mental state, providing a more intimate and secure driving environment.

[0585] The processing flow will be explained below.

[0586] Step 1:

[0587] The user speaks to the car navigation system. The user issues a voice request, such as "Tell me where the nearest gas station is."

[0588] Step 2:

[0589] The device records the user's voice. The device's microphone picks up what the user says and stores it as digital audio data.

[0590] Step 3:

[0591] The device converts the voice data into text data. The device uses a speech recognition engine to convert the recorded voice data into text. This text is in the format "Where is the nearest gas station?"

[0592] Step 4:

[0593] The device passes the voice data to the emotion engine, which then inputs the recorded voice data into the emotion engine and analyzes the user's emotional state.

[0594] Step 5:

[0595] The emotion engine recognizes emotions. The emotion engine analyzes the voice or text data and identifies emotions such as whether the user is stressed or relaxed.

[0596] Step 6:

[0597] The device sends text data and emotion data to the server, which then sends the data to the server as an HTTP POST request.

[0598] Step 7:

[0599] The server receives user input and emotion data. The server receives the text data and emotion data sent from the device and prepares them to be passed to the generative model.

[0600] Step 8:

[0601] The server provides user input and emotion data to the generative model, and instructs it to generate an appropriate response.

[0602] Step 9:

[0603] The generative model generates a response based on user input and emotion. Based on the identified emotion, the generative model generates a response with appropriate tone and content. For example, "Please stay calm. The nearest gas station is ____."

[0604] Step 10:

[0605] The server receives the generated response. The server receives the response generated by the generative model and prepares to send it back to the device.

[0606] Step 11:

[0607] The server sends a response back to the device. The server sends the generated text response back to the device in JSON format.

[0608] Step 12:

[0609] The device receives the response. The device receives the text response sent by the server and prepares to pass it to the speech synthesis engine.

[0610] Step 13:

[0611] The device converts the text response into speech. The device uses a speech synthesis engine to convert the text data into speech data.

[0612] Step 14:

[0613] The device presents a voice response to the user. The device plays the generated voice data through a speaker, guiding the user in a tone that takes into consideration the user's emotions, such as "The nearest gas station is ____. It will take ____ minutes to get there."

[0614] Example 2

[0615] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0616] Conventional car navigation systems can only provide guidance and information based on simple voice input from the user, and have the problem of being unable to flexibly respond to the user's emotional state or vehicle abnormalities. Furthermore, they lack the functionality to present appropriate options in response to abstract user requests. This can cause stress for users and reduces satisfaction.

[0617] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0618] In this invention, the server includes means for using a generative model to process user input, means for receiving voice input from the user and converting it into character string data, means for recognizing the user's emotion using an emotion engine, means for adjusting the content and tone of a response based on the user's emotion, means for sending the character string data to the generative model and obtaining a generated response, and means for converting the obtained response into voice and presenting it to the user. This makes it possible to provide flexible responses according to the user's emotional state, suggest specific ways to deal with vehicle abnormalities, and even provide multiple appropriate options for abstract requests.

[0619] A "generative model" is a program with an algorithm that generates appropriate responses based on user input information.

[0620] An "emotion engine" is software or algorithms that analyze voice and text data to identify a user's emotional state.

[0621] "Voice input" refers to the voice that a user speaks to a device, which is acquired as data via a microphone.

[0622] "Character string data" refers to data that has been converted from voice input into text format using voice recognition technology.

[0623] A "speech synthesis engine" is software or algorithms that convert text data so that it can be played as speech.

[0624] A "server" is a central computing device that processes data sent by users and generates the necessary responses.

[0625] A "terminal" is a device that is directly used by a user or a device with a user interface, which receives voice input and outputs voice.

[0626] This invention shows how to specifically implement an advanced car navigation system that combines a generative model and an emotion engine. The components of the system and their operation are described in detail below.

[0627] 1. A way to capture user voice input

[0628] When a user speaks to the car navigation system in the car, the device records the voice using a microphone built into the car. Voice input is acquired when the user speaks instructions, questions, requests, etc. to the car navigation system. For example, if a user says, "Tell me where the nearest gas station is," the voice is recorded. A built-in microphone is required as hardware.

[0629] 2. A method for converting audio into text data

[0630] The device converts the recorded voice data into text data using speech recognition software (e.g., Google Cloud Speech-to-Text). Natural language processing technology is used to convert the spoken voice into text data. During this conversion process, the voice waveform is analyzed to generate the appropriate text data. For example, the speech "Tell me where the nearest gas station is" is converted into text data "Tell me where the nearest gas station is."

[0631] 3. A method for recognizing user emotions using an emotion engine

[0632] The device or server analyzes the text data obtained from the speech recognition and the speech itself, and identifies the user's emotion using an emotion engine (e.g., IBM Watson Tone Analyzer). The emotion engine uses an algorithm to determine the user's emotional state, such as whether they are stressed or relaxed. For example, if a user says, "The roads are crowded and I'm getting annoyed," the emotion engine will recognize the emotion as "annoyed."

[0633] 4. A means of sending user input to the generative model

[0634] The device sends the converted text data and the recognized emotion data to the server. The server inputs this data into a generative AI model (e.g., OpenAI GPT-4) to generate an appropriate response. For example, in response to a request such as "Tell me where the nearest gas station is," the generative AI model generates a response such as "The nearest gas station is ____. It will take ____ minutes to get there."

[0635] 5. How to obtain the generated response

[0636] The server receives the response generated by the generative model and sends it back to the device as text data. This response includes the information requested by the user and notifications about the vehicle's status. It is also possible to adjust the content and tone of the response based on emotional data. Specifically, the server generates a response such as "It's okay. We'll arrive at our destination soon. Relax and drive safely," and sends it to the device.

[0637] 6. A means of converting the response into speech and presenting it to the user

[0638] The device converts the text data received from the server into speech using a speech synthesis engine (e.g., Amazon Polly). The speech synthesis engine converts the input text into a speech signal and outputs it from the speaker. This process allows the user to hear the generated response as audio. For example, the user may hear voice guidance such as, "The nearest gas station is ____. It will take ____ minutes to arrive."

[0639] Specific examples

[0640] 1. For navigation requests:

[0641] When a user speaks to a car navigation system saying, "Tell me where the nearest gas station is," the device's microphone records the speech and converts it into text data such as "Tell me where the nearest gas station is" using speech recognition. This data is sent to the server, which processes it using a generative AI model to generate a response such as "The nearest gas station is ____. It will take ____ minutes to get there," which is sent back to the device. The device converts this response into speech and provides the user with audible guidance such as, "The nearest gas station is ____. It will take ____ minutes to get there."

[0642] 2. Vehicle abnormality notification and instructions:

[0643] When the vehicle detects an abnormality, such as a notification that "tire pressure is low," the device sends that information to the server. The server uses the generative model to generate specific instructions, such as "Go to the nearest service station to check your tire pressure. You can check it by following the steps below...," and sends them back to the device. The device then converts these instructions into voice and provides the user with quick and specific instructions on how to deal with the problem.

[0644] 3. Responding according to the user's emotions:

[0645] The emotion engine recognizes when a user is tired or stressed. For example, if a user says, "The roads are crowded and I'm frustrated," the device records the audio and the emotion engine recognizes the emotion of frustration. The server uses the generative model to generate an encouraging response: "Don't worry. We're almost there. Relax and drive safely." The device then converts this response into speech and delivers it to the user in a gentle tone.

[0646] In this way, the car navigation system of the present invention provides more interactive and advanced support than conventional systems, significantly improving user convenience and satisfaction. The introduction of an emotion engine enables flexible responses according to the user's mental state, providing a more intimate and secure driving environment.

[0647] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0648] Step 1: Capture the user's voice input

[0649] When a user speaks to the car navigation system in the car, the device uses a built-in microphone to record the voice. The input is the user's natural voice, which is captured by the device as digital voice data. For example, if a user says, "Tell me where the nearest gas station is," that voice is recorded. The data output is digital voice data.

[0650] Step 2: Convert audio to text

[0651] The device converts the recorded voice data into text data using voice recognition software (e.g., Google Cloud Speech-to-Text). The input voice data is analyzed and converted into text data using its waveform information. This process transforms the voice signal into text data such as "Tell me where the nearest gas station is." The output data is text data corresponding to the voice data.

[0652] Step 3: Recognize user emotions using the emotion engine

[0653] The device or server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to analyze the text data and audio data obtained from speech recognition. The input is text data or raw audio data, and the emotion engine determines the emotional state. For example, if a user says, "The roads are crowded and I'm getting annoyed," the emotion engine analyzes the text data and audio data and recognizes the emotion of annoyance. The output data is the identified emotional information.

[0654] Step 4: Sending user input to the generative model

[0655] The device sends the converted text data and the recognized emotion data to the server. The input is text data and emotion data, which are sent to the server. The server inputs this data into a generative AI model (e.g., OpenAI GPT-4) and uses it as a prompt to generate an appropriate response to the user's request. For example, based on the request "Tell me where the nearest gas station is," the response generated is "The nearest gas station is XX. It will take XX minutes to get there." The output data is the text data of the generated response.

[0656] Step 5: Get the generated response

[0657] The server receives the response generated by the generative AI model and returns it to the device as text data. The input is the string data generated by the generative AI model, and an appropriate response is constructed based on this. The output data is the response string data. For example, the server might send a response to the device saying, "The nearest gas station is ____. It will take ____ minutes to arrive."

[0658] Step 6: Convert the response to speech and present it to the user

[0659] The device converts the text data received from the server into speech using a speech synthesis engine (e.g., Amazon Polly). The input is the text data of the response sent from the server, which the speech synthesis engine converts into a voice signal through a speech synthesis process. The device outputs this voice signal through a speaker and presents it to the user. The output data is a response in voice format. For example, the user is told aloud, "The nearest gas station is XX. It will take XX minutes to get there."

[0660] As described above, the car navigation system of the present invention realizes a series of processes that start with user voice input, combines a generative AI model and an emotion engine to provide advanced responses, and finally conveys information to the user through voice synthesis.

[0661] (Application example 2)

[0662] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0663] Conventional car navigation systems and shopping assistant applications simply convert user voice input into text and generate responses, but do not provide responses that take into account the user's mental state or emotions. As a result, it is difficult to provide an appropriate response that takes into account the user's emotional state, even when the user is confused, confused, or anxious. Furthermore, there has been no system that can recognize the user's emotions and provide appropriate product recommendations or advice in the shopping experience in a physical store.

[0664] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for processing user input using a generative model, means for receiving voice input from the user and converting it into character string data, means for recognizing the user's emotions, means for adjusting the generated response according to the user's emotional state, and means for acquiring the generated response, converting it into voice, and presenting it to the user. This makes it possible to recognize the user's emotions and generate and present flexible and appropriate responses accordingly. As a result, it is possible to improve user convenience and satisfaction and provide a more friendly and reassuring user experience.

[0665] A "generative model" is an algorithm that automatically generates appropriate responses and information based on user input data.

[0666] "User input" means data provided by a user for processing by the system, including voice input and text input.

[0667] "Voice input" is data provided by a user by speaking to the system through a microphone.

[0668] "Character string data" is data that is generated by analyzing voice input using voice recognition technology and converting it into text format.

[0669] The "means for recognizing emotions" is an algorithm for analyzing character string data or the voice itself to identify the user's emotional state.

[0670] A "means for tailoring a response" is a mechanism for modifying the generated response in response to the user's emotional state to provide a more appropriate, situation-specific response.

[0671] The "means for obtaining a response" is the process by which the system receives the response data generated by the generative model.

[0672] The "means for converting into voice" is a technology for converting a response in text format into voice data and presenting it in a form that the user can hear.

[0673] "User's emotional state" refers to the mental state or emotion the user is feeling (e.g., stressed, relaxed, confused, etc.).

[0674] The present invention relates to an advanced shopping assistant system that combines a generative model and an emotion recognition engine. Specific implementation methods of the invention will be described below with reference to the following embodiments.

[0675] Hardware Configuration

[0676] The main hardware components of this system are a smartphone and an internet connection. The smartphone is equipped with a microphone and a speaker, enabling voice interaction with the user.

[0677] Software Configuration

[0678] The software includes the following main components:

[0679] Speech recognition software: Google Speech-to-Text API

[0680] Emotion recognition engine: Transformer-based sentiment analysis model (e.g., RoBERTa)

[0681] Generative model: GPT-2

[0682] Text-to-speech software: AWS Polly

[0683] Communication library: A library for HTTP communication (e.g., requests)

[0684] Data processing and calculation

[0685] Acquiring and converting voice input

[0686] When a user speaks into a smartphone, the microphone picks up the audio, which is then converted into text data using the Google Speech-to-Text API.

[0687] emotion recognition

[0688] The converted text data is then used to recognize emotional states using a Transformer-based sentiment analysis model, which identifies the user's current mental state, such as stressed, relaxed, or confused.

[0689] Response Generation

[0690] The text data and the recognized emotion data are input into a generative model, which uses GPT-2 to generate an appropriate response to the user's input. The content and tone of the response are adjusted based on the emotion data.

[0691] Generate and present a voice response

[0692] The generated response is converted into speech using AWS Polly and presented to the user through the smartphone speaker.

[0693] Specific examples

[0694] Take the example of a user searching for a product in a physical store. The user speaks to their smartphone, saying, "Tell me more about this product." The smartphone's microphone picks up the audio, and the Google Speech-to-Text API converts the audio data into text. The converted text data is analyzed by a sentiment analysis engine to identify the user's questions and interests. Next, the generative model GPT-2 creates a response based on the user's request, such as, "This product is high quality and has the following features. We also recommend this new product." Finally, AWS Polly synthesizes the created response into speech and presents it to the user through the speaker.

[0695] Prompt Sentence Examples

[0696] "Tell me about the features of this product"

[0697] "Please tell me what products you recommend."

[0698] "What are the most popular items on the menu?"

[0699] As described above, the shopping assistant system of the present invention can provide a more personal shopping experience by providing flexible responses based on the user's voice input and emotional state.

[0700] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0701] Step 1: Getting voice input

[0702] The user speaks into the smartphone, and the smartphone's microphone captures the voice input. The specific input is the voice data that the user speaks into the microphone, which is output as voice data to be used in subsequent processing steps.

[0703] Step 2: Converting audio data to string data

[0704] The device converts the acquired voice data into text data using the Google Speech-to-Text API. The input is the acquired voice data, and the converted text data is output.

[0705] Step 3: Emotion Recognition

[0706] The device inputs the converted text data into an emotion recognition engine, which analyzes the text data to obtain emotion data. The input is text data, and the output is the user's emotional state (e.g., stressed, relaxed, confused).

[0707] Step 4: Input to the generative model

[0708] The device inputs text data and emotion data into a generative model (GPT-2) to generate an appropriate response. The input is text data and emotion data, and the generated response text is output.

[0709] Step 5: Adjust the response

[0710] The device adjusts the response text generated by the generative model based on the emotion data. The input is the generated response text and emotion data, and the device outputs the response text adjusted to match the user's emotional state.

[0711] Step 6: Transcribing responses

[0712] The device converts the tailored response text into speech data using AWS Polly. The input is the tailored response text, and the converted speech data is output.

[0713] Step 7: Present the voice response

[0714] The terminal presents the converted voice data to the user through the smartphone speaker. The input is the converted voice data, and the voice output to be presented to the user is obtained.

[0715] Through the above processing steps, an appropriate response that takes into account the user's emotions in response to the user's voice input is generated and presented in the form of voice.

[0716] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0717] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0718] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0719] [Third embodiment]

[0720] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0721] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0722] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0723] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0724] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0725] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0726] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0727] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0728] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0729] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0730] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0731] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0732] The present invention relates to a next-generation car navigation system that processes user input using a generative model. The following embodiments will explain how to specifically implement the system of the present invention.

[0733] 1. A way to capture user voice input

[0734] When a user speaks to the car navigation system, the device uses a microphone to record the voice. Here, voice input is acquired by the user speaking instructions, questions, requests, etc. to the car navigation system.

[0735] 2. A method for converting audio into text data

[0736] The device uses a voice recognition function to convert the recorded voice data into text data, using natural language processing technology to convert the spoken voice into appropriate text data.

[0737] 3. A means of sending user input to the generative model

[0738] The device sends the converted text data to the server. The server then inputs the received text data into a generative model to generate an appropriate response. A generative model is an artificial intelligence technology that uses natural language processing to generate responses to user input.

[0739] 4. How to retrieve the generated response

[0740] The server retrieves the response generated by the generative model and sends the corresponding text data back to the device, including the information requested by the user and notifications about the vehicle's status.

[0741] 5. A means of converting the response into speech and presenting it to the user

[0742] The device converts the text data received from the server into speech using a speech synthesis engine, and presents the converted speech response to the user, conveying information in a form that is easy for the user to understand.

[0743] Example: Navigation request

[0744] When a user speaks to a car navigation system saying, "Tell me where the nearest gas station is," the device's microphone records the speech and converts it into text data saying, "Tell me where the nearest gas station is" using speech recognition. This data is sent to the server, which processes the generative model to generate a response saying, "The nearest gas station is ____. It will take ____ minutes to get there," and sends it back to the device. The device then converts this response into speech and provides the user with audible guidance saying, "The nearest gas station is ____. It will take ____ minutes to get there."

[0745] Example: Notification of vehicle abnormality and instructions on how to deal with it

[0746] When the vehicle detects an abnormality, such as a notification that "tire pressure is low," the device sends that information to the server. The server uses the generative model to generate specific instructions, such as "Go to the nearest service station to check your tire pressure. You can check it by following the steps below...," and sends them back to the device. The device then converts these instructions into voice and provides the user with quick and specific instructions on how to deal with the problem.

[0747] In this way, the car navigation system of the present invention can provide interactive and advanced support compared to conventional systems, and can significantly improve user convenience.

[0748] The processing flow will be explained below.

[0749] Step 1:

[0750] The user speaks to the car navigation system. The user issues a voice request, such as "Tell me where the nearest gas station is."

[0751] Step 2:

[0752] The device records the user's voice. The device's microphone picks up what the user says and stores it as digital audio data.

[0753] Step 3:

[0754] The device converts the voice data into text data. The device uses a speech recognition engine to convert the recorded voice data into text. This text is in the format "Where is the nearest gas station?"

[0755] Step 4:

[0756] The terminal sends the string data to the server. The terminal then sends the converted text data to the server as an HTTP POST request.

[0757] Step 5:

[0758] The server receives user input. The server receives the string data sent from the device and prepares it to be passed to the generative model.

[0759] Step 6:

[0760] The server inputs user input into the generative model. The server inputs user text data into the generative model (e.g., a natural language processing model) and instructs it to generate an appropriate response.

[0761] Step 7:

[0762] The server receives the generated response. The generative model generates a response such as "The nearest gas station is ____. It will take ____ minutes to get there," and the server receives this.

[0763] Step 8:

[0764] The server sends a response back to the device. The server sends the generated text response back to the device in JSON format.

[0765] Step 9:

[0766] The device receives the response. The device receives the text response sent by the server and prepares to pass it to the speech synthesis engine.

[0767] Step 10:

[0768] The device converts the text response into speech. The device uses a speech synthesis engine to convert the text data into speech data.

[0769] Step 11:

[0770] The device presents a voice response to the user. The device plays the generated voice data through a speaker, informing the user, "The nearest gas station is ____. It will take ____ minutes to get there."

[0771] Example 1

[0772] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0773] Conventional car navigation systems have limitations in their ability to provide immediate responses to user voice input, making it difficult to quickly provide specific solutions to complex requests or vehicle abnormalities. In particular, when a user makes a request through voice input, it is important to accurately understand the request and provide appropriate information and instructions. However, existing technologies lack sufficient interactivity, resulting in a loss of user convenience.

[0774] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0775] In this invention, the server includes means for acquiring user input as voice, means for converting the acquired voice into character string data, means for transmitting the converted character string data to the generative model, means for acquiring a response generated by the generative model, and means for converting the acquired response into voice and presenting it to the user. This makes it possible to provide a response to a request made by the user through voice input with high accuracy and speed. It is also possible to detect vehicle abnormalities in real time, generate specific countermeasures, and provide guidance to the user. This allows the user to operate the vehicle safely and smoothly, improving convenience.

[0776] "Means for capturing user input as audio" refers to devices and techniques for recording a user's spoken instructions or questions.

[0777] "Means for converting acquired audio into character string data" refers to the function of converting recorded audio data into text data using natural language processing technology.

[0778] "Means for sending converted string data to a generative model" refers to a function for sending text data via a network to a generative model on a server or cloud.

[0779] "Means for obtaining responses generated by a generative model" refers to the function of receiving response data provided by a generative AI model and storing it within the system.

[0780] "Means for converting the acquired response into speech and presenting it to the user" refers to a function for converting a response acquired as text data into speech using speech synthesis technology and allowing the user to hear it.

[0781] "Means for detecting vehicle abnormalities" refers to the function of using sensors and monitoring systems inside the vehicle to detect whether there are any abnormalities in each part or operating condition of the vehicle.

[0782] "Means for presenting specific countermeasures" refers to a function that generates specific actions and procedures that the user should take in response to a detected vehicle abnormality and provides that information to the user.

[0783] "Means for transmitting to a cloud server" refers to a communication function for transmitting data to a remote server via the Internet.

[0784] MODE FOR CARRYING OUT THE INVENTION

[0785] The present invention relates to a next-generation car navigation system that processes user input using a generative model. Specific embodiments of the present invention are described below.

[0786] 1. Voice input acquisition

[0787] When a user issues voice commands to the car navigation system, the device records the voice using its built-in microphone. The recorded voice data is temporarily stored in the device's memory. This step uses a standard microphone and voice recording software.

[0788] example:

[0789] When a user says, "Tell me where the nearest gas station is," the audio is recorded.

[0790] 2. Convert audio to text data

[0791] The recorded voice data is converted into text data using natural language processing technology. Specifically, speech recognition software is used. In this example, the Google Cloud Speech-to-Text API is used.

[0792] example:

[0793] The recorded voice saying "Tell me where the nearest gas station is" is converted into text data saying "Tell me where the nearest gas station is."

[0794] 3. Send user input to the generative model

[0795] The converted text data is sent from the device to a cloud server. The cloud server receives this text data and inputs it into a generative AI model. This generative AI model generates an appropriate response based on the user input. Data is sent and received using standard cloud communication technologies.

[0796] example:

[0797] The data sent from the device, "Tell me where the nearest gas station is," reaches the cloud server and is input as a prompt into the generative AI model.

[0798] 4. Retrieving the generated response

[0799] A generative AI model generates a response to user input, and the data is received by a cloud server. This data is then sent back to the device and stored locally. An example of a generative AI model is "OpenAI GPT-4."

[0800] example:

[0801] The generative AI model generates a response such as, "The nearest gas station is XX. It will take XX minutes to get there," and sends it to the device via the cloud server.

[0802] 5. Convert the response into audio and present it

[0803] The device converts the text data received from the server into speech using a speech synthesis engine, such as the Amazon Polly API, and presents the speech data to the user through the speaker.

[0804] example:

[0805] The device converts the received text data, such as "The nearest gas station is XX. It will take XX minutes to get there," into speech and plays it back to the user.

[0806] Specific examples

[0807] For navigation requests

[0808] User: "Where's the nearest gas station?"

[0809] Device: Records voice and uses voice recognition technology to convert it into text data such as "Tell me where the nearest gas station is."

[0810] Server: Input the text data into the generative AI model and obtain the generated response, "The nearest gas station is XX. It will take XX minutes to arrive."

[0811] Terminal: Converts the response into speech and presents it to the user.

[0812] Notification of vehicle abnormality and instructions on how to deal with it

[0813] Vehicle: Detects abnormalities such as low tire pressure.

[0814] Terminal: Sends abnormality information to the cloud server.

[0815] Server: Using a generative AI model, it generates a response such as, "Go to the nearest service station to check your tire pressure. You can do this by following these steps..."

[0816] Terminal: Converts the generated response into speech and presents it to the user.

[0817] In this way, the car navigation system of the present invention can significantly improve user convenience by responding quickly and accurately to user voice inputs and dealing appropriately with vehicle abnormalities.

[0818] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0819] Step 1: User speaks

[0820] Specific explanation and operation

[0821] The user issues commands or questions to the car navigation system, for example, "Tell me where the nearest gas station is."

[0822] Input: User's voice

[0823] Output: Recorded audio data (e.g. audio_input.wav)

[0824] The user speaks to the car navigation system, and the voice is recorded through a microphone.

[0825] Step 2: The device converts the audio into text data

[0826] Specific explanation and operation

[0827] The device converts the recorded voice data into text data using natural language processing technology.

[0828] Software used: Speech recognition software (e.g., "Google Cloud Speech-to-Text")

[0829] Input: Recorded audio data (audio_input.wav)

[0830] Output: Text data "Where is the nearest gas station?"

[0831] Processing: The device sends the audio_input.wav file to the speech recognition API and receives the text data.

[0832] Step 3: The device sends the string data to the generative model

[0833] Specific explanation and operation

[0834] The device sends the converted text data to a cloud server, which then inputs the text data into a generative AI model.

[0835] Input: Text data "Tell me the nearest gas station"

[0836] Output: The prompt sentence that is input to the generative model

[0837] Processing: The device sends text data to the server via an HTTP request, and the server receives the data and passes it to the generative AI model.

[0838] Step 4: The server gets the generated response

[0839] Specific explanation and operation

[0840] The generative AI model generates an appropriate response based on the user's prompt. The server retrieves the generated response data and sends it back to the device.

[0841] Software used: Generative AI models (e.g., "OpenAI GPT-4")

[0842] Input: Prompt "Where is the nearest gas station?"

[0843] Output: Text data "The nearest gas station is ____. It will take ____ minutes to get there."

[0844] Processing: The server receives the response from the generative AI model and sends the data to the device.

[0845] Step 5: The device converts the response into speech and presents it to the user.

[0846] Specific explanation and operation

[0847] The terminal converts the received text data into speech using a speech synthesis engine and presents it to the user.

[0848] Software used: Speech synthesis engine (e.g., "Amazon Polly")

[0849] Input: Text data: "The nearest gas station is ____. It will take ____ minutes to get there."

[0850] Output: Speech data: "The nearest gas station is ____. It will take ____ minutes to get there."

[0851] Processing: The device sends the text data to the speech synthesis API, and the generated speech data is played on the speaker.

[0852] This series of processing steps allows the user to operate the car navigation system through voice and receive prompt and accurate responses.

[0853] (Application example 1)

[0854] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0855] Conventional car navigation systems have limited ability to respond to user instructions and questions, resulting in a lack of interactivity. Furthermore, they lack the ability to provide specific solutions when vehicle abnormalities are detected. Furthermore, they lack the functionality to present multiple options in response to abstract user requests. To address these issues, the development of more advanced, user-friendly navigation systems is needed.

[0856] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0857] In this invention, the server includes means for collecting user input via voice and converting it into text data, means for sending the collected text data to a generative model and obtaining a text response returned from the generative model, means for converting the obtained text response into voice and presenting it to the user, and means for generating responses to user instructions and questions using the generative model and providing vehicle status confirmation, route guidance, and real-time traffic information. This allows the user to enjoy advanced and interactive navigation, as well as to obtain quick and specific countermeasures in the event of a vehicle abnormality, and further allows the user to receive multiple options even for abstract requests.

[0858] "User input" refers to information such as voice instructions or questions that a user gives to the system.

[0859] "Text data" refers to data that has been analyzed and converted into a character string format from voice input.

[0860] A "generative model" is an artificial intelligence technique used to generate appropriate responses to user input.

[0861] A "server" is a computer system that operates a generative model and generates and provides responses to user input.

[0862] "Vehicle status check" refers to monitoring and evaluating the current condition of a vehicle based on data obtained from various sensors and systems on the vehicle.

[0863] "Route guidance" refers to calculating and presenting the optimal route for a user to reach their destination.

[0864] "Real-time traffic information" means providing information on current road conditions, traffic volume, etc. immediately.

[0865] The "voice collection means" refers to a device or technology that acquires the user's voice using a microphone or the like.

[0866] "Text conversion means" refers to software or algorithms for converting acquired voice data into character string data.

[0867] A "response acquisition means" refers to a communication means or protocol for receiving text responses generated from a generative model.

[0868] "Speech conversion means" refers to technology or devices for converting text responses from a generative model into speech data.

[0869] An "abstract request" is a vague question or instruction that does not include specific information or instructions from the user.

[0870] This invention relates to a next-generation car navigation system that processes user input using generative models. When a user issues a command or question by voice, the system recognizes the voice, generates a response using a generative model, and provides it to the user as voice.

[0871] The system has the following configuration:

[0872] Hardware and software configuration

[0873] 1. Voice collection means: The user's voice is acquired using a microphone in the vehicle.

[0874] 2. Text conversion method: Use speech recognition software (e.g., speech_recognition library) to convert the captured audio into text data.

[0875] 3. Means for transmitting text data: A communication protocol is used to transmit text data to the generative model.

[0876] 4. Response generator: A generative model (e.g., the transformers library) responsible for natural language processing generates an appropriate text response.

[0877] 5. Speech conversion means: Generates voice data using a speech synthesis engine (e.g., pyttsx3 library) that converts the generated text response into speech and presents it to the user.

[0878] Program processing

[0879] The server constructs a system including the following means.

[0880] 1. Collecting user voice input: The device captures the user's voice through the microphone and saves it as voice data.

[0881] 2. Speech-to-text conversion: The device's speech recognition software converts this speech data into text data.

[0882] 3. Sending data to the generative model: The string data is sent to the server, which then inputs the data into the generative model.

[0883] 4. Obtaining and sending the response: The server obtains the response generated by the generative model and sends the text response back to the device.

[0884] 5. Generating and presenting a voice response: The device converts the text data into voice data using a voice synthesis engine and presents it to the user.

[0885] For example, if a user says "Where is the nearest restaurant?" while in a vehicle, the system recognizes the speech and converts it into text data. The converted string data is sent to the server, and the generative model generates a response such as "The nearest restaurant is ____. Travel time is ____ minutes." This generated text response is sent back from the server to the device, where it is converted into speech by a speech synthesis engine and presented to the user.

[0886] For example, the prompt text might look like this:

[0887] Prompt statement:

[0888] User: Where is the nearest restaurant?

[0889] Generative AI model input sentence: Where is the nearest restaurant?

[0890] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0891] Step 1:

[0892] Acquiring voice input

[0893] The device uses a microphone to collect the user's voice. When the user speaks in the vehicle, "Where is the nearest restaurant?", this voice is captured by the microphone as input data. The input is the user's voice data, and this voice data is passed to the device.

[0894] Step 2:

[0895] Converting audio data to text

[0896] The device processes the input voice data using voice recognition software (for example, the speech_recognition library) and converts the voice into text data. This voice recognition process converts the voice data into text data such as "Where is the nearest restaurant?" The output is string data.

[0897] Step 3:

[0898] Sending text data

[0899] The terminal sends the converted text data to the server. In order for the terminal to send the text data "Where is the nearest restaurant?" to the server, it transfers the data using a communication protocol. The input is text data, which is sent to the server as output.

[0900] Step 4:

[0901] Generating a response

[0902] The server inputs the received text data into a generative model to generate an appropriate response. The generative model (e.g., the transformers library) generates the response "The nearest restaurant is ____. Travel time is ____ minutes" based on the input text data "Where is the nearest restaurant?". The output is the generated text response.

[0903] Step 5:

[0904] Sending a Response

[0905] The server sends the text response generated from the generative model to the terminal. The server then sends the generated text response, "The nearest restaurant is ____. Travel time is ____ minutes," to the terminal using a communication protocol. The input is the generated text response, which is sent to the terminal as output.

[0906] Step 6:

[0907] Generate a voice response

[0908] The device converts the text response received from the server into voice data using a speech synthesis engine (for example, the pyttsx3 library). The speech synthesis engine converts the text data, such as "The nearest restaurant is ____. Travel time is ____ minutes," into voice data and outputs it to the user through the speaker. The input is text data, and the output is voice data.

[0909] Step 7:

[0910] Voice response presentation

[0911] The device presents the generated voice data to the user through a speaker. The user receives a voice prompt saying, "The nearest restaurant is ____. Travel time is ____ minutes." The input is the voice data, and the output is the voice prompt presented to the user.

[0912] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0913] The present invention relates to an advanced car navigation system that combines a generative model and an emotion engine. The following embodiments will explain how to specifically implement the system of the present invention.

[0914] 1. A way to capture user voice input

[0915] When a user speaks to the car navigation system, the device uses a microphone to record the voice. Here, voice input is acquired by the user speaking instructions, questions, requests, etc. to the car navigation system.

[0916] 2. A method for converting audio into text data

[0917] The device uses a voice recognition function to convert the recorded voice data into text data, using natural language processing technology to convert the spoken voice into appropriate text data.

[0918] 3. A method for recognizing user emotions using an emotion engine

[0919] The device or server analyzes the text data obtained from the speech recognition and the speech itself to identify the user's emotions. The emotion engine uses algorithms to determine the user's emotional state, such as whether they are stressed or relaxed.

[0920] 4. A means of sending user input to the generative model

[0921] The device sends the converted text data and the recognized emotion data to the server, which then inputs the received data into a generative model to generate an appropriate response.

[0922] 5. How to obtain the generated response

[0923] The server retrieves the response generated by the generative model and sends the corresponding text data back to the device. This response can include information requested by the user or notifications about the vehicle's status. The content and tone of the response can also be adjusted based on emotion data.

[0924] 6. A means of converting the response into speech and presenting it to the user

[0925] The device converts the text data received from the server into speech using a speech synthesis engine, and presents the converted speech response to the user, conveying information in a form that is easy for the user to understand.

[0926] Example: Navigation request

[0927] When a user speaks to a car navigation system saying, "Tell me where the nearest gas station is," the device's microphone records the speech and converts it into text data saying, "Tell me where the nearest gas station is" using speech recognition. This data is sent to the server, which processes the generative model to generate a response saying, "The nearest gas station is ____. It will take ____ minutes to get there," and sends it back to the device. The device then converts this response into speech and provides the user with audible guidance saying, "The nearest gas station is ____. It will take ____ minutes to get there."

[0928] Example: Notification of vehicle abnormality and instructions on how to deal with it

[0929] When the vehicle detects an abnormality, such as a notification that "tire pressure is low," the device sends that information to the server. The server uses the generative model to generate specific instructions, such as "Go to the nearest service station to check your tire pressure. You can check it by following the steps below...," and sends them back to the device. The device then converts these instructions into voice and provides the user with quick and specific instructions on how to deal with the problem.

[0930] Example: Responding according to the user's emotions

[0931] The emotion engine recognizes when a user is tired or stressed. For example, if a user says, "The roads are crowded and I'm frustrated," the device records the audio and the emotion engine recognizes the emotion of frustration. The server uses the generative model to generate an encouraging response such as, "Don't worry. We're almost there. Relax and drive safely." The device then converts this response into voice and delivers it to the user in a gentle tone.

[0932] In this way, the car navigation system of the present invention provides more interactive and advanced support than conventional systems, significantly improving user convenience and satisfaction. The introduction of an emotion engine enables flexible responses according to the user's mental state, providing a more intimate and secure driving environment.

[0933] The processing flow will be explained below.

[0934] Step 1:

[0935] The user speaks to the car navigation system. The user issues a voice request, such as "Tell me where the nearest gas station is."

[0936] Step 2:

[0937] The device records the user's voice. The device's microphone picks up what the user says and stores it as digital audio data.

[0938] Step 3:

[0939] The device converts the voice data into text data. The device uses a speech recognition engine to convert the recorded voice data into text. This text is in the format "Where is the nearest gas station?"

[0940] Step 4:

[0941] The device passes the voice data to the emotion engine, which then inputs the recorded voice data into the emotion engine and analyzes the user's emotional state.

[0942] Step 5:

[0943] The emotion engine recognizes emotions. The emotion engine analyzes the voice or text data and identifies emotions such as whether the user is stressed or relaxed.

[0944] Step 6:

[0945] The device sends text data and emotion data to the server, which then sends the data to the server as an HTTP POST request.

[0946] Step 7:

[0947] The server receives user input and emotion data. The server receives the text data and emotion data sent from the device and prepares them to be passed to the generative model.

[0948] Step 8:

[0949] The server provides user input and emotion data to the generative model, and instructs it to generate an appropriate response.

[0950] Step 9:

[0951] The generative model generates a response based on user input and emotion. Based on the identified emotion, the generative model generates a response with appropriate tone and content. For example, "Please stay calm. The nearest gas station is ____."

[0952] Step 10:

[0953] The server receives the generated response. The server receives the response generated by the generative model and prepares to send it back to the device.

[0954] Step 11:

[0955] The server sends a response back to the device. The server sends the generated text response back to the device in JSON format.

[0956] Step 12:

[0957] The device receives the response. The device receives the text response sent by the server and prepares to pass it to the speech synthesis engine.

[0958] Step 13:

[0959] The device converts the text response into speech. The device uses a speech synthesis engine to convert the text data into speech data.

[0960] Step 14:

[0961] The device presents a voice response to the user. The device plays the generated voice data through a speaker, guiding the user in a tone that takes into consideration the user's emotions, such as "The nearest gas station is ____. It will take ____ minutes to get there."

[0962] Example 2

[0963] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0964] Conventional car navigation systems can only provide guidance and information based on simple voice input from the user, and have the problem of being unable to flexibly respond to the user's emotional state or vehicle abnormalities. Furthermore, they lack the functionality to present appropriate options in response to abstract user requests. This can cause stress for users and reduces satisfaction.

[0965] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0966] In this invention, the server includes means for using a generative model to process user input, means for receiving voice input from the user and converting it into character string data, means for recognizing the user's emotion using an emotion engine, means for adjusting the content and tone of a response based on the user's emotion, means for sending the character string data to the generative model and obtaining a generated response, and means for converting the obtained response into voice and presenting it to the user. This makes it possible to provide flexible responses according to the user's emotional state, suggest specific ways to deal with vehicle abnormalities, and even provide multiple appropriate options for abstract requests.

[0967] A "generative model" is a program with an algorithm that generates appropriate responses based on user input information.

[0968] An "emotion engine" is software or algorithms that analyze voice and text data to identify a user's emotional state.

[0969] "Voice input" refers to the voice that a user speaks to a device, which is acquired as data via a microphone.

[0970] "Character string data" refers to data that has been converted from voice input into text format using voice recognition technology.

[0971] A "speech synthesis engine" is software or algorithms that convert text data so that it can be played as speech.

[0972] A "server" is a central computing device that processes data sent by users and generates the necessary responses.

[0973] A "terminal" is a device that is directly used by a user or a device with a user interface, which receives voice input and outputs voice.

[0974] This invention shows how to specifically implement an advanced car navigation system that combines a generative model and an emotion engine. The components of the system and their operation are described in detail below.

[0975] 1. A way to capture user voice input

[0976] When a user speaks to the car navigation system in the car, the device records the voice using a microphone built into the car. Voice input is acquired when the user speaks instructions, questions, requests, etc. to the car navigation system. For example, if a user says, "Tell me where the nearest gas station is," the voice is recorded. A built-in microphone is required as hardware.

[0977] 2. A method for converting audio into text data

[0978] The device converts the recorded voice data into text data using speech recognition software (e.g., Google Cloud Speech-to-Text). Natural language processing technology is used to convert the spoken voice into text data. During this conversion process, the voice waveform is analyzed to generate the appropriate text data. For example, the speech "Tell me where the nearest gas station is" is converted into text data "Tell me where the nearest gas station is."

[0979] 3. A method for recognizing user emotions using an emotion engine

[0980] The device or server analyzes the text data obtained from the speech recognition and the speech itself, and identifies the user's emotion using an emotion engine (e.g., IBM Watson Tone Analyzer). The emotion engine uses an algorithm to determine the user's emotional state, such as whether they are stressed or relaxed. For example, if a user says, "The roads are crowded and I'm getting annoyed," the emotion engine will recognize the emotion as "annoyed."

[0981] 4. A means of sending user input to the generative model

[0982] The device sends the converted text data and the recognized emotion data to the server. The server inputs this data into a generative AI model (e.g., OpenAI GPT-4) to generate an appropriate response. For example, in response to a request such as "Tell me where the nearest gas station is," the generative AI model generates a response such as "The nearest gas station is ____. It will take ____ minutes to get there."

[0983] 5. How to obtain the generated response

[0984] The server receives the response generated by the generative model and sends it back to the device as text data. This response includes the information requested by the user and notifications about the vehicle's status. It is also possible to adjust the content and tone of the response based on emotional data. Specifically, the server generates a response such as "It's okay. We'll arrive at our destination soon. Relax and drive safely," and sends it to the device.

[0985] 6. A means of converting the response into speech and presenting it to the user

[0986] The device converts the text data received from the server into speech using a speech synthesis engine (e.g., Amazon Polly). The speech synthesis engine converts the input text into a speech signal and outputs it from the speaker. This process allows the user to hear the generated response as audio. For example, the user may hear voice guidance such as, "The nearest gas station is ____. It will take ____ minutes to arrive."

[0987] Specific examples

[0988] 1. For navigation requests:

[0989] When a user speaks to a car navigation system saying, "Tell me where the nearest gas station is," the device's microphone records the speech and converts it into text data such as "Tell me where the nearest gas station is" using speech recognition. This data is sent to the server, which processes it using a generative AI model to generate a response such as "The nearest gas station is ____. It will take ____ minutes to get there," which is sent back to the device. The device converts this response into speech and provides the user with audible guidance such as, "The nearest gas station is ____. It will take ____ minutes to get there."

[0990] 2. Vehicle abnormality notification and instructions:

[0991] When the vehicle detects an abnormality, such as a notification that "tire pressure is low," the device sends that information to the server. The server uses the generative model to generate specific instructions, such as "Go to the nearest service station to check your tire pressure. You can check it by following the steps below...," and sends them back to the device. The device then converts these instructions into voice and provides the user with quick and specific instructions on how to deal with the problem.

[0992] 3. Responding according to the user's emotions:

[0993] The emotion engine recognizes when a user is tired or stressed. For example, if a user says, "The roads are crowded and I'm frustrated," the device records the audio and the emotion engine recognizes the emotion of frustration. The server uses the generative model to generate an encouraging response: "Don't worry. We're almost there. Relax and drive safely." The device then converts this response into speech and delivers it to the user in a gentle tone.

[0994] In this way, the car navigation system of the present invention provides more interactive and advanced support than conventional systems, significantly improving user convenience and satisfaction. The introduction of an emotion engine enables flexible responses according to the user's mental state, providing a more intimate and secure driving environment.

[0995] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0996] Step 1: Capture the user's voice input

[0997] When a user speaks to the car navigation system in the car, the device uses a built-in microphone to record the voice. The input is the user's natural voice, which is captured by the device as digital voice data. For example, if a user says, "Tell me where the nearest gas station is," that voice is recorded. The data output is digital voice data.

[0998] Step 2: Convert audio to text

[0999] The device converts the recorded voice data into text data using voice recognition software (e.g., Google Cloud Speech-to-Text). The input voice data is analyzed and converted into text data using its waveform information. This process transforms the voice signal into text data such as "Tell me where the nearest gas station is." The output data is text data corresponding to the voice data.

[1000] Step 3: Recognize user emotions using the emotion engine

[1001] The device or server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to analyze the text data and audio data obtained from speech recognition. The input is text data or raw audio data, and the emotion engine determines the emotional state. For example, if a user says, "The roads are crowded and I'm getting annoyed," the emotion engine analyzes the text data and audio data and recognizes the emotion of annoyance. The output data is the identified emotional information.

[1002] Step 4: Sending user input to the generative model

[1003] The device sends the converted text data and the recognized emotion data to the server. The input is text data and emotion data, which are sent to the server. The server inputs this data into a generative AI model (e.g., OpenAI GPT-4) and uses it as a prompt to generate an appropriate response to the user's request. For example, based on the request "Tell me where the nearest gas station is," the response generated is "The nearest gas station is XX. It will take XX minutes to get there." The output data is the text data of the generated response.

[1004] Step 5: Get the generated response

[1005] The server receives the response generated by the generative AI model and returns it to the device as text data. The input is the string data generated by the generative AI model, and an appropriate response is constructed based on this. The output data is the response string data. For example, the server might send a response to the device saying, "The nearest gas station is ____. It will take ____ minutes to arrive."

[1006] Step 6: Convert the response to speech and present it to the user

[1007] The device converts the text data received from the server into speech using a speech synthesis engine (e.g., Amazon Polly). The input is the text data of the response sent from the server, which the speech synthesis engine converts into a voice signal through a speech synthesis process. The device outputs this voice signal through a speaker and presents it to the user. The output data is a response in voice format. For example, the user is told aloud, "The nearest gas station is XX. It will take XX minutes to get there."

[1008] As described above, the car navigation system of the present invention realizes a series of processes that start with user voice input, combines a generative AI model and an emotion engine to provide advanced responses, and finally conveys information to the user through voice synthesis.

[1009] (Application example 2)

[1010] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1011] Conventional car navigation systems and shopping assistant applications simply convert user voice input into text and generate responses, but do not provide responses that take into account the user's mental state or emotions. As a result, it is difficult to provide an appropriate response that takes into account the user's emotional state, even when the user is confused, confused, or anxious. Furthermore, there has been no system that can recognize the user's emotions and provide appropriate product recommendations or advice in the shopping experience in a physical store.

[1012] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for processing user input using a generative model, means for receiving voice input from the user and converting it into character string data, means for recognizing the user's emotions, means for adjusting the generated response according to the user's emotional state, and means for acquiring the generated response, converting it into voice, and presenting it to the user. This makes it possible to recognize the user's emotions and generate and present flexible and appropriate responses accordingly. As a result, it is possible to improve user convenience and satisfaction and provide a more friendly and reassuring user experience.

[1013] A "generative model" is an algorithm that automatically generates appropriate responses and information based on user input data.

[1014] "User input" means data provided by a user for processing by the system, including voice input and text input.

[1015] "Voice input" is data provided by a user by speaking to the system through a microphone.

[1016] "Character string data" is data that is generated by analyzing voice input using voice recognition technology and converting it into text format.

[1017] The "means for recognizing emotions" is an algorithm for analyzing character string data or the voice itself to identify the user's emotional state.

[1018] A "means for tailoring a response" is a mechanism for modifying the generated response in response to the user's emotional state to provide a more appropriate, situation-specific response.

[1019] The "means for obtaining a response" is the process by which the system receives the response data generated by the generative model.

[1020] The "means for converting into voice" is a technology for converting a response in text format into voice data and presenting it in a form that the user can hear.

[1021] "User's emotional state" refers to the mental state or emotion the user is feeling (e.g., stressed, relaxed, confused, etc.).

[1022] The present invention relates to an advanced shopping assistant system that combines a generative model and an emotion recognition engine. Specific implementation methods of the invention will be described below with reference to the following embodiments.

[1023] Hardware Configuration

[1024] The main hardware components of this system are a smartphone and an internet connection. The smartphone is equipped with a microphone and a speaker, enabling voice interaction with the user.

[1025] Software Configuration

[1026] The software includes the following main components:

[1027] Speech recognition software: Google Speech-to-Text API

[1028] Emotion recognition engine: Transformer-based sentiment analysis model (e.g., RoBERTa)

[1029] Generative model: GPT-2

[1030] Text-to-speech software: AWS Polly

[1031] Communication library: A library for HTTP communication (e.g., requests)

[1032] Data processing and calculation

[1033] Acquiring and converting voice input

[1034] When a user speaks into a smartphone, the microphone picks up the audio, which is then converted into text data using the Google Speech-to-Text API.

[1035] emotion recognition

[1036] The converted text data is then used to recognize emotional states using a Transformer-based sentiment analysis model, which identifies the user's current mental state, such as stressed, relaxed, or confused.

[1037] Response Generation

[1038] The text data and the recognized emotion data are input into a generative model, which uses GPT-2 to generate an appropriate response to the user's input. The content and tone of the response are adjusted based on the emotion data.

[1039] Generate and present a voice response

[1040] The generated response is converted into speech using AWS Polly and presented to the user through the smartphone speaker.

[1041] Specific examples

[1042] Take the example of a user searching for a product in a physical store. The user speaks to their smartphone, saying, "Tell me more about this product." The smartphone's microphone picks up the audio, and the Google Speech-to-Text API converts the audio data into text. The converted text data is analyzed by a sentiment analysis engine to identify the user's questions and interests. Next, the generative model GPT-2 creates a response based on the user's request, such as, "This product is high quality and has the following features. We also recommend this new product." Finally, AWS Polly synthesizes the created response into speech and presents it to the user through the speaker.

[1043] Prompt Sentence Examples

[1044] "Tell me about the features of this product"

[1045] "Please tell me what products you recommend."

[1046] "What are the most popular items on the menu?"

[1047] As described above, the shopping assistant system of the present invention can provide a more personal shopping experience by providing flexible responses based on the user's voice input and emotional state.

[1048] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1049] Step 1: Getting voice input

[1050] The user speaks into the smartphone, and the smartphone's microphone captures the voice input. The specific input is the voice data that the user speaks into the microphone, which is output as voice data to be used in subsequent processing steps.

[1051] Step 2: Converting audio data to string data

[1052] The device converts the acquired voice data into text data using the Google Speech-to-Text API. The input is the acquired voice data, and the converted text data is output.

[1053] Step 3: Emotion Recognition

[1054] The device inputs the converted text data into an emotion recognition engine, which analyzes the text data to obtain emotion data. The input is text data, and the output is the user's emotional state (e.g., stressed, relaxed, confused).

[1055] Step 4: Input to the generative model

[1056] The device inputs text data and emotion data into a generative model (GPT-2) to generate an appropriate response. The input is text data and emotion data, and the generated response text is output.

[1057] Step 5: Adjust the response

[1058] The device adjusts the response text generated by the generative model based on the emotion data. The input is the generated response text and emotion data, and the device outputs the response text adjusted to match the user's emotional state.

[1059] Step 6: Transcribing responses

[1060] The device converts the tailored response text into speech data using AWS Polly. The input is the tailored response text, and the converted speech data is output.

[1061] Step 7: Present the voice response

[1062] The terminal presents the converted voice data to the user through the smartphone speaker. The input is the converted voice data, and the voice output to be presented to the user is obtained.

[1063] Through the above processing steps, an appropriate response that takes into account the user's emotions in response to the user's voice input is generated and presented in the form of voice.

[1064] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1065] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1066] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1067] [Fourth embodiment]

[1068] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1069] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1070] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1071] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1072] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1073] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1074] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1075] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1076] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1077] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1078] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1079] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1080] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1081] The present invention relates to a next-generation car navigation system that processes user input using a generative model. The following embodiments will explain how to specifically implement the system of the present invention.

[1082] 1. A way to capture user voice input

[1083] When a user speaks to the car navigation system, the device uses a microphone to record the voice. Here, voice input is acquired by the user speaking instructions, questions, requests, etc. to the car navigation system.

[1084] 2. A method for converting audio into text data

[1085] The device uses a voice recognition function to convert the recorded voice data into text data, using natural language processing technology to convert the spoken voice into appropriate text data.

[1086] 3. A means of sending user input to the generative model

[1087] The device sends the converted text data to the server. The server then inputs the received text data into a generative model to generate an appropriate response. A generative model is an artificial intelligence technology that uses natural language processing to generate responses to user input.

[1088] 4. How to retrieve the generated response

[1089] The server retrieves the response generated by the generative model and sends the corresponding text data back to the device, including the information requested by the user and notifications about the vehicle's status.

[1090] 5. A means of converting the response into speech and presenting it to the user

[1091] The device converts the text data received from the server into speech using a speech synthesis engine, and presents the converted speech response to the user, conveying information in a form that is easy for the user to understand.

[1092] Example: Navigation request

[1093] When a user speaks to a car navigation system saying, "Tell me where the nearest gas station is," the device's microphone records the speech and converts it into text data saying, "Tell me where the nearest gas station is" using speech recognition. This data is sent to the server, which processes the generative model to generate a response saying, "The nearest gas station is ____. It will take ____ minutes to get there," and sends it back to the device. The device then converts this response into speech and provides the user with audible guidance saying, "The nearest gas station is ____. It will take ____ minutes to get there."

[1094] Example: Notification of vehicle abnormality and instructions on how to deal with it

[1095] When the vehicle detects an abnormality, such as a notification that "tire pressure is low," the device sends that information to the server. The server uses the generative model to generate specific instructions, such as "Go to the nearest service station to check your tire pressure. You can check it by following the steps below...," and sends them back to the device. The device then converts these instructions into voice and provides the user with quick and specific instructions on how to deal with the problem.

[1096] In this way, the car navigation system of the present invention can provide interactive and advanced support compared to conventional systems, and can significantly improve user convenience.

[1097] The processing flow will be explained below.

[1098] Step 1:

[1099] The user speaks to the car navigation system. The user issues a voice request, such as "Tell me where the nearest gas station is."

[1100] Step 2:

[1101] The device records the user's voice. The device's microphone picks up what the user says and stores it as digital audio data.

[1102] Step 3:

[1103] The device converts the voice data into text data. The device uses a speech recognition engine to convert the recorded voice data into text. This text is in the format "Where is the nearest gas station?"

[1104] Step 4:

[1105] The terminal sends the string data to the server. The terminal then sends the converted text data to the server as an HTTP POST request.

[1106] Step 5:

[1107] The server receives user input. The server receives the string data sent from the device and prepares it to be passed to the generative model.

[1108] Step 6:

[1109] The server inputs user input into the generative model. The server inputs user text data into the generative model (e.g., a natural language processing model) and instructs it to generate an appropriate response.

[1110] Step 7:

[1111] The server receives the generated response. The generative model generates a response such as "The nearest gas station is ____. It will take ____ minutes to get there," and the server receives this.

[1112] Step 8:

[1113] The server sends a response back to the device. The server sends the generated text response back to the device in JSON format.

[1114] Step 9:

[1115] The device receives the response. The device receives the text response sent by the server and prepares to pass it to the speech synthesis engine.

[1116] Step 10:

[1117] The device converts the text response into speech. The device uses a speech synthesis engine to convert the text data into speech data.

[1118] Step 11:

[1119] The device presents a voice response to the user. The device plays the generated voice data through a speaker, informing the user, "The nearest gas station is ____. It will take ____ minutes to get there."

[1120] Example 1

[1121] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1122] Conventional car navigation systems have limitations in their ability to provide immediate responses to user voice input, making it difficult to quickly provide specific solutions to complex requests or vehicle abnormalities. In particular, when a user makes a request through voice input, it is important to accurately understand the request and provide appropriate information and instructions. However, existing technologies lack sufficient interactivity, resulting in a loss of user convenience.

[1123] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1124] In this invention, the server includes means for acquiring user input as voice, means for converting the acquired voice into character string data, means for transmitting the converted character string data to the generative model, means for acquiring a response generated by the generative model, and means for converting the acquired response into voice and presenting it to the user. This makes it possible to provide a response to a request made by the user through voice input with high accuracy and speed. It is also possible to detect vehicle abnormalities in real time, generate specific countermeasures, and provide guidance to the user. This allows the user to operate the vehicle safely and smoothly, improving convenience.

[1125] "Means for capturing user input as audio" refers to devices and techniques for recording a user's spoken instructions or questions.

[1126] "Means for converting acquired audio into character string data" refers to the function of converting recorded audio data into text data using natural language processing technology.

[1127] "Means for sending converted string data to a generative model" refers to a function for sending text data via a network to a generative model on a server or cloud.

[1128] "Means for obtaining responses generated by a generative model" refers to the function of receiving response data provided by a generative AI model and storing it within the system.

[1129] "Means for converting the acquired response into speech and presenting it to the user" refers to a function for converting a response acquired as text data into speech using speech synthesis technology and allowing the user to hear it.

[1130] "Means for detecting vehicle abnormalities" refers to the function of using sensors and monitoring systems inside the vehicle to detect whether there are any abnormalities in each part or operating condition of the vehicle.

[1131] "Means for presenting specific countermeasures" refers to a function that generates specific actions and procedures that the user should take in response to a detected vehicle abnormality and provides that information to the user.

[1132] "Means for transmitting to a cloud server" refers to a communication function for transmitting data to a remote server via the Internet.

[1133] MODE FOR CARRYING OUT THE INVENTION

[1134] The present invention relates to a next-generation car navigation system that processes user input using a generative model. Specific embodiments of the present invention are described below.

[1135] 1. Voice input acquisition

[1136] When a user issues voice commands to the car navigation system, the device records the voice using its built-in microphone. The recorded voice data is temporarily stored in the device's memory. This step uses a standard microphone and voice recording software.

[1137] example:

[1138] When a user says, "Tell me where the nearest gas station is," the audio is recorded.

[1139] 2. Convert audio to text data

[1140] The recorded voice data is converted into text data using natural language processing technology. Specifically, speech recognition software is used. In this example, the Google Cloud Speech-to-Text API is used.

[1141] example:

[1142] The recorded voice saying "Tell me where the nearest gas station is" is converted into text data saying "Tell me where the nearest gas station is."

[1143] 3. Send user input to the generative model

[1144] The converted text data is sent from the device to a cloud server. The cloud server receives this text data and inputs it into a generative AI model. This generative AI model generates an appropriate response based on the user input. Data is sent and received using standard cloud communication technologies.

[1145] example:

[1146] The data sent from the device, "Tell me where the nearest gas station is," reaches the cloud server and is input as a prompt into the generative AI model.

[1147] 4. Retrieving the generated response

[1148] A generative AI model generates a response to user input, and the data is received by a cloud server. This data is then sent back to the device and stored locally. An example of a generative AI model is "OpenAI GPT-4."

[1149] example:

[1150] The generative AI model generates a response such as, "The nearest gas station is XX. It will take XX minutes to get there," and sends it to the device via the cloud server.

[1151] 5. Convert the response into audio and present it

[1152] The device converts the text data received from the server into speech using a speech synthesis engine, such as the Amazon Polly API, and presents the speech data to the user through the speaker.

[1153] example:

[1154] The device converts the received text data, such as "The nearest gas station is XX. It will take XX minutes to get there," into speech and plays it back to the user.

[1155] Specific examples

[1156] For navigation requests

[1157] User: "Where's the nearest gas station?"

[1158] Device: Records voice and uses voice recognition technology to convert it into text data such as "Tell me where the nearest gas station is."

[1159] Server: Input the text data into the generative AI model and obtain the generated response, "The nearest gas station is XX. It will take XX minutes to arrive."

[1160] Terminal: Converts the response into speech and presents it to the user.

[1161] Notification of vehicle abnormality and instructions on how to deal with it

[1162] Vehicle: Detects abnormalities such as low tire pressure.

[1163] Terminal: Sends abnormality information to the cloud server.

[1164] Server: Using a generative AI model, it generates a response such as, "Go to the nearest service station to check your tire pressure. You can do this by following these steps..."

[1165] Terminal: Converts the generated response into speech and presents it to the user.

[1166] In this way, the car navigation system of the present invention can significantly improve user convenience by responding quickly and accurately to user voice inputs and dealing appropriately with vehicle abnormalities.

[1167] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1168] Step 1: User speaks

[1169] Specific explanation and operation

[1170] The user issues commands or questions to the car navigation system, for example, "Tell me where the nearest gas station is."

[1171] Input: User's voice

[1172] Output: Recorded audio data (e.g. audio_input.wav)

[1173] The user speaks to the car navigation system, and the voice is recorded through a microphone.

[1174] Step 2: The device converts the audio into text data

[1175] Specific explanation and operation

[1176] The device converts the recorded voice data into text data using natural language processing technology.

[1177] Software used: Speech recognition software (e.g., "Google Cloud Speech-to-Text")

[1178] Input: Recorded audio data (audio_input.wav)

[1179] Output: Text data "Where is the nearest gas station?"

[1180] Processing: The device sends the audio_input.wav file to the speech recognition API and receives the text data.

[1181] Step 3: The device sends the string data to the generative model

[1182] Specific explanation and operation

[1183] The device sends the converted text data to a cloud server, which then inputs the text data into a generative AI model.

[1184] Input: Text data "Tell me the nearest gas station"

[1185] Output: The prompt sentence that is input to the generative model

[1186] Processing: The device sends text data to the server via an HTTP request, and the server receives the data and passes it to the generative AI model.

[1187] Step 4: The server gets the generated response

[1188] Specific explanation and operation

[1189] The generative AI model generates an appropriate response based on the user's prompt. The server retrieves the generated response data and sends it back to the device.

[1190] Software used: Generative AI models (e.g., "OpenAI GPT-4")

[1191] Input: Prompt "Where is the nearest gas station?"

[1192] Output: Text data "The nearest gas station is ____. It will take ____ minutes to get there."

[1193] Processing: The server receives the response from the generative AI model and sends the data to the device.

[1194] Step 5: The device converts the response into speech and presents it to the user.

[1195] Specific explanation and operation

[1196] The terminal converts the received text data into speech using a speech synthesis engine and presents it to the user.

[1197] Software used: Speech synthesis engine (e.g., "Amazon Polly")

[1198] Input: Text data: "The nearest gas station is ____. It will take ____ minutes to get there."

[1199] Output: Speech data: "The nearest gas station is ____. It will take ____ minutes to get there."

[1200] Processing: The device sends the text data to the speech synthesis API, and the generated speech data is played on the speaker.

[1201] This series of processing steps allows the user to operate the car navigation system through voice and receive prompt and accurate responses.

[1202] (Application example 1)

[1203] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1204] Conventional car navigation systems have limited ability to respond to user instructions and questions, resulting in a lack of interactivity. Furthermore, they lack the ability to provide specific solutions when vehicle abnormalities are detected. Furthermore, they lack the functionality to present multiple options in response to abstract user requests. To address these issues, the development of more advanced, user-friendly navigation systems is needed.

[1205] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1206] In this invention, the server includes means for collecting user input via voice and converting it into text data, means for sending the collected text data to a generative model and obtaining a text response returned from the generative model, means for converting the obtained text response into voice and presenting it to the user, and means for generating responses to user instructions and questions using the generative model and providing vehicle status confirmation, route guidance, and real-time traffic information. This allows the user to enjoy advanced and interactive navigation, as well as to obtain quick and specific countermeasures in the event of a vehicle abnormality, and further allows the user to receive multiple options even for abstract requests.

[1207] "User input" refers to information such as voice instructions or questions that a user gives to the system.

[1208] "Text data" refers to data that has been analyzed and converted into a character string format from voice input.

[1209] A "generative model" is an artificial intelligence technique used to generate appropriate responses to user input.

[1210] A "server" is a computer system that operates a generative model and generates and provides responses to user input.

[1211] "Vehicle status check" refers to monitoring and evaluating the current condition of a vehicle based on data obtained from various sensors and systems on the vehicle.

[1212] "Route guidance" refers to calculating and presenting the optimal route for a user to reach their destination.

[1213] "Real-time traffic information" means providing information on current road conditions, traffic volume, etc. immediately.

[1214] The "voice collection means" refers to a device or technology that acquires the user's voice using a microphone or the like.

[1215] "Text conversion means" refers to software or algorithms for converting acquired voice data into character string data.

[1216] A "response acquisition means" refers to a communication means or protocol for receiving text responses generated from a generative model.

[1217] "Speech conversion means" refers to technology or devices for converting text responses from a generative model into speech data.

[1218] An "abstract request" is a vague question or instruction that does not include specific information or instructions from the user.

[1219] This invention relates to a next-generation car navigation system that processes user input using generative models. When a user issues a command or question by voice, the system recognizes the voice, generates a response using a generative model, and provides it to the user as voice.

[1220] The system has the following configuration:

[1221] Hardware and software configuration

[1222] 1. Voice collection means: The user's voice is acquired using a microphone in the vehicle.

[1223] 2. Text conversion method: Use speech recognition software (e.g., speech_recognition library) to convert the captured audio into text data.

[1224] 3. Means for transmitting text data: A communication protocol is used to transmit text data to the generative model.

[1225] 4. Response generator: A generative model (e.g., the transformers library) responsible for natural language processing generates an appropriate text response.

[1226] 5. Speech conversion means: Generates voice data using a speech synthesis engine (e.g., pyttsx3 library) that converts the generated text response into speech and presents it to the user.

[1227] Program processing

[1228] The server constructs a system including the following means.

[1229] 1. Collecting user voice input: The device captures the user's voice through the microphone and saves it as voice data.

[1230] 2. Speech-to-text conversion: The device's speech recognition software converts this speech data into text data.

[1231] 3. Sending data to the generative model: The string data is sent to the server, which then inputs the data into the generative model.

[1232] 4. Obtaining and sending the response: The server obtains the response generated by the generative model and sends the text response back to the device.

[1233] 5. Generating and presenting a voice response: The device converts the text data into voice data using a voice synthesis engine and presents it to the user.

[1234] For example, if a user says "Where is the nearest restaurant?" while in a vehicle, the system recognizes the speech and converts it into text data. The converted string data is sent to the server, and the generative model generates a response such as "The nearest restaurant is ____. Travel time is ____ minutes." This generated text response is sent back from the server to the device, where it is converted into speech by a speech synthesis engine and presented to the user.

[1235] For example, the prompt text might look like this:

[1236] Prompt statement:

[1237] User: Where is the nearest restaurant?

[1238] Generative AI model input sentence: Where is the nearest restaurant?

[1239] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1240] Step 1:

[1241] Acquiring voice input

[1242] The device uses a microphone to collect the user's voice. When the user speaks in the vehicle, "Where is the nearest restaurant?", this voice is captured by the microphone as input data. The input is the user's voice data, and this voice data is passed to the device.

[1243] Step 2:

[1244] Converting audio data to text

[1245] The device processes the input voice data using voice recognition software (for example, the speech_recognition library) and converts the voice into text data. This voice recognition process converts the voice data into text data such as "Where is the nearest restaurant?" The output is string data.

[1246] Step 3:

[1247] Sending text data

[1248] The terminal sends the converted text data to the server. In order for the terminal to send the text data "Where is the nearest restaurant?" to the server, it transfers the data using a communication protocol. The input is text data, which is sent to the server as output.

[1249] Step 4:

[1250] Generating a response

[1251] The server inputs the received text data into a generative model to generate an appropriate response. The generative model (e.g., the transformers library) generates the response "The nearest restaurant is ____. Travel time is ____ minutes" based on the input text data "Where is the nearest restaurant?". The output is the generated text response.

[1252] Step 5:

[1253] Sending a Response

[1254] The server sends the text response generated from the generative model to the terminal. The server then sends the generated text response, "The nearest restaurant is ____. Travel time is ____ minutes," to the terminal using a communication protocol. The input is the generated text response, which is sent to the terminal as output.

[1255] Step 6:

[1256] Generate a voice response

[1257] The device converts the text response received from the server into voice data using a speech synthesis engine (for example, the pyttsx3 library). The speech synthesis engine converts the text data, such as "The nearest restaurant is ____. Travel time is ____ minutes," into voice data and outputs it to the user through the speaker. The input is text data, and the output is voice data.

[1258] Step 7:

[1259] Voice response presentation

[1260] The device presents the generated voice data to the user through a speaker. The user receives a voice prompt saying, "The nearest restaurant is ____. Travel time is ____ minutes." The input is the voice data, and the output is the voice prompt presented to the user.

[1261] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1262] The present invention relates to an advanced car navigation system that combines a generative model and an emotion engine. The following embodiments will explain how to specifically implement the system of the present invention.

[1263] 1. A way to capture user voice input

[1264] When a user speaks to the car navigation system, the device uses a microphone to record the voice. Here, voice input is acquired by the user speaking instructions, questions, requests, etc. to the car navigation system.

[1265] 2. A method for converting audio into text data

[1266] The device uses a voice recognition function to convert the recorded voice data into text data, using natural language processing technology to convert the spoken voice into appropriate text data.

[1267] 3. A method for recognizing user emotions using an emotion engine

[1268] The device or server analyzes the text data obtained from the speech recognition and the speech itself to identify the user's emotions. The emotion engine uses algorithms to determine the user's emotional state, such as whether they are stressed or relaxed.

[1269] 4. A means of sending user input to the generative model

[1270] The device sends the converted text data and the recognized emotion data to the server, which then inputs the received data into a generative model to generate an appropriate response.

[1271] 5. How to obtain the generated response

[1272] The server retrieves the response generated by the generative model and sends the corresponding text data back to the device. This response can include information requested by the user or notifications about the vehicle's status. The content and tone of the response can also be adjusted based on emotion data.

[1273] 6. A means of converting the response into speech and presenting it to the user

[1274] The device converts the text data received from the server into speech using a speech synthesis engine, and presents the converted speech response to the user, conveying information in a form that is easy for the user to understand.

[1275] Example: Navigation request

[1276] When a user speaks to a car navigation system saying, "Tell me where the nearest gas station is," the device's microphone records the speech and converts it into text data saying, "Tell me where the nearest gas station is" using speech recognition. This data is sent to the server, which processes the generative model to generate a response saying, "The nearest gas station is ____. It will take ____ minutes to get there," and sends it back to the device. The device then converts this response into speech and provides the user with audible guidance saying, "The nearest gas station is ____. It will take ____ minutes to get there."

[1277] Example: Notification of vehicle abnormality and instructions on how to deal with it

[1278] When the vehicle detects an abnormality, such as a notification that "tire pressure is low," the device sends that information to the server. The server uses the generative model to generate specific instructions, such as "Go to the nearest service station to check your tire pressure. You can check it by following the steps below...," and sends them back to the device. The device then converts these instructions into voice and provides the user with quick and specific instructions on how to deal with the problem.

[1279] Example: Responding according to the user's emotions

[1280] The emotion engine recognizes when a user is tired or stressed. For example, if a user says, "The roads are crowded and I'm frustrated," the device records the audio and the emotion engine recognizes the emotion of frustration. The server uses the generative model to generate an encouraging response such as, "Don't worry. We're almost there. Relax and drive safely." The device then converts this response into voice and delivers it to the user in a gentle tone.

[1281] In this way, the car navigation system of the present invention provides more interactive and advanced support than conventional systems, significantly improving user convenience and satisfaction. The introduction of an emotion engine enables flexible responses according to the user's mental state, providing a more intimate and secure driving environment.

[1282] The processing flow will be explained below.

[1283] Step 1:

[1284] The user speaks to the car navigation system. The user issues a voice request, such as "Tell me where the nearest gas station is."

[1285] Step 2:

[1286] The device records the user's voice. The device's microphone picks up what the user says and stores it as digital audio data.

[1287] Step 3:

[1288] The device converts the voice data into text data. The device uses a speech recognition engine to convert the recorded voice data into text. This text is in the format "Where is the nearest gas station?"

[1289] Step 4:

[1290] The device passes the voice data to the emotion engine, which then inputs the recorded voice data into the emotion engine and analyzes the user's emotional state.

[1291] Step 5:

[1292] The emotion engine recognizes emotions. The emotion engine analyzes the voice or text data and identifies emotions such as whether the user is stressed or relaxed.

[1293] Step 6:

[1294] The device sends text data and emotion data to the server, which then sends the data to the server as an HTTP POST request.

[1295] Step 7:

[1296] The server receives user input and emotion data. The server receives the text data and emotion data sent from the device and prepares them to be passed to the generative model.

[1297] Step 8:

[1298] The server provides user input and emotion data to the generative model, and instructs it to generate an appropriate response.

[1299] Step 9:

[1300] The generative model generates a response based on user input and emotion. Based on the identified emotion, the generative model generates a response with appropriate tone and content. For example, "Please stay calm. The nearest gas station is ____."

[1301] Step 10:

[1302] The server receives the generated response. The server receives the response generated by the generative model and prepares to send it back to the device.

[1303] Step 11:

[1304] The server sends a response back to the device. The server sends the generated text response back to the device in JSON format.

[1305] Step 12:

[1306] The device receives the response. The device receives the text response sent by the server and prepares to pass it to the speech synthesis engine.

[1307] Step 13:

[1308] The device converts the text response into speech. The device uses a speech synthesis engine to convert the text data into speech data.

[1309] Step 14:

[1310] The device presents a voice response to the user. The device plays the generated voice data through a speaker, guiding the user in a tone that takes into consideration the user's emotions, such as "The nearest gas station is ____. It will take ____ minutes to get there."

[1311] Example 2

[1312] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1313] Conventional car navigation systems can only provide guidance and information based on simple voice input from the user, and have the problem of being unable to flexibly respond to the user's emotional state or vehicle abnormalities. Furthermore, they lack the functionality to present appropriate options in response to abstract user requests. This can cause stress for users and reduces satisfaction.

[1314] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1315] In this invention, the server includes means for using a generative model to process user input, means for receiving voice input from the user and converting it into character string data, means for recognizing the user's emotion using an emotion engine, means for adjusting the content and tone of a response based on the user's emotion, means for sending the character string data to the generative model and obtaining a generated response, and means for converting the obtained response into voice and presenting it to the user. This makes it possible to provide flexible responses according to the user's emotional state, suggest specific ways to deal with vehicle abnormalities, and even provide multiple appropriate options for abstract requests.

[1316] A "generative model" is a program with an algorithm that generates appropriate responses based on user input information.

[1317] An "emotion engine" is software or algorithms that analyze voice and text data to identify a user's emotional state.

[1318] "Voice input" refers to the voice that a user speaks to a device, which is acquired as data via a microphone.

[1319] "Character string data" refers to data that has been converted from voice input into text format using voice recognition technology.

[1320] A "speech synthesis engine" is software or algorithms that convert text data so that it can be played as speech.

[1321] A "server" is a central computing device that processes data sent by users and generates the necessary responses.

[1322] A "terminal" is a device that is directly used by a user or a device with a user interface, which receives voice input and outputs voice.

[1323] This invention shows how to specifically implement an advanced car navigation system that combines a generative model and an emotion engine. The components of the system and their operation are described in detail below.

[1324] 1. A way to capture user voice input

[1325] When a user speaks to the car navigation system in the car, the device records the voice using a microphone built into the car. Voice input is acquired when the user speaks instructions, questions, requests, etc. to the car navigation system. For example, if a user says, "Tell me where the nearest gas station is," the voice is recorded. A built-in microphone is required as hardware.

[1326] 2. A method for converting audio into text data

[1327] The device converts the recorded voice data into text data using speech recognition software (e.g., Google Cloud Speech-to-Text). Natural language processing technology is used to convert the spoken voice into text data. During this conversion process, the voice waveform is analyzed to generate the appropriate text data. For example, the speech "Tell me where the nearest gas station is" is converted into text data "Tell me where the nearest gas station is."

[1328] 3. A method for recognizing user emotions using an emotion engine

[1329] The device or server analyzes the text data obtained from the speech recognition and the speech itself, and identifies the user's emotion using an emotion engine (e.g., IBM Watson Tone Analyzer). The emotion engine uses an algorithm to determine the user's emotional state, such as whether they are stressed or relaxed. For example, if a user says, "The roads are crowded and I'm getting annoyed," the emotion engine will recognize the emotion as "annoyed."

[1330] 4. A means of sending user input to the generative model

[1331] The device sends the converted text data and the recognized emotion data to the server. The server inputs this data into a generative AI model (e.g., OpenAI GPT-4) to generate an appropriate response. For example, in response to a request such as "Tell me where the nearest gas station is," the generative AI model generates a response such as "The nearest gas station is ____. It will take ____ minutes to get there."

[1332] 5. How to obtain the generated response

[1333] The server receives the response generated by the generative model and sends it back to the device as text data. This response includes the information requested by the user and notifications about the vehicle's status. It is also possible to adjust the content and tone of the response based on emotional data. Specifically, the server generates a response such as "It's okay. We'll arrive at our destination soon. Relax and drive safely," and sends it to the device.

[1334] 6. A means of converting the response into speech and presenting it to the user

[1335] The device converts the text data received from the server into speech using a speech synthesis engine (e.g., Amazon Polly). The speech synthesis engine converts the input text into a speech signal and outputs it from the speaker. This process allows the user to hear the generated response as audio. For example, the user may hear voice guidance such as, "The nearest gas station is ____. It will take ____ minutes to arrive."

[1336] Specific examples

[1337] 1. For navigation requests:

[1338] When a user speaks to a car navigation system saying, "Tell me where the nearest gas station is," the device's microphone records the speech and converts it into text data such as "Tell me where the nearest gas station is" using speech recognition. This data is sent to the server, which processes it using a generative AI model to generate a response such as "The nearest gas station is ____. It will take ____ minutes to get there," which is sent back to the device. The device converts this response into speech and provides the user with audible guidance such as, "The nearest gas station is ____. It will take ____ minutes to get there."

[1339] 2. Vehicle abnormality notification and instructions:

[1340] When the vehicle detects an abnormality, such as a notification that "tire pressure is low," the device sends that information to the server. The server uses the generative model to generate specific instructions, such as "Go to the nearest service station to check your tire pressure. You can check it by following the steps below...," and sends them back to the device. The device then converts these instructions into voice and provides the user with quick and specific instructions on how to deal with the problem.

[1341] 3. Responding according to the user's emotions:

[1342] The emotion engine recognizes when a user is tired or stressed. For example, if a user says, "The roads are crowded and I'm frustrated," the device records the audio and the emotion engine recognizes the emotion of frustration. The server uses the generative model to generate an encouraging response: "Don't worry. We're almost there. Relax and drive safely." The device then converts this response into speech and delivers it to the user in a gentle tone.

[1343] In this way, the car navigation system of the present invention provides more interactive and advanced support than conventional systems, significantly improving user convenience and satisfaction. The introduction of an emotion engine enables flexible responses according to the user's mental state, providing a more intimate and secure driving environment.

[1344] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1345] Step 1: Capture the user's voice input

[1346] When a user speaks to the car navigation system in the car, the device uses a built-in microphone to record the voice. The input is the user's natural voice, which is captured by the device as digital voice data. For example, if a user says, "Tell me where the nearest gas station is," that voice is recorded. The data output is digital voice data.

[1347] Step 2: Convert audio to text

[1348] The device converts the recorded voice data into text data using voice recognition software (e.g., Google Cloud Speech-to-Text). The input voice data is analyzed and converted into text data using its waveform information. This process transforms the voice signal into text data such as "Tell me where the nearest gas station is." The output data is text data corresponding to the voice data.

[1349] Step 3: Recognize user emotions using the emotion engine

[1350] The device or server uses an emotion engine (e.g., IBM Watson Tone Analyzer) to analyze the text data and audio data obtained from speech recognition. The input is text data or raw audio data, and the emotion engine determines the emotional state. For example, if a user says, "The roads are crowded and I'm getting annoyed," the emotion engine analyzes the text data and audio data and recognizes the emotion of annoyance. The output data is the identified emotional information.

[1351] Step 4: Sending user input to the generative model

[1352] The device sends the converted text data and the recognized emotion data to the server. The input is text data and emotion data, which are sent to the server. The server inputs this data into a generative AI model (e.g., OpenAI GPT-4) and uses it as a prompt to generate an appropriate response to the user's request. For example, based on the request "Tell me where the nearest gas station is," the response generated is "The nearest gas station is XX. It will take XX minutes to get there." The output data is the text data of the generated response.

[1353] Step 5: Get the generated response

[1354] The server receives the response generated by the generative AI model and returns it to the device as text data. The input is the string data generated by the generative AI model, and an appropriate response is constructed based on this. The output data is the response string data. For example, the server might send a response to the device saying, "The nearest gas station is ____. It will take ____ minutes to arrive."

[1355] Step 6: Convert the response to speech and present it to the user

[1356] The device converts the text data received from the server into speech using a speech synthesis engine (e.g., Amazon Polly). The input is the text data of the response sent from the server, which the speech synthesis engine converts into a voice signal through a speech synthesis process. The device outputs this voice signal through a speaker and presents it to the user. The output data is a response in voice format. For example, the user is told aloud, "The nearest gas station is XX. It will take XX minutes to get there."

[1357] As described above, the car navigation system of the present invention realizes a series of processes that start with user voice input, combines a generative AI model and an emotion engine to provide advanced responses, and finally conveys information to the user through voice synthesis.

[1358] (Application example 2)

[1359] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1360] Conventional car navigation systems and shopping assistant applications simply convert user voice input into text and generate responses, but do not provide responses that take into account the user's mental state or emotions. As a result, it is difficult to provide an appropriate response that takes into account the user's emotional state, even when the user is confused, confused, or anxious. Furthermore, there has been no system that can recognize the user's emotions and provide appropriate product recommendations or advice in the shopping experience in a physical store.

[1361] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for processing user input using a generative model, means for receiving voice input from the user and converting it into character string data, means for recognizing the user's emotions, means for adjusting the generated response according to the user's emotional state, and means for acquiring the generated response, converting it into voice, and presenting it to the user. This makes it possible to recognize the user's emotions and generate and present flexible and appropriate responses accordingly. As a result, it is possible to improve user convenience and satisfaction and provide a more friendly and reassuring user experience.

[1362] A "generative model" is an algorithm that automatically generates appropriate responses and information based on user input data.

[1363] "User input" means data provided by a user for processing by the system, including voice input and text input.

[1364] "Voice input" is data provided by a user by speaking to the system through a microphone.

[1365] "Character string data" is data that is generated by analyzing voice input using voice recognition technology and converting it into text format.

[1366] The "means for recognizing emotions" is an algorithm for analyzing character string data or the voice itself to identify the user's emotional state.

[1367] A "means for tailoring a response" is a mechanism for modifying the generated response in response to the user's emotional state to provide a more appropriate, situation-specific response.

[1368] The "means for obtaining a response" is the process by which the system receives the response data generated by the generative model.

[1369] The "means for converting into voice" is a technology for converting a response in text format into voice data and presenting it in a form that the user can hear.

[1370] "User's emotional state" refers to the mental state or emotion the user is feeling (e.g., stressed, relaxed, confused, etc.).

[1371] The present invention relates to an advanced shopping assistant system that combines a generative model and an emotion recognition engine. Specific implementation methods of the invention will be described below with reference to the following embodiments.

[1372] Hardware Configuration

[1373] The main hardware components of this system are a smartphone and an internet connection. The smartphone is equipped with a microphone and a speaker, enabling voice interaction with the user.

[1374] Software Configuration

[1375] The software includes the following main components:

[1376] Speech recognition software: Google Speech-to-Text API

[1377] Emotion recognition engine: Transformer-based sentiment analysis model (e.g., RoBERTa)

[1378] Generative model: GPT-2

[1379] Text-to-speech software: AWS Polly

[1380] Communication library: A library for HTTP communication (e.g., requests)

[1381] Data processing and calculation

[1382] Acquiring and converting voice input

[1383] When a user speaks into a smartphone, the microphone picks up the audio, which is then converted into text data using the Google Speech-to-Text API.

[1384] emotion recognition

[1385] The converted text data is then used to recognize emotional states using a Transformer-based sentiment analysis model, which identifies the user's current mental state, such as stressed, relaxed, or confused.

[1386] Response Generation

[1387] The text data and the recognized emotion data are input into a generative model, which uses GPT-2 to generate an appropriate response to the user's input. The content and tone of the response are adjusted based on the emotion data.

[1388] Generate and present a voice response

[1389] The generated response is converted into speech using AWS Polly and presented to the user through the smartphone speaker.

[1390] Specific examples

[1391] Take the example of a user searching for a product in a physical store. The user speaks to their smartphone, saying, "Tell me more about this product." The smartphone's microphone picks up the audio, and the Google Speech-to-Text API converts the audio data into text. The converted text data is analyzed by a sentiment analysis engine to identify the user's questions and interests. Next, the generative model GPT-2 creates a response based on the user's request, such as, "This product is high quality and has the following features. We also recommend this new product." Finally, AWS Polly synthesizes the created response into speech and presents it to the user through the speaker.

[1392] Prompt Sentence Examples

[1393] "Tell me about the features of this product"

[1394] "Please tell me what products you recommend."

[1395] "What are the most popular items on the menu?"

[1396] As described above, the shopping assistant system of the present invention can provide a more personal shopping experience by providing flexible responses based on the user's voice input and emotional state.

[1397] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1398] Step 1: Getting voice input

[1399] The user speaks into the smartphone, and the smartphone's microphone captures the voice input. The specific input is the voice data that the user speaks into the microphone, which is output as voice data to be used in subsequent processing steps.

[1400] Step 2: Converting audio data to string data

[1401] The device converts the acquired voice data into text data using the Google Speech-to-Text API. The input is the acquired voice data, and the converted text data is output.

[1402] Step 3: Emotion Recognition

[1403] The device inputs the converted text data into an emotion recognition engine, which analyzes the text data to obtain emotion data. The input is text data, and the output is the user's emotional state (e.g., stressed, relaxed, confused).

[1404] Step 4: Input to the generative model

[1405] The device inputs text data and emotion data into a generative model (GPT-2) to generate an appropriate response. The input is text data and emotion data, and the generated response text is output.

[1406] Step 5: Adjust the response

[1407] The device adjusts the response text generated by the generative model based on the emotion data. The input is the generated response text and emotion data, and the device outputs the response text adjusted to match the user's emotional state.

[1408] Step 6: Transcribing responses

[1409] The device converts the tailored response text into speech data using AWS Polly. The input is the tailored response text, and the converted speech data is output.

[1410] Step 7: Present the voice response

[1411] The terminal presents the converted voice data to the user through the smartphone speaker. The input is the converted voice data, and the voice output to be presented to the user is obtained.

[1412] Through the above processing steps, an appropriate response that takes into account the user's emotions in response to the user's voice input is generated and presented in the form of voice.

[1413] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1414] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1415] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1416] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1417] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1418] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1419] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1420] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1421] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1422] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1423] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1424] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1425] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1426] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1427] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1428] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1429] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1430] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1431] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1432] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1433] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1434] The following is further disclosed regarding the above embodiment.

[1435] (Claim 1)

[1436] means for processing user input using the generative model;

[1437] means for receiving speech input from a user and converting it into text data;

[1438] means for sending string data to the generative model and obtaining a generated response;

[1439] means for converting the acquired response into speech and presenting it to the user;

[1440] A system including:

[1441] (Claim 2)

[1442] The system of claim 1, further comprising a means for the generative model to monitor the vehicle's condition and present specific countermeasures when an abnormality is detected.

[1443] (Claim 3)

[1444] 2. The system of claim 1, wherein the generative model includes means for presenting multiple appropriate options for a user's abstract request.

[1445] "Example 1"

[1446] (Claim 1)

[1447] means for obtaining user input as speech;

[1448] A means for converting the acquired voice into character string data;

[1449] means for transmitting the transformed string data to a generative model;

[1450] a means for obtaining a response generated by the generative model;

[1451] means for converting the acquired response into speech and presenting it to the user;

[1452] A system including:

[1453] (Claim 2)

[1454] The system according to claim 1, further comprising means for suggesting specific countermeasures when the generative model detects an abnormality in the vehicle.

[1455] (Claim 3)

[1456] 2. The system according to claim 1, further comprising means for transmitting character string data obtained from a user's voice input to a cloud server and obtaining a generated response from the cloud server.

[1457] "Application Example 1"

[1458] (Claim 1)

[1459] means for collecting user input via speech and converting it into text data;

[1460] means for sending the collected text data to a generative model and obtaining a text response returned from the generative model;

[1461] means for converting the obtained text response into speech and presenting it to the user;

[1462] A means to generate responses to user instructions and questions using a generative model, and provide vehicle status confirmation, route guidance, and real-time traffic information;

[1463] A system including:

[1464] (Claim 2)

[1465] The system of claim 1, wherein the generative model monitors the vehicle's condition and suggests specific countermeasures when an abnormality is detected.

[1466] (Claim 3)

[1467] 2. The system of claim 1, wherein the generative model includes means for presenting multiple appropriate options for a user's abstract request.

[1468] "Example 2: Combining Emotion Engines"

[1469] (Claim 1)

[1470] a means for using a generative model to process user input;

[1471] means for receiving speech input from a user and converting it into text data;

[1472] means for sending string data to the generative model and obtaining a generated response;

[1473] means for converting the acquired response into speech and presenting it to the user;

[1474] means for recognizing a user's emotion using an emotion engine;

[1475] a means for adjusting the content and tone of a response based on the user's emotions;

[1476] A system including:

[1477] (Claim 2)

[1478] The system of claim 1, further comprising a means for the generative model to monitor the vehicle's condition and present specific countermeasures when an abnormality is detected.

[1479] (Claim 3)

[1480] 2. The system of claim 1, wherein the generative model includes means for presenting multiple appropriate options for a user's abstract request.

[1481] "Application example 2 when combining emotion engines"

[1482] (Claim 1)

[1483] means for processing user input using the generative model;

[1484] means for receiving speech input from a user and converting it into text data;

[1485] means for sending string data to the generative model and obtaining a generated response;

[1486] means for converting the acquired response into speech and presenting it to the user;

[1487] means for recognizing a user's emotion;

[1488] means for adjusting the generated response according to the emotional state of the user;

[1489] A system including:

[1490] (Claim 2)

[1491] The system of claim 1, further comprising a means for the generative model to monitor the vehicle's condition and present specific countermeasures when an abnormality is detected.

[1492] (Claim 3)

[1493] 2. The system of claim 1, wherein the generative model includes means for presenting multiple appropriate options for a user's abstract request. [Explanation of symbols]

[1494] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for processing user input using the generative model; means for receiving speech input from a user and converting it into text data; means for sending string data to the generative model and obtaining a generated response; means for converting the acquired response into speech and presenting it to the user; A system including:

2. The system according to claim 1, further comprising means for the generative model to monitor the state of the vehicle and present specific countermeasures when an abnormality is detected.

3. The system of claim 1 , wherein the generative model includes means for presenting multiple appropriate options for a user's abstract request.

Images