system

Abstract

Provide a system. A means for accepting voice input; means for converting the received speech into text; A means for analyzing the converted text to determine user intent; and A means of suggesting destinations based on the user's intent; Once the destination is determined, a means for calculating the optimal route using that information; a means for obtaining real-time traffic information; A means of selecting the optimal route taking into account real-time traffic information; a means for providing the selected route information to an automated driving system of the vehicle; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] As fully autonomous vehicles become more widespread, conventional destination input methods tend to be cumbersome and often counterintuitive for users. Furthermore, it is difficult to propose optimal routes that take traffic conditions into account in real time when specifying a destination, making it difficult to achieve efficient travel. Furthermore, users often get lost when their destination is undecided, and detailed destination information may be lacking. To address these issues, a system is needed that can be used intuitively with voice control, provides optimal routes taking real-time traffic information into account, and can provide interactive suggestions and detailed destination information when the destination is undecided. [Means for solving the problem]

[0005] The present invention provides a system including means for accepting voice input, means for converting the accepted voice into text, means for identifying a user's intent by analyzing the converted text, means for suggesting a destination based on the user's intent, means for calculating an optimal route using the destination once the destination has been determined, means for acquiring real-time traffic information, means for selecting an optimal route taking the real-time traffic information into consideration, and means for providing the selected route information to the vehicle's autonomous driving system. Furthermore, by further providing means for suggesting a destination in an interactive manner when the user has not specified a destination, and means for acquiring and providing detailed destination information to the user, the system achieves intuitive and efficient use of an autonomous vehicle.

[0006] "Voice input" is a means by which a system accepts spoken words or instructions from a user.

[0007] "Convert to text" refers to the process of converting received voice data into textual information.

[0008] "Analysis" is the process of understanding text data and identifying the user's intent and meaning.

[0009] "User intent" refers to the purpose or intent of what the user is requesting from the system.

[0010] "Suggesting a destination" means that the system suggests suitable locations if the user does not have a specific destination in mind.

[0011] "Calculating the optimal route" means calculating the most efficient route from a starting point to a destination.

[0012] "Real-time traffic information" refers to the latest data that reflects current traffic conditions.

[0013] "Selecting" means choosing the most appropriate option from among multiple options.

[0014] An "autonomous driving system" is a technology that automatically controls and drives a vehicle based on input data.

[0015] "Dialogue" means that the user and the system exchange questions and answers continuously.

[0016] "Destination details" refers to specific information about the destination (such as opening hours, ratings, menu, etc.). [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention is a system for determining a destination using voice control in a fully automated driving vehicle and efficiently providing an optimal route. Specific embodiments of this system will be described below.

[0039] 1. Speech Recognition Module

[0040] Device:

[0041] The user inputs destinations and travel instructions by voice. The device is equipped with a microphone that captures the voice input and saves it as voice data. The saved voice data is sent to a voice recognition system and converted into text data. This text data is then sent to the server.

[0042] 2. Dialogue Engine

[0043] server:

[0044] The server receives the text data sent from the speech recognition module and analyzes it. This analysis identifies the user's intent and determines the next step. For example, if the user says, "I want to go to a nice restaurant," the server understands that the user is searching for a restaurant and then suggests suitable restaurant options.

[0045] For example, the server might proceed with the following dialogue:

[0046] User: "I want to go to a nice restaurant."

[0047] Server: "We'd like to introduce you to some popular restaurants in the area. Would you like to choose Italian, Japanese, or Chinese?"

[0048] 3. Navigation Module

[0049] server:

[0050] Once the destination is confirmed, the server launches the navigation module to calculate the optimal route to the destination. The navigation module obtains current traffic condition data from the real-time traffic information module and selects the optimal route based on this data.

[0051] The navigation module sends calculated route information to the terminal and provides instructions to the vehicle's autonomous driving system.

[0052] 4. Real-time traffic information module

[0053] server:

[0054] The server accesses an external traffic information service API to obtain the latest traffic information, including road congestion, accidents, construction information, etc. The obtained information is provided to the navigation module and used to calculate the optimal route.

[0055] 5. Destination Information Module

[0056] server:

[0057] The server queries external databases and APIs to obtain detailed information about destinations that the user has expressed interest in. The information obtained (e.g., restaurant ratings, menus, opening hours, etc.) is provided to the user through a dialogue engine.

[0058] For example, if a user requests "Tell me more about this restaurant," the server provides the following information:

[0059] Server: "This restaurant has a 4.5 rating. They have pasta and pizza on the menu. They're open from 11:00 AM to 10:00 PM."

[0060] summary

[0061] The system supports a series of processes, from voice input to destination suggestions, route calculation, real-time traffic information, and detailed destination information, allowing users to intuitively and efficiently operate an autonomous vehicle and providing a comfortable travel experience.

[0062] The processing flow will be explained below.

[0063] Step 1:

[0064] User:

[0065] The user says, "I want to go to a nice restaurant."

[0066] Step 2:

[0067] Device:

[0068] The microphone on the device captures the user's voice and saves it as audio data.

[0069] Step 3:

[0070] Device:

[0071] The saved voice data is sent to a voice recognition service and converted into text data.

[0072] Step 4:

[0073] Device:

[0074] The converted text data is sent to the server.

[0075] Step 5:

[0076] server:

[0077] The server analyzes the received text data and determines the user's intent. In this case, it understands that the user is looking for a restaurant.

[0078] Step 6:

[0079] server:

[0080] The dialogue engine asks the user a question: "We'll introduce you to some popular restaurants in the area. Would you like to choose Italian, Japanese, or Chinese?" The message is sent to the device.

[0081] Step 7:

[0082] User:

[0083] A user says, "I want Italian food."

[0084] Step 8:

[0085] Device:

[0086] The user's voice is captured again and sent to a speech recognition service for conversion into text data.

[0087] Step 9:

[0088] Device:

[0089] The converted text data is sent to the server.

[0090] Step 10:

[0091] server:

[0092] The server analyzes the text data and identifies the user's request. In this case, it understands that the user wants an Italian restaurant.

[0093] Step 11:

[0094] server:

[0095] The server retrieves information about recommended Italian restaurants from an external database and makes suggestions to the user, for example, sending a message to the terminal saying, "Nearby Italian restaurants are Restaurant A, Restaurant B, and Restaurant C."

[0096] Step 12:

[0097] User:

[0098] A user says, "I want to go to Restaurant A."

[0099] Step 13:

[0100] Device:

[0101] The user's voice is captured and sent to a speech recognition service to be converted into text data.

[0102] Step 14:

[0103] Device:

[0104] The converted text data is sent to the server.

[0105] Step 15:

[0106] server:

[0107] The server analyzes the text data and determines that Restaurant A is the destination.

[0108] Step 16:

[0109] server:

[0110] The server passes the destination information to the navigation module and calculates the optimal route to Restaurant A.

[0111] Step 17:

[0112] server:

[0113] The navigation module obtains traffic information from the real-time traffic information module and selects the optimal route taking into account traffic congestion.

[0114] Step 18:

[0115] server:

[0116] The selected route information is sent to the terminal.

[0117] Step 19:

[0118] Device:

[0119] The route information received by the terminal is reflected in the vehicle's autonomous driving system, and the vehicle begins driving automatically toward the destination.

[0120] Step 20:

[0121] server:

[0122] When a user requests "more information about this restaurant," the server retrieves information such as restaurant ratings, menus, and opening hours from an external information service and provides it to the user.

[0123] Example 1

[0124] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0125] Current autonomous vehicle systems do not fully realize the process in which a user can intuitively and efficiently specify a destination using voice input, and the autonomous vehicle selects the optimal route and automatically guides the user to the destination. Therefore, a system is needed that integrates a series of processes: determining a destination using voice control, reflecting real-time traffic information, and providing detailed destination information.

[0126] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0127] In this invention, the server includes means for accepting voice input, means for converting the accepted voice into text, means for identifying the user's intention by analyzing the converted text, means for suggesting a destination based on the user's intention, means for calculating an optimal route using the destination information when the destination is confirmed, means for acquiring real-time traffic information, means for selecting an optimal route taking the real-time traffic information into consideration, means for providing the selected route information to the vehicle's autonomous driving system, and means for querying an external database to acquire detailed destination information and providing it to the user. This allows the user to specify a destination through intuitive voice operation and use a system that integrates optimal route calculation and detailed information provision.

[0128] "Means for accepting voice input" refers to devices or software that capture voice instructions given by a user and convert them into digital data.

[0129] "Means for converting received speech into text" refers to a system that uses speech recognition technology to convert speech data into text data.

[0130] "Means for analyzing the converted text to identify the user's intent" refers to a system that uses natural language processing technology to analyze text data and identify the user's requests and wishes.

[0131] "Means for suggesting destinations based on the user's intentions" refers to a system that predicts the user's desired destination or route and makes suggestions accordingly.

[0132] "A means for calculating the optimal route using information once the destination is confirmed" refers to a system for calculating the shortest distance and the most time-efficient route based on confirmed destination information.

[0133] "Means of obtaining real-time traffic information" refers to APIs and database access means for obtaining current road conditions and traffic data from external services.

[0134] "Means for selecting the optimal route taking into account real-time traffic information" refers to algorithms or systems that determine the optimal route to a destination based on real-time traffic information.

[0135] "Means for providing selected route information to the vehicle's automated driving system" refers to a system that transmits calculated route data to the automated vehicle so that the vehicle can follow that route.

[0136] "Means of querying an external database to obtain detailed destination information and providing it to the user" refers to a system that queries an external database or API for detailed destination information (e.g., opening hours, ratings, menu, etc.) and provides it to the user.

[0137] The present invention provides a fully automated driving system that allows a user to specify a destination using voice input and efficiently provides an optimal route. Specific embodiments of the present invention will be described in detail below.

[0138] 1. Speech Recognition Module

[0139] Device:

[0140] The user inputs destinations and travel instructions by voice. The device is equipped with a microphone that captures the user's voice input. This voice data is sent to a voice recognition system (e.g., voice recognition API) and converted into text data. The converted text data is then sent to the server.

[0141] Examples:

[0142] When a user speaks into a microphone, "I want to go to a nice restaurant," the speech is converted into text.

[0143] 2. Dialogue Engine

[0144] server:

[0145] The server receives the text data sent from the speech recognition module and analyzes it using a natural language processing engine (e.g., a generative AI model). The analysis identifies the user's intent and generates appropriate suggestions as the next step.

[0146] Examples:

[0147] User: "I want to go to a nice restaurant."

[0148] Server: "We'd like to introduce you to some popular restaurants in the area. Would you like to choose Italian, Japanese, or Chinese?"

[0149] 3. Navigation Module

[0150] server:

[0151] Once the destination is confirmed, the server launches the navigation module (e.g., map API) to calculate the optimal route to the destination. This module obtains current traffic condition data from the real-time traffic information module and selects the optimal route. The calculation results are sent to the terminal, which issues instructions to the vehicle's autonomous driving system.

[0152] Examples:

[0153] After the user selects "Japanese food," the server calculates the optimal route to the Japanese restaurant and provides instructions to the autonomous driving system.

[0154] 4. Real-time traffic information module

[0155] server:

[0156] The server accesses an external traffic information service API (e.g., traffic information API) to obtain the latest traffic information. This information includes road congestion, accidents, construction information, etc. The obtained information is provided to the navigation module and used to calculate the optimal route.

[0157] Examples:

[0158] The server obtains traffic congestion information and calculates the optimal detour route.

[0159] 5. Destination Information Module

[0160] server:

[0161] Detailed information about destinations that the user has shown interest in is obtained by querying external databases or APIs (e.g., information provision APIs). The obtained information includes restaurant ratings, menus, opening hours, etc. This information is provided to the user through a dialogue engine.

[0162] Examples:

[0163] User: "Tell me more about this restaurant."

[0164] Server: "This restaurant has a 4.5 rating and offers pasta and pizza. It's open from 11:00 AM to 10:00 PM."

[0165] summary

[0166] The system supports a series of processes, from voice input to destination suggestions, route calculation, real-time traffic information, and detailed destination information, allowing users to intuitively and efficiently operate an autonomous vehicle and enjoy a comfortable travel experience.

[0167] Prompt Sentence Examples

[0168] "In a fully autonomous vehicle system, if a user says, 'I want to go to a nice restaurant,' explain how each module in the system processes the data and provides information to the user."

[0169] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0170] Step 1:

[0171] The user provides voice input.

[0172] Specific action: The user speaks into the microphone, "I want to go to a nice restaurant."

[0173] Input: User's voice command.

[0174] Output: Audio data.

[0175] Step 2:

[0176] The device captures the audio data.

[0177] Specific operation: The microphone installed on the device captures the user's voice and saves it as audio data.

[0178] Input: User's voice.

[0179] Output: The captured audio data.

[0180] Step 3:

[0181] The terminal transmits the voice data to the voice recognition system.

[0182] Specific operation: The device sends the captured voice data to the voice recognition API.

[0183] Input: Audio data.

[0184] Output: Data sent to the speech recognition API.

[0185] Step 4:

[0186] A speech recognition system converts the speech data into text data.

[0187] Specific operation: The speech recognition API analyzes the speech data and converts it into text data.

[0188] Input: Audio data.

[0189] Output: The converted text data.

[0190] Step 5:

[0191] The terminal transmits the text data to the server.

[0192] Specific operation: The terminal sends the converted text data to the server.

[0193] Input: Text data.

[0194] Output: The data sent to the server.

[0195] Step 6:

[0196] The server parses the text data.

[0197] Specific operation: The server sends the received text data to a natural language processing engine (e.g., a generative AI model) for analysis.

[0198] Input: Text data.

[0199] Output: Data identifying the user's intent.

[0200] Step 7:

[0201] The server suggests destinations based on the user's intentions.

[0202] Specific operation: The server generates suggestions and responds to the user in an interactive format.

[0203] Input: Parsed text data.

[0204] Output: Destination suggestion data.

[0205] Step 8:

[0206] The server launches the navigation module.

[0207] Specific operation: After the destination is determined, the server launches the navigation module and calculates the optimal route to the destination.

[0208] Input: Confirmed destination information.

[0209] Output: Launch of navigation module.

[0210] Step 9:

[0211] The server obtains real-time traffic information.

[0212] Specific operation: The server accesses the traffic information API and obtains the latest traffic information (traffic congestion, accidents, construction, etc.).

[0213] Input: A navigation launch request.

[0214] Output: The obtained traffic information.

[0215] Step 10:

[0216] The server calculates the optimal route.

[0217] Specific operation: The server calculates the optimal route based on the acquired real-time traffic information.

[0218] Input: Real-time traffic information.

[0219] Output: Calculated optimal route data.

[0220] Step 11:

[0221] The server sends the route information to the device.

[0222] Specific operation: The server sends the calculated route information to the terminal.

[0223] Input: Optimal route data.

[0224] Output: Route information sent to the device.

[0225] Step 12:

[0226] The terminal issues instructions to the autonomous driving system.

[0227] Specific operation: The terminal issues instructions to the autonomous driving system based on the received route information, and the vehicle moves along the route.

[0228] Input: Route information.

[0229] Output: Instructions to the autonomous driving system.

[0230] Step 13:

[0231] The server obtains the destination information.

[0232] Specific operation: The server queries external databases or APIs to obtain detailed information about the destination (ratings, menu, opening hours, etc.).

[0233] Input: Destination information.

[0234] Output: The obtained destination information.

[0235] Step 14:

[0236] The server provides the destination details to the user.

[0237] Specific operation: The server provides the acquired destination information to the user through the dialogue engine.

[0238] Input: Destination information.

[0239] Output: Detailed information provided to the user.

[0240] (Application example 1)

[0241] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0242] In modern self-driving vehicles, there are systems that allow users to specify their destination via voice and provide the optimal route based on that destination. However, there are problems when users do not specify a specific destination, and there is a lack of systems that provide detailed destination information. There is also a need for dialogue-based destination suggestions that allow users to intuitively select their destination. Furthermore, natural dialogue using prompt sentences generated using generative AI models has not yet been realized. In this situation, a comprehensive and highly functional system is needed to improve user convenience.

[0243] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0244] In this invention, the server includes means for accepting voice input, means for converting voice to text, means for analyzing the text to identify a user's intent, means for suggesting a destination based on the user's intent, means for calculating an optimal route when the destination is confirmed, means for acquiring real-time traffic information, means for selecting an optimal route, means for providing the selected route information to the vehicle's autonomous driving system, means for acquiring detailed information about the identified destination from an external database and providing it to the user, and means for suggesting destinations in an interactive format using a generative AI model and generating prompt sentences to help the user make an appropriate selection. This allows the user to intuitively and efficiently determine their destination through voice operation and travel along the optimal route while obtaining detailed information.

[0245] "Means for accepting voice input" refers to means for capturing voice uttered by a user and inputting the voice data into the system.

[0246] The "means for converting voice to text" is a means for converting received voice data into text data in a specified language.

[0247] "Means for analyzing text to identify user intent" refers to means for analyzing converted text data and recognizing and identifying the user's intent.

[0248] The "means for suggesting a destination based on the user's intention" is a means for suggesting destination candidates that match the specified user's intention.

[0249] The "means for calculating the optimum route when the destination is determined" is a means for calculating the optimum route to reach the determined destination.

[0250] "Means for obtaining real-time traffic information" refers to means for obtaining real-time traffic data such as current traffic conditions, congestion, and accident information.

[0251] The "means for selecting an optimal route" is a means for selecting the most efficient route in consideration of the acquired real-time traffic information.

[0252] "Means for providing selected route information to the vehicle's automated driving system" refers to means for issuing driving instructions to the vehicle's automated driving system based on the selected route.

[0253] The "means for obtaining detailed information about the identified destination from an external database and providing it to the user" refers to a means for obtaining detailed information about the destination from an external database and providing it to the user.

[0254] "A means of using a generative AI model to suggest destinations in an interactive format and generate prompts to help the user make an appropriate choice" refers to a means of utilizing generative AI to suggest destinations in a natural interactive format and generate prompts to guide the user in making an appropriate choice.

[0255] This invention is a system for fully automated driving vehicles that uses voice control to determine destinations and efficiently provide optimal routes. This system supports everything from voice input to destination suggestions, route calculations, real-time traffic information, and detailed destination information.

[0256] Voice Input and Recognition Module

[0257] The terminal is a device that allows users to input destinations and travel instructions by voice. The terminal is equipped with a microphone that captures voice input and sends the data to a speech recognition module (e.g., Google® Cloud Speech-to-Text API), which converts the voice data into text.

[0258] Text analysis and dialogue engine

[0259] The server analyzes the text data sent from the speech recognition module and identifies the user's intention. A dialogue engine (e.g., Dialogflow) is used for the analysis, and suggests destinations in a dialogue format. For example, if the user says, "I want to go to a nice restaurant," the server responds as follows:

[0260] "We'll introduce you to popular restaurants in the area. You can choose from Italian, Japanese, or Chinese cuisine."

[0261] Destination and navigation

[0262] Once the destination is determined, the server launches the navigation module, which uses the Google Maps API to calculate the optimal route to the destination, retrieves real-time traffic information (e.g., Google Maps Traffic API), selects the optimal route, and provides that information to the vehicle's autonomous driving system.

[0263] Real-time traffic information

[0264] The server accesses an external traffic information service API to obtain the latest traffic information, including road congestion, accidents, and construction information, and uses this information to calculate the optimal route in the navigation module.

[0265] Providing destination information

[0266] The server queries an external database or API (e.g., Yelp API) for details about the destination selected by the user and provides the information obtained to the user. For example, if the user requests "Tell me the details of this restaurant," the server provides the following information:

[0267] "This restaurant has a 4.5 rating, offers pasta and pizza on the menu, and is open from 11am to 10pm."

[0268] Leveraging generative AI models

[0269] The server uses a generative AI model to interactively suggest destinations and generate prompts to help users make appropriate choices. This makes it easier for users to decide on a destination through natural dialogue. Here are some examples of prompts:

[0270] "I want to go to a nice restaurant."

[0271] "We'll introduce you to popular restaurants in the area. You can choose from Italian, Japanese, or Chinese cuisine."

[0272] "If you select Italian, we will retrieve and provide detailed information about your destination to you."

[0273] This will enable users to intuitively and efficiently operate autonomous vehicles, providing a comfortable travel experience.

[0274] As described above, the system of the present invention comprehensively supports everything from voice input to destination setting, route guidance, and the provision of detailed information, and by utilizing a generative AI model in the process, it provides users with high convenience and intuitive operability.

[0275] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0276] Step 1:

[0277] Accepting voice input

[0278] The user inputs destinations and travel instructions into the device by voice, which is captured through the device's microphone and saved as audio data.

[0279] Input: User's voice command

[0280] Output: Captured audio data

[0281] Step 2:

[0282] Converting audio data to text

[0283] The device sends the captured voice data to a voice recognition module (e.g., Google Cloud Speech-to-Text API) and converts the voice data into text data.

[0284] Input: Audio data

[0285] Output: Converted text data

[0286] Step 3:

[0287] Text data analysis

[0288] The server analyzes the converted text data to identify the user's intent. A dialogue engine (e.g., Dialogflow) is used for the analysis, and the user's intent regarding destination and route is identified.

[0289] Input: Text data

[0290] Output: Identified user intent data

[0291] Step 4:

[0292] Destination suggestions

[0293] The server suggests destinations based on the user's intentions, and uses a generative AI model to interactively suggest potential destinations and generate prompts to help the user make the appropriate choice.

[0294] Input: User intent data

[0295] Output: Possible destinations and prompts

[0296] Step 5:

[0297] Destination determination and route calculation

[0298] Once the user has decided on a destination, the server uses the Google Maps API to calculate the optimal route to the destination.

[0299] Input: Selected destination

[0300] Output: Optimal route information

[0301] Step 6:

[0302] Obtaining real-time traffic information

[0303] The server uses external traffic information service APIs, such as Google Maps Traffic API, to obtain the latest traffic information, including road congestion, accidents, and construction information.

[0304] Input: None (Access to external traffic information services)

[0305] Output: Real-time traffic information

[0306] Step 7:

[0307] Selection of the optimal route

[0308] The server selects the optimal route taking into account real-time traffic information.

[0309] Input: Initial route calculation, real-time traffic information

[0310] Output: Optimal route reflecting traffic information

[0311] Step 8:

[0312] Providing route information

[0313] The server provides the selected route information to the vehicle's autonomous driving system and controls the vehicle's driving.

[0314] Input: Optimal route information

[0315] Output: Driving instructions to the vehicle's autonomous driving system

[0316] Step 9:

[0317] Providing destination information

[0318] The server queries an external database (e.g., Yelp API) for details about the identified destination and provides the retrieved information to the user.

[0319] Input: Confirmed destination

[0320] Output: Destination details (ratings, menu, opening hours, etc.)

[0321] This allows users to intuitively and efficiently determine their destination through voice control, and travel along the optimal route while obtaining detailed information.

[0322] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0323] The present invention provides a system for determining a destination using voice control in a fully automated driving vehicle, recognizing the user's emotions, and efficiently providing an optimal route. Specific embodiments of this system are described below.

[0324] 1. Speech Recognition Module

[0325] Device:

[0326] The user inputs destinations and travel instructions by voice. The device is equipped with a microphone that captures the voice input and saves it as voice data. The saved voice data is sent to a voice recognition system and converted into text data. This text data is then sent to the server.

[0327] 2. Emotion Engine

[0328] server:

[0329] The server analyzes the text data and voice sent from the voice recognition module to identify the user's emotions. Using an emotion engine, it recognizes the user's emotional state (e.g., joy, anger, sadness, etc.) and records the results. This makes it possible to understand the emotional state of the user when giving instructions and provide appropriate responses and destination suggestions.

[0330] For example, the emotion engine works as follows:

[0331] User: "I want to go to a nice restaurant." (Audio contains a tone of delight)

[0332] Server: "Here are some popular restaurants in the area. Would you like to choose Italian, Japanese, or Chinese?" (The emotion engine recognizes that the user is happy and responds in a positive tone.)

[0333] 3. Dialogue Engine

[0334] server:

[0335] The server analyzes the text data received from the speech recognition module to identify the user's intent. Through this analysis, the server understands the user's intent (command, question, suggestion, etc.) and determines the next step. The server also takes into account the results of the emotion engine to generate optimal destination suggestions and responses.

[0336] 4. Navigation Module

[0337] server:

[0338] When the server receives the destination information, the navigation module is activated and calculates the optimal route to the destination. The navigation module obtains the latest traffic information from the real-time traffic information module and selects the optimal route based on that information.

[0339] The navigation module sends calculated route information to the terminal and provides instructions to the vehicle's autonomous driving system.

[0340] 5. Real-time traffic information module

[0341] server:

[0342] The server accesses an external traffic information service API to obtain the latest traffic information, including road congestion, accidents, construction information, etc. The obtained information is provided to the navigation module and used to calculate the optimal route.

[0343] 6. Destination Information Module

[0344] server:

[0345] The server queries external databases and APIs to obtain detailed information about destinations that the user has expressed interest in. The information obtained (e.g., restaurant ratings, menus, opening hours, etc.) is provided to the user through a dialogue engine.

[0346] For example, if the user says "Tell me more about this restaurant," the server provides the following information:

[0347] Server: "This restaurant has a 4.5 rating. They have pasta and pizza on the menu. They're open from 11:00 AM to 10:00 PM."

[0348] summary

[0349] The system covers a range of processes, from voice input to emotion recognition, destination suggestions, route calculation, real-time traffic information, and detailed destination information. This allows users to not only intuitively operate an autonomous vehicle, but also to receive more detailed emotional services, providing a comfortable travel experience.

[0350] The processing flow will be explained below.

[0351] Step 1:

[0352] User:

[0353] The user says, "I want to go to a nice restaurant."

[0354] Step 2:

[0355] Device:

[0356] The microphone on the device captures the user's voice and saves it as audio data.

[0357] Step 3:

[0358] Device:

[0359] The saved voice data is sent to a voice recognition service and converted into text data.

[0360] Step 4:

[0361] Device:

[0362] The converted text data is sent to the server.

[0363] Step 5:

[0364] server:

[0365] The server analyzes the received text data and determines the user's intent. In this case, it understands that the user is looking for a restaurant.

[0366] Step 6:

[0367] server:

[0368] The emotion engine is used to analyze the user's emotions from the received voice data, for example, recognizing that the user is happy from the tone and accent of the voice.

[0369] Step 7:

[0370] server:

[0371] Based on the analysis results of the emotion engine, the dialogue engine asks the user a question: "We'll introduce you to some popular restaurants in the area. Would you like to choose from Italian, Japanese, or Chinese cuisine?" The message is sent to the device.

[0372] Step 8:

[0373] User:

[0374] A user says, "I want Italian food."

[0375] Step 9:

[0376] Device:

[0377] The user's voice is captured again and sent to a speech recognition service for conversion into text data.

[0378] Step 10:

[0379] Device:

[0380] The converted text data is sent to the server.

[0381] Step 11:

[0382] server:

[0383] The server analyzes the text data and identifies the user's request. In this case, it understands that the user wants an Italian restaurant.

[0384] Step 12:

[0385] server:

[0386] The server retrieves information about recommended Italian restaurants from an external database and makes suggestions to the user, for example, sending a message to the terminal saying, "Nearby Italian restaurants are Restaurant A, Restaurant B, and Restaurant C."

[0387] Step 13:

[0388] User:

[0389] A user says, "I want to go to Restaurant A."

[0390] Step 14:

[0391] Device:

[0392] The user's voice is captured and sent to a speech recognition service to be converted into text data.

[0393] Step 15:

[0394] Device:

[0395] The converted text data is sent to the server.

[0396] Step 16:

[0397] server:

[0398] The server analyzes the text data and determines that Restaurant A is the destination.

[0399] Step 17:

[0400] server:

[0401] The server launches the navigation module and calculates the optimal route to the destination, Restaurant A.

[0402] Step 18:

[0403] server:

[0404] The navigation module accesses the real-time traffic information module to obtain the latest traffic information, including congestion, traffic accidents, and construction information.

[0405] Step 19:

[0406] server:

[0407] The navigation module takes into account traffic information obtained and selects the optimal route.

[0408] Step 20:

[0409] server:

[0410] The selected route information is sent to the terminal.

[0411] Step 21:

[0412] Device:

[0413] The route information received by the terminal is reflected in the vehicle's autonomous driving system, and the vehicle begins driving automatically toward the destination.

[0414] Step 22:

[0415] User:

[0416] A user says, "Tell me more about this restaurant."

[0417] Step 23:

[0418] Device:

[0419] The user's voice is captured and sent to a speech recognition service to be converted into text data.

[0420] Step 24:

[0421] Device:

[0422] The converted text data is sent to the server.

[0423] Step 25:

[0424] server:

[0425] The server parses the text data and understands the user's request.

[0426] Step 26:

[0427] server:

[0428] The server accesses an external information service to obtain detailed restaurant information (ratings, menu, opening hours, etc.).

[0429] Step 27:

[0430] server:

[0431] The acquired detailed information is then provided to the user through a dialogue engine, for example, "This restaurant has a rating of 4.5. The menu includes pasta and pizza. It is open from 11:00 AM to 10:00 PM."

[0432] Example 2

[0433] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0434] Conventional autonomous driving systems require users to manually input their destination and set up routes, resulting in unintuitive operation. Furthermore, they do not take into account the user's emotional state, resulting in issues with not being able to provide optimal service or responses. Furthermore, they are not sufficient in selecting optimal routes that take real-time traffic information into account, leaving a need for improving the quality of the travel experience.

[0435] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0436] In this invention, the server includes means for accepting voice input, means for converting the accepted voice into text, means for analyzing the converted text and voice data to identify the user's emotional state, means for identifying the user's intention based on the analyzed text data, means for suggesting a destination based on the identified user's intention and emotional state, means for calculating an optimal route using the destination information when the destination is confirmed, means for acquiring real-time traffic information from an external traffic information service, means for selecting an optimal route taking into account the acquired real-time traffic information, and means for providing the selected route information to the vehicle's autonomous driving system. This allows the user to specify a destination in an intuitive manner, provides appropriate services and responses taking into account the user's emotional state, and enables optimal route selection based on real-time traffic information.

[0437] "Means for accepting voice input" refers to a device or software that has the function of recognizing voice uttered by a user and inputting it into the system.

[0438] "Means for converting received speech into text" refers to a technology or system for analyzing speech data and converting it into text data.

[0439] "Means for analyzing converted text and voice data to identify a user's emotional state" refers to technology or systems that recognize and identify a user's emotions (e.g., joy, anger, sadness, etc.) by analyzing text and voice data.

[0440] "Means for identifying a user's intent based on analyzed text data" refers to a technology or system that analyzes text data to understand a user's instructions or requests and identify their intent.

[0441] "Means for suggesting destinations based on identified user intentions and emotional state" refers to a technology or system for suggesting appropriate destinations taking into account a user's intentions and emotional state.

[0442] "Means for calculating the optimal route using information once the destination is determined" refers to technology or a system that calculates the optimal route to a destination based on the destination specified or agreed to by the user.

[0443] "Means for obtaining real-time traffic information from external traffic information services" refers to a technology or system that accesses external traffic information services and obtains the latest traffic situation data.

[0444] "Means for selecting the optimal route taking into account acquired real-time traffic information" refers to a technology or system that selects the most efficient route based on real-time traffic conditions.

[0445] "Means for providing selected route information to the vehicle's automated driving system" refers to a technology or system that communicates the selected route information to the vehicle's automated driving system and issues instructions.

[0446] This invention is a system for fully autonomous vehicles that uses voice control to determine destinations, recognizes the user's emotions, and efficiently provides the optimal route. This system is mainly composed of the following modules.

[0447] 1. Voice Input Module

[0448] Device:

[0449] The user inputs destinations and travel instructions by voice. The device is equipped with a microphone that captures the user's voice and saves it as audio data. The saved audio data is then sent to a server. This process uses a common hardware microphone and voice recognition software called a "voice recognition API" (for example, Google Cloud Speech-to-Text API).

[0450] For example, when a user says, "I want to go to a nice restaurant," the microphone on the device captures the voice and generates voice data, which is then sent to the server.

[0451] 2. Speech Recognition and Text Conversion

[0452] server:

[0453] The server passes the received voice data to a voice recognition engine (for example, Google Cloud Speech-to-Text API) and converts the voice data into text data, which is then stored on the server.

[0454] 3. Emotion Recognition Engine

[0455] server:

[0456] An emotion engine (e.g., IBM Watson® Tone Analyzer) analyzes the converted text data and the original voice data to identify the user's emotional state (e.g., joy, anger, sadness, etc.).

[0457] For example, if a user says, "I want to go to a nice restaurant," the voice contains a tone of joy. The server uses an emotion engine to recognize the emotional state and determine that the user is happy.

[0458] 4. Dialogue Engine

[0459] server:

[0460] The server analyzes the text data received from the speech recognition module and uses a dialogue engine (e.g., OpenAI® GPT-3®) to identify the user's intent. The analysis allows the user's intent (e.g., command, question, suggestion, etc.) to be understood.

[0461] 5. Destination suggestions

[0462] server:

[0463] Destination suggestions are made based on the user's intent and emotional state. The server combines the identified user intent and emotional state to generate optimal destination suggestions and responses.

[0464] For example, if a user says, "I want to go to a nice restaurant," the server will suggest, "We'll introduce you to some popular restaurants in the area. Would you like to choose from Italian, Japanese, or Chinese cuisine?"

[0465] 6. Route calculation

[0466] server:

[0467] Once the destination information is confirmed, the navigation module (e.g., Google Maps API) is used to calculate the optimal route to the destination. The latest traffic information is obtained from the real-time traffic information module, and the optimal route is selected based on that information.

[0468] As a concrete example, if you select "Italian restaurant," the server will calculate the optimal route through the navigation module.

[0469] 7. Real-time traffic information

[0470] server:

[0471] The server obtains the latest traffic information from an external traffic information service API and provides it to the navigation module, which then selects the optimal route based on the real-time traffic information.

[0472] 8. Providing route information

[0473] server:

[0474] The calculated route information is sent to the terminal and instructions are sent to the autonomous driving system, which then guides the vehicle to its destination.

[0475] 9. Providing destination information

[0476] server:

[0477] Detailed information about the destinations the user has expressed interest in is retrieved from external databases and APIs and provided to the user, including suggested restaurant details such as ratings, menus, and opening hours.

[0478] For example, if a user says, "Tell me more about this restaurant," the server might provide information like, "This restaurant has a rating of 4.5. The menu includes pasta and pizza. It's open from 11:00 a.m. to 10:00 p.m."

[0479] Prompt Sentence Examples

[0480] User: "I want to go to a nice restaurant."

[0481] This system covers a range of processes, from voice input to emotion recognition, destination suggestions, route calculation, real-time traffic information, and detailed destination information, thereby providing users with intuitive and emotionally responsive services and a comfortable travel experience.

[0482] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0483] Step 1: Getting voice input

[0484] The user inputs destinations and instructions into the terminal by voice.

[0485] The device captures the user's voice with a microphone and saves it as audio data.

[0486] Input: User's voice commands

[0487] Output: Audio data

[0488] How it works: When a user says, "I want to go to a nice restaurant," the microphone captures the voice, which is converted into digital audio data and stored on the device. This audio data is then sent to the server.

[0489] Step 2: Speech recognition and text conversion

[0490] The server passes the received voice data to a voice recognition engine (Google Cloud Speech-to-Text API) and converts the voice data into text data.

[0491] Input: Audio data

[0492] Output: Text data

[0493] Specific operation: The voice data is sent to the server and passed to the speech recognition engine. This engine analyzes the voice data and converts it into text data such as "I want to go to a delicious restaurant."

[0494] Step 3: Emotion Recognition

[0495] The server passes the text and voice data to an emotion engine (IBM Watson Tone Analyzer) to analyze the user's emotions.

[0496] Input: Text data, audio data

[0497] Output: User's emotional state (e.g., joy)

[0498] How it works: Text and voice data are passed to the emotion engine, which then identifies the user's emotional state. After analysis, it recognizes that the voice contains a tone of joy.

[0499] Step 4: User Intent Analysis

[0500] The server passes the text data to a dialogue engine (OpenAI GPT-3) for analysis to identify the user's intent.

[0501] Input: Text data

[0502] Output: User intent (e.g., search for restaurants)

[0503] How it works: The text data is passed to the dialogue engine, which analyzes the user's intent. As a result of the analysis, it determines that the user wants to search for restaurants.

[0504] Step 5: Generate destination suggestions

[0505] The server generates destination suggestions based on the user's intentions and emotional state.

[0506] Input: User intent, emotional state

[0507] Output: Text data of destination suggestions

[0508] Specific operation: Based on the user's intent of "I want to go to a delicious restaurant" and the emotional state of "delight," the server generates a destination suggestion: "We'll introduce you to some popular restaurants in the area. Would you like to choose from Italian, Japanese, or Chinese cuisine?"

[0509] Step 6: Route calculation

[0510] The server passes the destination information to the navigation module (Google Maps API) and calculates the optimal route.

[0511] Input: Destination information

[0512] Output: Optimal route information

[0513] Specific operation: If the user selects "Italian restaurant," the server sends the destination information to the navigation module, which calculates the optimal route.

[0514] Step 7: Obtaining real-time traffic information

[0515] The server accesses an external traffic information service API to obtain the latest traffic information.

[0516] Input: None

[0517] Output: Real-time traffic information

[0518] Specific operation: The server accesses an external traffic information service and obtains real-time data such as current road conditions, congestion information, and accident information.

[0519] Step 8: Route optimization

[0520] The optimal route is recalculated based on real-time traffic information obtained by the server.

[0521] Input: Real-time traffic information, destination information

[0522] Output: Optimized route information

[0523] Specific operation: The acquired real-time traffic information is reflected in the navigation module, and the optimal route is recalculated to avoid delays and congestion.

[0524] Step 9: Provide route information

[0525] The server sends optimized route information to the device and issues instructions to the autonomous driving system.

[0526] Input: Optimal route information

[0527] Output: None

[0528] Specific operation: The optimized route information is sent to the terminal, and the autonomous driving system begins driving according to the route.

[0529] Step 10: Provide destination information

[0530] In response to user requests, the server retrieves and provides detailed destination information from external databases and APIs.

[0531] Input: User request

[0532] Output: Destination details

[0533] What happens: If a user says, "Tell me more about this restaurant," the server retrieves information from an external database and provides, "This restaurant has a rating of 4.5. The menu includes pasta and pizza. It's open from 11:00 AM to 10:00 PM."

[0534] (Application example 2)

[0535] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0536] When users use voice commands to operate systems such as self-driving vehicles and food delivery services, they need systems that can not only recognize their voices but also make suggestions that take into account the user's emotions and mood. Conventional systems have difficulty analyzing user emotions and providing optimal routes and services, making it difficult to provide personalized services to individual users.

[0537] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for accepting voice input, means for converting the accepted voice into text, means for recognizing the user's emotions from the analyzed text, means for suggesting destinations and services based on the user's emotions and intentions, means for calculating the optimal route and service provision using the information when the destination and service are determined, means for acquiring real-time information, means for selecting the optimal route and service taking the real-time information into consideration, and means for providing information on the selected route and service. This enables personalized suggestions that take the user's emotions and intentions into consideration.

[0538] "Voice input" refers to voice information that a user speaks to a system and that is received through an input device such as a microphone.

[0539] "Converting to text" means converting the received voice information into a string of characters using voice recognition technology.

[0540] "User intent" refers to the wishes, requests, commands, etc. that the user is trying to convey to the system through voice input.

[0541] "User emotion" refers to the emotional state of the user identified by voice analysis, and includes different emotions such as joy, sadness, anger, fatigue, etc.

[0542] "Suggestions" refers to recommendations of destinations and services made by the system based on the user's intentions and emotions.

[0543] An "optimal route" refers to an efficient and safe route to a destination calculated taking into account real-time information.

[0544] "Real-time information" includes current data such as traffic conditions and service usage status, and optimal routes and services are provided based on this information.

[0545] "Selection" means choosing the most appropriate option from multiple options.

[0546] "Provision" refers to the act of the system providing information or services to the user.

[0547] "Autonomous driving system" refers to a system that includes hardware and software for autonomously controlling a vehicle and driving it automatically.

[0548] The present invention provides a personalized system that combines user voice input and emotion recognition to enhance user experience. This system can be applied as a food delivery assistant. Specific embodiments of the system are described below.

[0549] 1. Voice Input Module

[0550] Device:

[0551] Users input their voice using a device such as a smartphone. The device is equipped with a microphone that captures the user's voice. The voice input is then converted into text data using speech recognition software (e.g., Python's speech_recognition library).

[0552] Examples:

[0553] User: "I want to eat some delicious Chinese food today."

[0554] 2. Emotion Recognition Module

[0555] server:

[0556] The text data sent from the voice input module is analyzed by an emotion recognition engine on the server, using, for example, Python's TextBlob library, to identify the user's emotional state (positive, negative, neutral).

[0557] Examples:

[0558] Text data: "I want to eat some delicious Chinese food today."

[0559] Sentiment analysis result: Negative (slightly tired tone)

[0560] 3. Suggestion Engine

[0561] server:

[0562] Based on the user's emotions and intentions identified by the emotion recognition module, the server suggests potential destinations and services. A suggestion engine runs and accesses external restaurant databases and APIs to retrieve appropriate options.

[0563] Examples:

[0564] Server: "You're looking for a Chinese restaurant in the area. It sounds like you're in the mood for a little relaxation. Would you like a quiet restaurant with a festive atmosphere?"

[0565] 4. Optimization Engine

[0566] server:

[0567] If the user accepts the suggestion, the server will obtain real-time information based on the request and calculate the optimal route and service provision, accessing the external traffic information service API to obtain real-time data.

[0568] 5. Service Provision Module

[0569] Device:

[0570] The device provides the information received from the server to the user, who can then select the suggested options or check the detailed information.

[0571] Examples:

[0572] Selected restaurant information: "This restaurant has a 4.5 rating. The menu includes pasta and pizza. It is open from 11:00 AM to 10:00 PM."

[0573] Prompt Sentence Examples

[0574] An example of input a user might make to the system is shown below.

[0575] User: "I want to eat some delicious Chinese food today."

[0576] System (Response): "You're looking for a Chinese restaurant nearby. It sounds like you're looking for a place to relax. Would you like a quiet restaurant with a festive atmosphere?"

[0577] In this way, the system of the present invention provides a more personalized service by simultaneously considering the user's voice input and emotions.

[0578] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0579] Step 1:

[0580] The user speaks, which is captured by the device's microphone. The device converts this speech data into text using a generative AI model (e.g., a speech recognition library). The converted text data is sent to the server.

[0581] input:

[0582] Audio data

[0583] output:

[0584] Text data

[0585] Step 2:

[0586] The server sends the received text data to an emotion recognition engine, which analyzes the emotional state of the text data using a generative AI model (e.g., Python's TextBlob library). The result of the emotion analysis (positive, negative, neutral) is used in the next step.

[0587] input:

[0588] Text data

[0589] output:

[0590] Emotion analysis results

[0591] Step 3:

[0592] The server uses a generative AI model that identifies the user's intent using the sentiment analysis results and text data as input. Based on the user's intent, destination and service candidates are generated. The generated destination and service candidates are stored on the server.

[0593] input:

[0594] Emotion analysis results

[0595] Text data

[0596] output:

[0597] Destination and service suggestions

[0598] Step 4:

[0599] The server proposes optimal options from the generated list of destinations and services based on the user's intentions and the results of emotion analysis. The proposed options are sent to the terminal and presented to the user.

[0600] input:

[0601] Destination and service suggestions

[0602] output:

[0603] suggestion

[0604] Step 5:

[0605] If the user accepts the suggestion, the device sends the selection to the server, which then accesses an external API to obtain real-time information (e.g., traffic conditions and store congestion status). Based on the obtained real-time information, the server calculates the optimal route and service provision.

[0606] input:

[0607] Accepting the proposal

[0608] output:

[0609] Real-time information

[0610] Step 6:

[0611] The server sends the calculated optimal route and service information to the device, which then provides this information to the user, allowing the user to view detailed information.

[0612] input:

[0613] Real-time information

[0614] output:

[0615] Optimal route and service information

[0616] Step 7:

[0617] The user can check the optimal route and service information provided and take action as needed, allowing the user to take optimal action based on personalized information.

[0618] input:

[0619] Optimal route and service information

[0620] output:

[0621] User Behavior

[0622] This series of processing steps allows users to receive personalized suggestions and services through voice input and emotion recognition.

[0623] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0624] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0625] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0626] [Second embodiment]

[0627] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0628] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0629] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0630] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0631] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0632] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0633] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0634] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0635] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0636] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0637] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0638] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0639] The present invention is a system for determining a destination using voice control in a fully automated driving vehicle and efficiently providing an optimal route. Specific embodiments of this system will be described below.

[0640] 1. Speech Recognition Module

[0641] Device:

[0642] The user inputs destinations and travel instructions by voice. The device is equipped with a microphone that captures the voice input and saves it as voice data. The saved voice data is sent to a voice recognition system and converted into text data. This text data is then sent to the server.

[0643] 2. Dialogue Engine

[0644] server:

[0645] The server receives the text data sent from the speech recognition module and analyzes it. This analysis identifies the user's intent and determines the next step. For example, if the user says, "I want to go to a nice restaurant," the server understands that the user is searching for a restaurant and then suggests suitable restaurant options.

[0646] For example, the server might proceed with the following dialogue:

[0647] User: "I want to go to a nice restaurant."

[0648] Server: "We'd like to introduce you to some popular restaurants in the area. Would you like to choose Italian, Japanese, or Chinese?"

[0649] 3. Navigation Module

[0650] server:

[0651] Once the destination is confirmed, the server launches the navigation module to calculate the optimal route to the destination. The navigation module obtains current traffic condition data from the real-time traffic information module and selects the optimal route based on this data.

[0652] The navigation module sends calculated route information to the terminal and provides instructions to the vehicle's autonomous driving system.

[0653] 4. Real-time traffic information module

[0654] server:

[0655] The server accesses an external traffic information service API to obtain the latest traffic information, including road congestion, accidents, construction information, etc. The obtained information is provided to the navigation module and used to calculate the optimal route.

[0656] 5. Destination Information Module

[0657] server:

[0658] The server queries external databases and APIs to obtain detailed information about destinations that the user has expressed interest in. The information obtained (e.g., restaurant ratings, menus, opening hours, etc.) is provided to the user through a dialogue engine.

[0659] For example, if a user requests "Tell me more about this restaurant," the server provides the following information:

[0660] Server: "This restaurant has a 4.5 rating. They have pasta and pizza on the menu. They're open from 11:00 AM to 10:00 PM."

[0661] summary

[0662] The system supports a series of processes, from voice input to destination suggestions, route calculation, real-time traffic information, and detailed destination information, allowing users to intuitively and efficiently operate an autonomous vehicle and providing a comfortable travel experience.

[0663] The processing flow will be explained below.

[0664] Step 1:

[0665] User:

[0666] The user says, "I want to go to a nice restaurant."

[0667] Step 2:

[0668] Device:

[0669] The microphone on the device captures the user's voice and saves it as audio data.

[0670] Step 3:

[0671] Device:

[0672] The saved voice data is sent to a voice recognition service and converted into text data.

[0673] Step 4:

[0674] Device:

[0675] The converted text data is sent to the server.

[0676] Step 5:

[0677] server:

[0678] The server analyzes the received text data and determines the user's intent. In this case, it understands that the user is looking for a restaurant.

[0679] Step 6:

[0680] server:

[0681] The dialogue engine asks the user a question: "We'll introduce you to some popular restaurants in the area. Would you like to choose Italian, Japanese, or Chinese?" The message is sent to the device.

[0682] Step 7:

[0683] User:

[0684] A user says, "I want Italian food."

[0685] Step 8:

[0686] Device:

[0687] The user's voice is captured again and sent to a speech recognition service for conversion into text data.

[0688] Step 9:

[0689] Device:

[0690] The converted text data is sent to the server.

[0691] Step 10:

[0692] server:

[0693] The server analyzes the text data and identifies the user's request. In this case, it understands that the user wants an Italian restaurant.

[0694] Step 11:

[0695] server:

[0696] The server retrieves information about recommended Italian restaurants from an external database and makes suggestions to the user, for example, sending a message to the terminal saying, "Nearby Italian restaurants are Restaurant A, Restaurant B, and Restaurant C."

[0697] Step 12:

[0698] User:

[0699] A user says, "I want to go to Restaurant A."

[0700] Step 13:

[0701] Device:

[0702] The user's voice is captured and sent to a speech recognition service to be converted into text data.

[0703] Step 14:

[0704] Device:

[0705] The converted text data is sent to the server.

[0706] Step 15:

[0707] server:

[0708] The server analyzes the text data and determines that Restaurant A is the destination.

[0709] Step 16:

[0710] server:

[0711] The server passes the destination information to the navigation module and calculates the optimal route to Restaurant A.

[0712] Step 17:

[0713] server:

[0714] The navigation module obtains traffic information from the real-time traffic information module and selects the optimal route taking into account traffic congestion.

[0715] Step 18:

[0716] server:

[0717] The selected route information is sent to the terminal.

[0718] Step 19:

[0719] Device:

[0720] The route information received by the terminal is reflected in the vehicle's autonomous driving system, and the vehicle begins driving automatically toward the destination.

[0721] Step 20:

[0722] server:

[0723] When a user requests "more information about this restaurant," the server retrieves information such as restaurant ratings, menus, and opening hours from an external information service and provides it to the user.

[0724] Example 1

[0725] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0726] Current autonomous vehicle systems do not fully realize the process in which a user can intuitively and efficiently specify a destination using voice input, and the autonomous vehicle selects the optimal route and automatically guides the user to the destination. Therefore, a system is needed that integrates a series of processes: determining a destination using voice control, reflecting real-time traffic information, and providing detailed destination information.

[0727] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0728] In this invention, the server includes means for accepting voice input, means for converting the accepted voice into text, means for identifying the user's intention by analyzing the converted text, means for suggesting a destination based on the user's intention, means for calculating an optimal route using the destination information when the destination is confirmed, means for acquiring real-time traffic information, means for selecting an optimal route taking the real-time traffic information into consideration, means for providing the selected route information to the vehicle's autonomous driving system, and means for querying an external database to acquire detailed destination information and providing it to the user. This allows the user to specify a destination through intuitive voice operation and use a system that integrates optimal route calculation and detailed information provision.

[0729] "Means for accepting voice input" refers to devices or software that capture voice instructions given by a user and convert them into digital data.

[0730] "Means for converting received speech into text" refers to a system that uses speech recognition technology to convert speech data into text data.

[0731] "Means for analyzing the converted text to identify the user's intent" refers to a system that uses natural language processing technology to analyze text data and identify the user's requests and wishes.

[0732] "Means for suggesting destinations based on the user's intentions" refers to a system that predicts the user's desired destination or route and makes suggestions accordingly.

[0733] "A means for calculating the optimal route using information once the destination is confirmed" refers to a system for calculating the shortest distance and the most time-efficient route based on confirmed destination information.

[0734] "Means of obtaining real-time traffic information" refers to APIs and database access means for obtaining current road conditions and traffic data from external services.

[0735] "Means for selecting the optimal route taking into account real-time traffic information" refers to algorithms or systems that determine the optimal route to a destination based on real-time traffic information.

[0736] "Means for providing selected route information to the vehicle's automated driving system" refers to a system that transmits calculated route data to the automated vehicle so that the vehicle can follow that route.

[0737] "Means of querying an external database to obtain detailed destination information and providing it to the user" refers to a system that queries an external database or API for detailed destination information (e.g., opening hours, ratings, menu, etc.) and provides it to the user.

[0738] The present invention provides a fully automated driving system that allows a user to specify a destination using voice input and efficiently provides an optimal route. Specific embodiments of the present invention will be described in detail below.

[0739] 1. Speech Recognition Module

[0740] Device:

[0741] The user inputs destinations and travel instructions by voice. The device is equipped with a microphone that captures the user's voice input. This voice data is sent to a voice recognition system (e.g., voice recognition API) and converted into text data. The converted text data is then sent to the server.

[0742] Examples:

[0743] When a user speaks into a microphone, "I want to go to a nice restaurant," the speech is converted into text.

[0744] 2. Dialogue Engine

[0745] server:

[0746] The server receives the text data sent from the speech recognition module and analyzes it using a natural language processing engine (e.g., a generative AI model). The analysis identifies the user's intent and generates appropriate suggestions as the next step.

[0747] Examples:

[0748] User: "I want to go to a nice restaurant."

[0749] Server: "We'd like to introduce you to some popular restaurants in the area. Would you like to choose Italian, Japanese, or Chinese?"

[0750] 3. Navigation Module

[0751] server:

[0752] Once the destination is confirmed, the server launches the navigation module (e.g., map API) to calculate the optimal route to the destination. This module obtains current traffic condition data from the real-time traffic information module and selects the optimal route. The calculation results are sent to the terminal, which issues instructions to the vehicle's autonomous driving system.

[0753] Examples:

[0754] After the user selects "Japanese food," the server calculates the optimal route to the Japanese restaurant and provides instructions to the autonomous driving system.

[0755] 4. Real-time traffic information module

[0756] server:

[0757] The server accesses an external traffic information service API (e.g., traffic information API) to obtain the latest traffic information. This information includes road congestion, accidents, construction information, etc. The obtained information is provided to the navigation module and used to calculate the optimal route.

[0758] Examples:

[0759] The server obtains traffic congestion information and calculates the optimal detour route.

[0760] 5. Destination Information Module

[0761] server:

[0762] Detailed information about destinations that the user has shown interest in is obtained by querying external databases or APIs (e.g., information provision APIs). The obtained information includes restaurant ratings, menus, opening hours, etc. This information is provided to the user through a dialogue engine.

[0763] Examples:

[0764] User: "Tell me more about this restaurant."

[0765] Server: "This restaurant has a 4.5 rating and offers pasta and pizza. It's open from 11:00 AM to 10:00 PM."

[0766] summary

[0767] The system supports a series of processes, from voice input to destination suggestions, route calculation, real-time traffic information, and detailed destination information, allowing users to intuitively and efficiently operate an autonomous vehicle and enjoy a comfortable travel experience.

[0768] Prompt Sentence Examples

[0769] "In a fully autonomous vehicle system, if a user says, 'I want to go to a nice restaurant,' explain how each module in the system processes the data and provides information to the user."

[0770] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0771] Step 1:

[0772] The user provides voice input.

[0773] Specific action: The user speaks into the microphone, "I want to go to a nice restaurant."

[0774] Input: User's voice command.

[0775] Output: Audio data.

[0776] Step 2:

[0777] The device captures the audio data.

[0778] Specific operation: The microphone installed on the device captures the user's voice and saves it as audio data.

[0779] Input: User's voice.

[0780] Output: The captured audio data.

[0781] Step 3:

[0782] The terminal transmits the voice data to the voice recognition system.

[0783] Specific operation: The device sends the captured voice data to the voice recognition API.

[0784] Input: Audio data.

[0785] Output: Data sent to the speech recognition API.

[0786] Step 4:

[0787] A speech recognition system converts the speech data into text data.

[0788] Specific operation: The speech recognition API analyzes the speech data and converts it into text data.

[0789] Input: Audio data.

[0790] Output: The converted text data.

[0791] Step 5:

[0792] The terminal transmits the text data to the server.

[0793] Specific operation: The terminal sends the converted text data to the server.

[0794] Input: Text data.

[0795] Output: The data sent to the server.

[0796] Step 6:

[0797] The server parses the text data.

[0798] Specific operation: The server sends the received text data to a natural language processing engine (e.g., a generative AI model) for analysis.

[0799] Input: Text data.

[0800] Output: Data identifying the user's intent.

[0801] Step 7:

[0802] The server suggests destinations based on the user's intentions.

[0803] Specific operation: The server generates suggestions and responds to the user in an interactive format.

[0804] Input: Parsed text data.

[0805] Output: Destination suggestion data.

[0806] Step 8:

[0807] The server launches the navigation module.

[0808] Specific operation: After the destination is determined, the server launches the navigation module and calculates the optimal route to the destination.

[0809] Input: Confirmed destination information.

[0810] Output: Launch of navigation module.

[0811] Step 9:

[0812] The server obtains real-time traffic information.

[0813] Specific operation: The server accesses the traffic information API and obtains the latest traffic information (traffic congestion, accidents, construction, etc.).

[0814] Input: A navigation launch request.

[0815] Output: The obtained traffic information.

[0816] Step 10:

[0817] The server calculates the optimal route.

[0818] Specific operation: The server calculates the optimal route based on the acquired real-time traffic information.

[0819] Input: Real-time traffic information.

[0820] Output: Calculated optimal route data.

[0821] Step 11:

[0822] The server sends the route information to the device.

[0823] Specific operation: The server sends the calculated route information to the terminal.

[0824] Input: Optimal route data.

[0825] Output: Route information sent to the device.

[0826] Step 12:

[0827] The terminal issues instructions to the autonomous driving system.

[0828] Specific operation: The terminal issues instructions to the autonomous driving system based on the received route information, and the vehicle moves along the route.

[0829] Input: Route information.

[0830] Output: Instructions to the autonomous driving system.

[0831] Step 13:

[0832] The server obtains the destination information.

[0833] Specific operation: The server queries external databases or APIs to obtain detailed information about the destination (ratings, menu, opening hours, etc.).

[0834] Input: Destination information.

[0835] Output: The obtained destination information.

[0836] Step 14:

[0837] The server provides the destination details to the user.

[0838] Specific operation: The server provides the acquired destination information to the user through the dialogue engine.

[0839] Input: Destination information.

[0840] Output: Detailed information provided to the user.

[0841] (Application example 1)

[0842] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0843] In modern self-driving vehicles, there are systems that allow users to specify their destination via voice and provide the optimal route based on that destination. However, there are problems when users do not specify a specific destination, and there is a lack of systems that provide detailed destination information. There is also a need for dialogue-based destination suggestions that allow users to intuitively select their destination. Furthermore, natural dialogue using prompt sentences generated using generative AI models has not yet been realized. In this situation, a comprehensive and highly functional system is needed to improve user convenience.

[0844] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0845] In this invention, the server includes means for accepting voice input, means for converting voice to text, means for analyzing the text to identify a user's intent, means for suggesting a destination based on the user's intent, means for calculating an optimal route when the destination is confirmed, means for acquiring real-time traffic information, means for selecting an optimal route, means for providing the selected route information to the vehicle's autonomous driving system, means for acquiring detailed information about the identified destination from an external database and providing it to the user, and means for suggesting destinations in an interactive format using a generative AI model and generating prompt sentences to help the user make an appropriate selection. This allows the user to intuitively and efficiently determine their destination through voice operation and travel along the optimal route while obtaining detailed information.

[0846] "Means for accepting voice input" refers to means for capturing voice uttered by a user and inputting the voice data into the system.

[0847] The "means for converting voice to text" is a means for converting received voice data into text data in a specified language.

[0848] "Means for analyzing text to identify user intent" refers to means for analyzing converted text data and recognizing and identifying the user's intent.

[0849] The "means for suggesting a destination based on the user's intention" is a means for suggesting destination candidates that match the specified user's intention.

[0850] The "means for calculating the optimum route when the destination is determined" is a means for calculating the optimum route to reach the determined destination.

[0851] "Means for obtaining real-time traffic information" refers to means for obtaining real-time traffic data such as current traffic conditions, congestion, and accident information.

[0852] The "means for selecting an optimal route" is a means for selecting the most efficient route in consideration of the acquired real-time traffic information.

[0853] "Means for providing selected route information to the vehicle's automated driving system" refers to means for issuing driving instructions to the vehicle's automated driving system based on the selected route.

[0854] The "means for obtaining detailed information about the identified destination from an external database and providing it to the user" refers to a means for obtaining detailed information about the destination from an external database and providing it to the user.

[0855] "A means of using a generative AI model to suggest destinations in an interactive format and generate prompts to help the user make an appropriate choice" refers to a means of utilizing generative AI to suggest destinations in a natural interactive format and generate prompts to guide the user in making an appropriate choice.

[0856] This invention is a system for fully automated driving vehicles that uses voice control to determine destinations and efficiently provide optimal routes. This system supports everything from voice input to destination suggestions, route calculations, real-time traffic information, and detailed destination information.

[0857] Voice Input and Recognition Module

[0858] The terminal is a device that allows users to input destinations and travel instructions by voice. The terminal is equipped with a microphone that captures voice input and sends the data to a speech recognition module (e.g., Google Cloud Speech-to-Text API), which converts the voice data into text.

[0859] Text analysis and dialogue engine

[0860] The server analyzes the text data sent from the speech recognition module and identifies the user's intention. A dialogue engine (e.g., Dialogflow) is used for the analysis, and suggests destinations in a dialogue format. For example, if the user says, "I want to go to a nice restaurant," the server responds as follows:

[0861] "We'll introduce you to popular restaurants in the area. You can choose from Italian, Japanese, or Chinese cuisine."

[0862] Destination and navigation

[0863] Once the destination is determined, the server launches the navigation module, which uses the Google Maps API to calculate the optimal route to the destination, retrieves real-time traffic information (e.g., Google Maps Traffic API), selects the optimal route, and provides that information to the vehicle's autonomous driving system.

[0864] Real-time traffic information

[0865] The server accesses an external traffic information service API to obtain the latest traffic information, including road congestion, accidents, and construction information, and uses this information to calculate the optimal route in the navigation module.

[0866] Providing destination information

[0867] The server queries an external database or API (e.g., Yelp API) for details about the destination selected by the user and provides the information obtained to the user. For example, if the user requests "Tell me the details of this restaurant," the server provides the following information:

[0868] "This restaurant has a 4.5 rating, offers pasta and pizza on the menu, and is open from 11am to 10pm."

[0869] Leveraging generative AI models

[0870] The server uses a generative AI model to interactively suggest destinations and generate prompts to help users make appropriate choices. This makes it easier for users to decide on a destination through natural dialogue. Here are some examples of prompts:

[0871] "I want to go to a nice restaurant."

[0872] "We'll introduce you to popular restaurants in the area. You can choose from Italian, Japanese, or Chinese cuisine."

[0873] "If you select Italian, we will retrieve and provide detailed information about your destination to you."

[0874] This will enable users to intuitively and efficiently operate autonomous vehicles, providing a comfortable travel experience.

[0875] As described above, the system of the present invention comprehensively supports everything from voice input to destination setting, route guidance, and the provision of detailed information, and by utilizing a generative AI model in the process, it provides users with high convenience and intuitive operability.

[0876] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0877] Step 1:

[0878] Accepting voice input

[0879] The user inputs destinations and travel instructions into the device by voice, which is captured through the device's microphone and saved as audio data.

[0880] Input: User's voice command

[0881] Output: Captured audio data

[0882] Step 2:

[0883] Converting audio data to text

[0884] The device sends the captured voice data to a voice recognition module (e.g., Google Cloud Speech-to-Text API) and converts the voice data into text data.

[0885] Input: Audio data

[0886] Output: Converted text data

[0887] Step 3:

[0888] Text data analysis

[0889] The server analyzes the converted text data to identify the user's intent. A dialogue engine (e.g., Dialogflow) is used for the analysis, and the user's intent regarding destination and route is identified.

[0890] Input: Text data

[0891] Output: Identified user intent data

[0892] Step 4:

[0893] Destination suggestions

[0894] The server suggests destinations based on the user's intentions, and uses a generative AI model to interactively suggest potential destinations and generate prompts to help the user make the appropriate choice.

[0895] Input: User intent data

[0896] Output: Possible destinations and prompts

[0897] Step 5:

[0898] Destination determination and route calculation

[0899] Once the user has decided on a destination, the server uses the Google Maps API to calculate the optimal route to the destination.

[0900] Input: Selected destination

[0901] Output: Optimal route information

[0902] Step 6:

[0903] Obtaining real-time traffic information

[0904] The server uses external traffic information service APIs, such as Google Maps Traffic API, to obtain the latest traffic information, including road congestion, accidents, and construction information.

[0905] Input: None (Access to external traffic information services)

[0906] Output: Real-time traffic information

[0907] Step 7:

[0908] Selection of the optimal route

[0909] The server selects the optimal route taking into account real-time traffic information.

[0910] Input: Initial route calculation, real-time traffic information

[0911] Output: Optimal route reflecting traffic information

[0912] Step 8:

[0913] Providing route information

[0914] The server provides the selected route information to the vehicle's autonomous driving system and controls the vehicle's driving.

[0915] Input: Optimal route information

[0916] Output: Driving instructions to the vehicle's autonomous driving system

[0917] Step 9:

[0918] Providing destination information

[0919] The server queries an external database (e.g., Yelp API) for details about the identified destination and provides the retrieved information to the user.

[0920] Input: Confirmed destination

[0921] Output: Destination details (ratings, menu, opening hours, etc.)

[0922] This allows users to intuitively and efficiently determine their destination through voice control, and travel along the optimal route while obtaining detailed information.

[0923] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0924] The present invention provides a system for determining a destination using voice control in a fully automated driving vehicle, recognizing the user's emotions, and efficiently providing an optimal route. Specific embodiments of this system are described below.

[0925] 1. Speech Recognition Module

[0926] Device:

[0927] The user inputs destinations and travel instructions by voice. The device is equipped with a microphone that captures the voice input and saves it as voice data. The saved voice data is sent to a voice recognition system and converted into text data. This text data is then sent to the server.

[0928] 2. Emotion Engine

[0929] server:

[0930] The server analyzes the text data and voice sent from the voice recognition module to identify the user's emotions. Using an emotion engine, it recognizes the user's emotional state (e.g., joy, anger, sadness, etc.) and records the results. This makes it possible to understand the emotional state of the user when giving instructions and provide appropriate responses and destination suggestions.

[0931] For example, the emotion engine works as follows:

[0932] User: "I want to go to a nice restaurant." (Audio contains a tone of delight)

[0933] Server: "Here are some popular restaurants in the area. Would you like to choose Italian, Japanese, or Chinese?" (The emotion engine recognizes that the user is happy and responds in a positive tone.)

[0934] 3. Dialogue Engine

[0935] server:

[0936] The server analyzes the text data received from the speech recognition module to identify the user's intent. Through this analysis, the server understands the user's intent (command, question, suggestion, etc.) and determines the next step. The server also takes into account the results of the emotion engine to generate optimal destination suggestions and responses.

[0937] 4. Navigation Module

[0938] server:

[0939] When the server receives the destination information, the navigation module is activated and calculates the optimal route to the destination. The navigation module obtains the latest traffic information from the real-time traffic information module and selects the optimal route based on that information.

[0940] The navigation module sends calculated route information to the terminal and provides instructions to the vehicle's autonomous driving system.

[0941] 5. Real-time traffic information module

[0942] server:

[0943] The server accesses an external traffic information service API to obtain the latest traffic information, including road congestion, accidents, construction information, etc. The obtained information is provided to the navigation module and used to calculate the optimal route.

[0944] 6. Destination Information Module

[0945] server:

[0946] The server queries external databases and APIs to obtain detailed information about destinations that the user has expressed interest in. The information obtained (e.g., restaurant ratings, menus, opening hours, etc.) is provided to the user through a dialogue engine.

[0947] For example, if the user says "Tell me more about this restaurant," the server provides the following information:

[0948] Server: "This restaurant has a 4.5 rating. They have pasta and pizza on the menu. They're open from 11:00 AM to 10:00 PM."

[0949] summary

[0950] The system covers a range of processes, from voice input to emotion recognition, destination suggestions, route calculation, real-time traffic information, and detailed destination information. This allows users to not only intuitively operate an autonomous vehicle, but also to receive more detailed emotional services, providing a comfortable travel experience.

[0951] The processing flow will be explained below.

[0952] Step 1:

[0953] User:

[0954] The user says, "I want to go to a nice restaurant."

[0955] Step 2:

[0956] Device:

[0957] The microphone on the device captures the user's voice and saves it as audio data.

[0958] Step 3:

[0959] Device:

[0960] The saved voice data is sent to a voice recognition service and converted into text data.

[0961] Step 4:

[0962] Device:

[0963] The converted text data is sent to the server.

[0964] Step 5:

[0965] server:

[0966] The server analyzes the received text data and determines the user's intent. In this case, it understands that the user is looking for a restaurant.

[0967] Step 6:

[0968] server:

[0969] The emotion engine is used to analyze the user's emotions from the received voice data, for example, recognizing that the user is happy from the tone and accent of the voice.

[0970] Step 7:

[0971] server:

[0972] Based on the analysis results of the emotion engine, the dialogue engine asks the user a question: "We'll introduce you to some popular restaurants in the area. Would you like to choose from Italian, Japanese, or Chinese cuisine?" The message is sent to the device.

[0973] Step 8:

[0974] User:

[0975] A user says, "I want Italian food."

[0976] Step 9:

[0977] Device:

[0978] The user's voice is captured again and sent to a speech recognition service for conversion into text data.

[0979] Step 10:

[0980] Device:

[0981] The converted text data is sent to the server.

[0982] Step 11:

[0983] server:

[0984] The server analyzes the text data and identifies the user's request. In this case, it understands that the user wants an Italian restaurant.

[0985] Step 12:

[0986] server:

[0987] The server retrieves information about recommended Italian restaurants from an external database and makes suggestions to the user, for example, sending a message to the terminal saying, "Nearby Italian restaurants are Restaurant A, Restaurant B, and Restaurant C."

[0988] Step 13:

[0989] User:

[0990] A user says, "I want to go to Restaurant A."

[0991] Step 14:

[0992] Device:

[0993] The user's voice is captured and sent to a speech recognition service to be converted into text data.

[0994] Step 15:

[0995] Device:

[0996] The converted text data is sent to the server.

[0997] Step 16:

[0998] server:

[0999] The server analyzes the text data and determines that Restaurant A is the destination.

[1000] Step 17:

[1001] server:

[1002] The server launches the navigation module and calculates the optimal route to the destination, Restaurant A.

[1003] Step 18:

[1004] server:

[1005] The navigation module accesses the real-time traffic information module to obtain the latest traffic information, including congestion, traffic accidents, and construction information.

[1006] Step 19:

[1007] server:

[1008] The navigation module takes into account traffic information obtained and selects the optimal route.

[1009] Step 20:

[1010] server:

[1011] The selected route information is sent to the terminal.

[1012] Step 21:

[1013] Device:

[1014] The route information received by the terminal is reflected in the vehicle's autonomous driving system, and the vehicle begins driving automatically toward the destination.

[1015] Step 22:

[1016] User:

[1017] A user says, "Tell me more about this restaurant."

[1018] Step 23:

[1019] Device:

[1020] The user's voice is captured and sent to a speech recognition service to be converted into text data.

[1021] Step 24:

[1022] Device:

[1023] The converted text data is sent to the server.

[1024] Step 25:

[1025] server:

[1026] The server parses the text data and understands the user's request.

[1027] Step 26:

[1028] server:

[1029] The server accesses an external information service to obtain detailed restaurant information (ratings, menu, opening hours, etc.).

[1030] Step 27:

[1031] server:

[1032] The acquired detailed information is then provided to the user through a dialogue engine, for example, "This restaurant has a rating of 4.5. The menu includes pasta and pizza. It is open from 11:00 AM to 10:00 PM."

[1033] Example 2

[1034] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1035] Conventional autonomous driving systems require users to manually input their destination and set up routes, resulting in unintuitive operation. Furthermore, they do not take into account the user's emotional state, resulting in issues with not being able to provide optimal service or responses. Furthermore, they are not sufficient in selecting optimal routes that take real-time traffic information into account, leaving a need for improving the quality of the travel experience.

[1036] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1037] In this invention, the server includes means for accepting voice input, means for converting the accepted voice into text, means for analyzing the converted text and voice data to identify the user's emotional state, means for identifying the user's intention based on the analyzed text data, means for suggesting a destination based on the identified user's intention and emotional state, means for calculating an optimal route using the destination information when the destination is confirmed, means for acquiring real-time traffic information from an external traffic information service, means for selecting an optimal route taking into account the acquired real-time traffic information, and means for providing the selected route information to the vehicle's autonomous driving system. This allows the user to specify a destination in an intuitive manner, provides appropriate services and responses taking into account the user's emotional state, and enables optimal route selection based on real-time traffic information.

[1038] "Means for accepting voice input" refers to a device or software that has the function of recognizing voice uttered by a user and inputting it into the system.

[1039] "Means for converting received speech into text" refers to a technology or system for analyzing speech data and converting it into text data.

[1040] "Means for analyzing converted text and voice data to identify a user's emotional state" refers to technology or systems that recognize and identify a user's emotions (e.g., joy, anger, sadness, etc.) by analyzing text and voice data.

[1041] "Means for identifying a user's intent based on analyzed text data" refers to a technology or system that analyzes text data to understand a user's instructions or requests and identify their intent.

[1042] "Means for suggesting destinations based on identified user intentions and emotional state" refers to a technology or system for suggesting appropriate destinations taking into account a user's intentions and emotional state.

[1043] "Means for calculating the optimal route using information once the destination is determined" refers to technology or a system that calculates the optimal route to a destination based on the destination specified or agreed to by the user.

[1044] "Means for obtaining real-time traffic information from external traffic information services" refers to a technology or system that accesses external traffic information services and obtains the latest traffic situation data.

[1045] "Means for selecting the optimal route taking into account acquired real-time traffic information" refers to a technology or system that selects the most efficient route based on real-time traffic conditions.

[1046] "Means for providing selected route information to the vehicle's automated driving system" refers to a technology or system that communicates the selected route information to the vehicle's automated driving system and issues instructions.

[1047] This invention is a system for fully autonomous vehicles that uses voice control to determine destinations, recognizes the user's emotions, and efficiently provides the optimal route. This system is mainly composed of the following modules.

[1048] 1. Voice Input Module

[1049] Device:

[1050] The user inputs destinations and travel instructions by voice. The device is equipped with a microphone that captures the user's voice and saves it as audio data. The saved audio data is then sent to a server. This process uses a common hardware microphone and voice recognition software called a "voice recognition API" (for example, Google Cloud Speech-to-Text API).

[1051] For example, when a user says, "I want to go to a nice restaurant," the microphone on the device captures the voice and generates voice data, which is then sent to the server.

[1052] 2. Speech Recognition and Text Conversion

[1053] server:

[1054] The server passes the received voice data to a voice recognition engine (for example, Google Cloud Speech-to-Text API) and converts the voice data into text data, which is then stored on the server.

[1055] 3. Emotion Recognition Engine

[1056] server:

[1057] An emotion engine (e.g., IBM Watson Tone Analyzer) analyzes the converted text data and the original voice data to identify the user's emotional state (e.g., joy, anger, sadness, etc.).

[1058] For example, if a user says, "I want to go to a nice restaurant," the voice contains a tone of joy. The server uses an emotion engine to recognize the emotional state and determine that the user is happy.

[1059] 4. Dialogue Engine

[1060] server:

[1061] The server analyzes the text data received from the speech recognition module. A dialogue engine (e.g., OpenAI GPT-3) is used to identify the user's intent. Through the analysis, the user's intent (e.g., command, question, suggestion, etc.) is understood.

[1062] 5. Destination suggestions

[1063] server:

[1064] Destination suggestions are made based on the user's intent and emotional state. The server combines the identified user intent and emotional state to generate optimal destination suggestions and responses.

[1065] For example, if a user says, "I want to go to a nice restaurant," the server will suggest, "We'll introduce you to some popular restaurants in the area. Would you like to choose from Italian, Japanese, or Chinese cuisine?"

[1066] 6. Route calculation

[1067] server:

[1068] Once the destination information is confirmed, the navigation module (e.g., Google Maps API) is used to calculate the optimal route to the destination. The latest traffic information is obtained from the real-time traffic information module, and the optimal route is selected based on that information.

[1069] As a concrete example, if you select "Italian restaurant," the server will calculate the optimal route through the navigation module.

[1070] 7. Real-time traffic information

[1071] server:

[1072] The server obtains the latest traffic information from an external traffic information service API and provides it to the navigation module, which then selects the optimal route based on the real-time traffic information.

[1073] 8. Providing route information

[1074] server:

[1075] The calculated route information is sent to the terminal and instructions are sent to the autonomous driving system, which then guides the vehicle to its destination.

[1076] 9. Providing destination information

[1077] server:

[1078] Detailed information about the destinations the user has expressed interest in is retrieved from external databases and APIs and provided to the user, including suggested restaurant details such as ratings, menus, and opening hours.

[1079] For example, if a user says, "Tell me more about this restaurant," the server might provide information like, "This restaurant has a rating of 4.5. The menu includes pasta and pizza. It's open from 11:00 a.m. to 10:00 p.m."

[1080] Prompt Sentence Examples

[1081] User: "I want to go to a nice restaurant."

[1082] This system covers a range of processes, from voice input to emotion recognition, destination suggestions, route calculation, real-time traffic information, and detailed destination information, thereby providing users with intuitive and emotionally responsive services and a comfortable travel experience.

[1083] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1084] Step 1: Getting voice input

[1085] The user inputs destinations and instructions into the terminal by voice.

[1086] The device captures the user's voice with a microphone and saves it as audio data.

[1087] Input: User's voice commands

[1088] Output: Audio data

[1089] How it works: When a user says, "I want to go to a nice restaurant," the microphone captures the voice, which is converted into digital audio data and stored on the device. This audio data is then sent to the server.

[1090] Step 2: Speech recognition and text conversion

[1091] The server passes the received voice data to a voice recognition engine (Google Cloud Speech-to-Text API) and converts the voice data into text data.

[1092] Input: Audio data

[1093] Output: Text data

[1094] Specific operation: The voice data is sent to the server and passed to the speech recognition engine. This engine analyzes the voice data and converts it into text data such as "I want to go to a delicious restaurant."

[1095] Step 3: Emotion Recognition

[1096] The server passes the text and voice data to an emotion engine (IBM Watson Tone Analyzer) to analyze the user's emotions.

[1097] Input: Text data, audio data

[1098] Output: User's emotional state (e.g., joy)

[1099] How it works: Text and voice data are passed to the emotion engine, which then identifies the user's emotional state. After analysis, it recognizes that the voice contains a tone of joy.

[1100] Step 4: User Intent Analysis

[1101] The server passes the text data to a dialogue engine (OpenAI GPT-3) for analysis to identify the user's intent.

[1102] Input: Text data

[1103] Output: User intent (e.g., search for restaurants)

[1104] How it works: The text data is passed to the dialogue engine, which analyzes the user's intent. As a result of the analysis, it determines that the user wants to search for restaurants.

[1105] Step 5: Generate destination suggestions

[1106] The server generates destination suggestions based on the user's intentions and emotional state.

[1107] Input: User intent, emotional state

[1108] Output: Text data of destination suggestions

[1109] Specific operation: Based on the user's intent of "I want to go to a delicious restaurant" and the emotional state of "delight," the server generates a destination suggestion: "We'll introduce you to some popular restaurants in the area. Would you like to choose from Italian, Japanese, or Chinese cuisine?"

[1110] Step 6: Route calculation

[1111] The server passes the destination information to the navigation module (Google Maps API) and calculates the optimal route.

[1112] Input: Destination information

[1113] Output: Optimal route information

[1114] Specific operation: If the user selects "Italian restaurant," the server sends the destination information to the navigation module, which calculates the optimal route.

[1115] Step 7: Obtaining real-time traffic information

[1116] The server accesses an external traffic information service API to obtain the latest traffic information.

[1117] Input: None

[1118] Output: Real-time traffic information

[1119] Specific operation: The server accesses an external traffic information service and obtains real-time data such as current road conditions, congestion information, and accident information.

[1120] Step 8: Route optimization

[1121] The optimal route is recalculated based on real-time traffic information obtained by the server.

[1122] Input: Real-time traffic information, destination information

[1123] Output: Optimized route information

[1124] Specific operation: The acquired real-time traffic information is reflected in the navigation module, and the optimal route is recalculated to avoid delays and congestion.

[1125] Step 9: Provide route information

[1126] The server sends optimized route information to the device and issues instructions to the autonomous driving system.

[1127] Input: Optimal route information

[1128] Output: None

[1129] Specific operation: The optimized route information is sent to the terminal, and the autonomous driving system begins driving according to the route.

[1130] Step 10: Provide destination information

[1131] In response to user requests, the server retrieves and provides detailed destination information from external databases and APIs.

[1132] Input: User request

[1133] Output: Destination details

[1134] What happens: If a user says, "Tell me more about this restaurant," the server retrieves information from an external database and provides, "This restaurant has a rating of 4.5. The menu includes pasta and pizza. It's open from 11:00 AM to 10:00 PM."

[1135] (Application example 2)

[1136] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1137] When users use voice commands to operate systems such as self-driving vehicles and food delivery services, they need systems that can not only recognize their voices but also make suggestions that take into account the user's emotions and mood. Conventional systems have difficulty analyzing user emotions and providing optimal routes and services, making it difficult to provide personalized services to individual users.

[1138] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for accepting voice input, means for converting the accepted voice into text, means for recognizing the user's emotions from the analyzed text, means for suggesting destinations and services based on the user's emotions and intentions, means for calculating the optimal route and service provision using the information when the destination and service are determined, means for acquiring real-time information, means for selecting the optimal route and service taking the real-time information into consideration, and means for providing information on the selected route and service. This enables personalized suggestions that take the user's emotions and intentions into consideration.

[1139] "Voice input" refers to voice information that a user speaks to a system and that is received through an input device such as a microphone.

[1140] "Converting to text" means converting the received voice information into a string of characters using voice recognition technology.

[1141] "User intent" refers to the wishes, requests, commands, etc. that the user is trying to convey to the system through voice input.

[1142] "User emotion" refers to the emotional state of the user identified by voice analysis, and includes different emotions such as joy, sadness, anger, fatigue, etc.

[1143] "Suggestions" refers to recommendations of destinations and services made by the system based on the user's intentions and emotions.

[1144] An "optimal route" refers to an efficient and safe route to a destination calculated taking into account real-time information.

[1145] "Real-time information" includes current data such as traffic conditions and service usage status, and optimal routes and services are provided based on this information.

[1146] "Selection" means choosing the most appropriate option from multiple options.

[1147] "Provision" refers to the act of the system providing information or services to the user.

[1148] "Autonomous driving system" refers to a system that includes hardware and software for autonomously controlling a vehicle and driving it automatically.

[1149] The present invention provides a personalized system that combines user voice input and emotion recognition to enhance user experience. This system can be applied as a food delivery assistant. Specific embodiments of the system are described below.

[1150] 1. Voice Input Module

[1151] Device:

[1152] Users input their voice using a device such as a smartphone. The device is equipped with a microphone that captures the user's voice. The voice input is then converted into text data using speech recognition software (e.g., Python's speech_recognition library).

[1153] Examples:

[1154] User: "I want to eat some delicious Chinese food today."

[1155] 2. Emotion Recognition Module

[1156] server:

[1157] The text data sent from the voice input module is analyzed by an emotion recognition engine on the server, using, for example, Python's TextBlob library, to identify the user's emotional state (positive, negative, neutral).

[1158] Examples:

[1159] Text data: "I want to eat some delicious Chinese food today."

[1160] Sentiment analysis result: Negative (slightly tired tone)

[1161] 3. Suggestion Engine

[1162] server:

[1163] Based on the user's emotions and intentions identified by the emotion recognition module, the server suggests potential destinations and services. A suggestion engine runs and accesses external restaurant databases and APIs to retrieve appropriate options.

[1164] Examples:

[1165] Server: "You're looking for a Chinese restaurant in the area. It sounds like you're in the mood for a little relaxation. Would you like a quiet restaurant with a festive atmosphere?"

[1166] 4. Optimization Engine

[1167] server:

[1168] If the user accepts the suggestion, the server will obtain real-time information based on the request and calculate the optimal route and service provision, accessing the external traffic information service API to obtain real-time data.

[1169] 5. Service Provision Module

[1170] Device:

[1171] The device provides the information received from the server to the user, who can then select the suggested options or check the detailed information.

[1172] Examples:

[1173] Selected restaurant information: "This restaurant has a 4.5 rating. The menu includes pasta and pizza. It is open from 11:00 AM to 10:00 PM."

[1174] Prompt Sentence Examples

[1175] An example of input a user might make to the system is shown below.

[1176] User: "I want to eat some delicious Chinese food today."

[1177] System (Response): "You're looking for a Chinese restaurant nearby. It sounds like you're looking for a place to relax. Would you like a quiet restaurant with a festive atmosphere?"

[1178] In this way, the system of the present invention provides a more personalized service by simultaneously considering the user's voice input and emotions.

[1179] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1180] Step 1:

[1181] The user speaks, which is captured by the device's microphone. The device converts this speech data into text using a generative AI model (e.g., a speech recognition library). The converted text data is sent to the server.

[1182] input:

[1183] Audio data

[1184] output:

[1185] Text data

[1186] Step 2:

[1187] The server sends the received text data to an emotion recognition engine, which analyzes the emotional state of the text data using a generative AI model (e.g., Python's TextBlob library). The result of the emotion analysis (positive, negative, neutral) is used in the next step.

[1188] input:

[1189] Text data

[1190] output:

[1191] Emotion analysis results

[1192] Step 3:

[1193] The server uses a generative AI model that identifies the user's intent using the sentiment analysis results and text data as input. Based on the user's intent, destination and service candidates are generated. The generated destination and service candidates are stored on the server.

[1194] input:

[1195] Emotion analysis results

[1196] Text data

[1197] output:

[1198] Destination and service suggestions

[1199] Step 4:

[1200] The server proposes optimal options from the generated list of destinations and services based on the user's intentions and the results of emotion analysis. The proposed options are sent to the terminal and presented to the user.

[1201] input:

[1202] Destination and service suggestions

[1203] output:

[1204] suggestion

[1205] Step 5:

[1206] If the user accepts the suggestion, the device sends the selection to the server, which then accesses an external API to obtain real-time information (e.g., traffic conditions and store congestion status). Based on the obtained real-time information, the server calculates the optimal route and service provision.

[1207] input:

[1208] Accepting the proposal

[1209] output:

[1210] Real-time information

[1211] Step 6:

[1212] The server sends the calculated optimal route and service information to the device, which then provides this information to the user, allowing the user to view detailed information.

[1213] input:

[1214] Real-time information

[1215] output:

[1216] Optimal route and service information

[1217] Step 7:

[1218] The user can check the optimal route and service information provided and take action as needed, allowing the user to take optimal action based on personalized information.

[1219] input:

[1220] Optimal route and service information

[1221] output:

[1222] User Behavior

[1223] This series of processing steps allows users to receive personalized suggestions and services through voice input and emotion recognition.

[1224] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1225] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1226] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1227] [Third embodiment]

[1228] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1229] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1230] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1231] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1232] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1233] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1234] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1235] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1236] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1237] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1238] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1239] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1240] The present invention is a system for determining a destination using voice control in a fully automated driving vehicle and efficiently providing an optimal route. Specific embodiments of this system will be described below.

[1241] 1. Speech Recognition Module

[1242] Device:

[1243] The user inputs destinations and travel instructions by voice. The device is equipped with a microphone that captures the voice input and saves it as voice data. The saved voice data is sent to a voice recognition system and converted into text data. This text data is then sent to the server.

[1244] 2. Dialogue Engine

[1245] server:

[1246] The server receives the text data sent from the speech recognition module and analyzes it. This analysis identifies the user's intent and determines the next step. For example, if the user says, "I want to go to a nice restaurant," the server understands that the user is searching for a restaurant and then suggests suitable restaurant options.

[1247] For example, the server might proceed with the following dialogue:

[1248] User: "I want to go to a nice restaurant."

[1249] Server: "We'd like to introduce you to some popular restaurants in the area. Would you like to choose Italian, Japanese, or Chinese?"

[1250] 3. Navigation Module

[1251] server:

[1252] Once the destination is confirmed, the server launches the navigation module to calculate the optimal route to the destination. The navigation module obtains current traffic condition data from the real-time traffic information module and selects the optimal route based on this data.

[1253] The navigation module sends calculated route information to the terminal and provides instructions to the vehicle's autonomous driving system.

[1254] 4. Real-time traffic information module

[1255] server:

[1256] The server accesses an external traffic information service API to obtain the latest traffic information, including road congestion, accidents, construction information, etc. The obtained information is provided to the navigation module and used to calculate the optimal route.

[1257] 5. Destination Information Module

[1258] server:

[1259] The server queries external databases and APIs to obtain detailed information about destinations that the user has expressed interest in. The information obtained (e.g., restaurant ratings, menus, opening hours, etc.) is provided to the user through a dialogue engine.

[1260] For example, if a user requests "Tell me more about this restaurant," the server provides the following information:

[1261] Server: "This restaurant has a 4.5 rating. They have pasta and pizza on the menu. They're open from 11:00 AM to 10:00 PM."

[1262] summary

[1263] The system supports a series of processes, from voice input to destination suggestions, route calculation, real-time traffic information, and detailed destination information, allowing users to intuitively and efficiently operate an autonomous vehicle and providing a comfortable travel experience.

[1264] The processing flow will be explained below.

[1265] Step 1:

[1266] User:

[1267] The user says, "I want to go to a nice restaurant."

[1268] Step 2:

[1269] Device:

[1270] The microphone on the device captures the user's voice and saves it as audio data.

[1271] Step 3:

[1272] Device:

[1273] The saved voice data is sent to a voice recognition service and converted into text data.

[1274] Step 4:

[1275] Device:

[1276] The converted text data is sent to the server.

[1277] Step 5:

[1278] server:

[1279] The server analyzes the received text data and determines the user's intent. In this case, it understands that the user is looking for a restaurant.

[1280] Step 6:

[1281] server:

[1282] The dialogue engine asks the user a question: "We'll introduce you to some popular restaurants in the area. Would you like to choose Italian, Japanese, or Chinese?" The message is sent to the device.

[1283] Step 7:

[1284] User:

[1285] A user says, "I want Italian food."

[1286] Step 8:

[1287] Device:

[1288] The user's voice is captured again and sent to a speech recognition service for conversion into text data.

[1289] Step 9:

[1290] Device:

[1291] The converted text data is sent to the server.

[1292] Step 10:

[1293] server:

[1294] The server analyzes the text data and identifies the user's request. In this case, it understands that the user wants an Italian restaurant.

[1295] Step 11:

[1296] server:

[1297] The server retrieves information about recommended Italian restaurants from an external database and makes suggestions to the user, for example, sending a message to the terminal saying, "Nearby Italian restaurants are Restaurant A, Restaurant B, and Restaurant C."

[1298] Step 12:

[1299] User:

[1300] A user says, "I want to go to Restaurant A."

[1301] Step 13:

[1302] Device:

[1303] The user's voice is captured and sent to a speech recognition service to be converted into text data.

[1304] Step 14:

[1305] Device:

[1306] The converted text data is sent to the server.

[1307] Step 15:

[1308] server:

[1309] The server analyzes the text data and determines that Restaurant A is the destination.

[1310] Step 16:

[1311] server:

[1312] The server passes the destination information to the navigation module and calculates the optimal route to Restaurant A.

[1313] Step 17:

[1314] server:

[1315] The navigation module obtains traffic information from the real-time traffic information module and selects the optimal route taking into account traffic congestion.

[1316] Step 18:

[1317] server:

[1318] The selected route information is sent to the terminal.

[1319] Step 19:

[1320] Device:

[1321] The route information received by the terminal is reflected in the vehicle's autonomous driving system, and the vehicle begins driving automatically toward the destination.

[1322] Step 20:

[1323] server:

[1324] When a user requests "more information about this restaurant," the server retrieves information such as restaurant ratings, menus, and opening hours from an external information service and provides it to the user.

[1325] Example 1

[1326] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1327] Current autonomous vehicle systems do not fully realize the process in which a user can intuitively and efficiently specify a destination using voice input, and the autonomous vehicle selects the optimal route and automatically guides the user to the destination. Therefore, a system is needed that integrates a series of processes: determining a destination using voice control, reflecting real-time traffic information, and providing detailed destination information.

[1328] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1329] In this invention, the server includes means for accepting voice input, means for converting the accepted voice into text, means for identifying the user's intention by analyzing the converted text, means for suggesting a destination based on the user's intention, means for calculating an optimal route using the destination information when the destination is confirmed, means for acquiring real-time traffic information, means for selecting an optimal route taking the real-time traffic information into consideration, means for providing the selected route information to the vehicle's autonomous driving system, and means for querying an external database to acquire detailed destination information and providing it to the user. This allows the user to specify a destination through intuitive voice operation and use a system that integrates optimal route calculation and detailed information provision.

[1330] "Means for accepting voice input" refers to devices or software that capture voice instructions given by a user and convert them into digital data.

[1331] "Means for converting received speech into text" refers to a system that uses speech recognition technology to convert speech data into text data.

[1332] "Means for analyzing the converted text to identify the user's intent" refers to a system that uses natural language processing technology to analyze text data and identify the user's requests and wishes.

[1333] "Means for suggesting destinations based on the user's intentions" refers to a system that predicts the user's desired destination or route and makes suggestions accordingly.

[1334] "A means for calculating the optimal route using information once the destination is confirmed" refers to a system for calculating the shortest distance and the most time-efficient route based on confirmed destination information.

[1335] "Means of obtaining real-time traffic information" refers to APIs and database access means for obtaining current road conditions and traffic data from external services.

[1336] "Means for selecting the optimal route taking into account real-time traffic information" refers to algorithms or systems that determine the optimal route to a destination based on real-time traffic information.

[1337] "Means for providing selected route information to the vehicle's automated driving system" refers to a system that transmits calculated route data to the automated vehicle so that the vehicle can follow that route.

[1338] "Means of querying an external database to obtain detailed destination information and providing it to the user" refers to a system that queries an external database or API for detailed destination information (e.g., opening hours, ratings, menu, etc.) and provides it to the user.

[1339] The present invention provides a fully automated driving system that allows a user to specify a destination using voice input and efficiently provides an optimal route. Specific embodiments of the present invention will be described in detail below.

[1340] 1. Speech Recognition Module

[1341] Device:

[1342] The user inputs destinations and travel instructions by voice. The device is equipped with a microphone that captures the user's voice input. This voice data is sent to a voice recognition system (e.g., voice recognition API) and converted into text data. The converted text data is then sent to the server.

[1343] Examples:

[1344] When a user speaks into a microphone, "I want to go to a nice restaurant," the speech is converted into text.

[1345] 2. Dialogue Engine

[1346] server:

[1347] The server receives the text data sent from the speech recognition module and analyzes it using a natural language processing engine (e.g., a generative AI model). The analysis identifies the user's intent and generates appropriate suggestions as the next step.

[1348] Examples:

[1349] User: "I want to go to a nice restaurant."

[1350] Server: "We'd like to introduce you to some popular restaurants in the area. Would you like to choose Italian, Japanese, or Chinese?"

[1351] 3. Navigation Module

[1352] server:

[1353] Once the destination is confirmed, the server launches the navigation module (e.g., map API) to calculate the optimal route to the destination. This module obtains current traffic condition data from the real-time traffic information module and selects the optimal route. The calculation results are sent to the terminal, which issues instructions to the vehicle's autonomous driving system.

[1354] Examples:

[1355] After the user selects "Japanese food," the server calculates the optimal route to the Japanese restaurant and provides instructions to the autonomous driving system.

[1356] 4. Real-time traffic information module

[1357] server:

[1358] The server accesses an external traffic information service API (e.g., traffic information API) to obtain the latest traffic information. This information includes road congestion, accidents, construction information, etc. The obtained information is provided to the navigation module and used to calculate the optimal route.

[1359] Examples:

[1360] The server obtains traffic congestion information and calculates the optimal detour route.

[1361] 5. Destination Information Module

[1362] server:

[1363] Detailed information about destinations that the user has shown interest in is obtained by querying external databases or APIs (e.g., information provision APIs). The obtained information includes restaurant ratings, menus, opening hours, etc. This information is provided to the user through a dialogue engine.

[1364] Examples:

[1365] User: "Tell me more about this restaurant."

[1366] Server: "This restaurant has a 4.5 rating and offers pasta and pizza. It's open from 11:00 AM to 10:00 PM."

[1367] summary

[1368] The system supports a series of processes, from voice input to destination suggestions, route calculation, real-time traffic information, and detailed destination information, allowing users to intuitively and efficiently operate an autonomous vehicle and enjoy a comfortable travel experience.

[1369] Prompt Sentence Examples

[1370] "In a fully autonomous vehicle system, if a user says, 'I want to go to a nice restaurant,' explain how each module in the system processes the data and provides information to the user."

[1371] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1372] Step 1:

[1373] The user provides voice input.

[1374] Specific action: The user speaks into the microphone, "I want to go to a nice restaurant."

[1375] Input: User's voice command.

[1376] Output: Audio data.

[1377] Step 2:

[1378] The device captures the audio data.

[1379] Specific operation: The microphone installed on the device captures the user's voice and saves it as audio data.

[1380] Input: User's voice.

[1381] Output: The captured audio data.

[1382] Step 3:

[1383] The terminal transmits the voice data to the voice recognition system.

[1384] Specific operation: The device sends the captured voice data to the voice recognition API.

[1385] Input: Audio data.

[1386] Output: Data sent to the speech recognition API.

[1387] Step 4:

[1388] A speech recognition system converts the speech data into text data.

[1389] Specific operation: The speech recognition API analyzes the speech data and converts it into text data.

[1390] Input: Audio data.

[1391] Output: The converted text data.

[1392] Step 5:

[1393] The terminal transmits the text data to the server.

[1394] Specific operation: The terminal sends the converted text data to the server.

[1395] Input: Text data.

[1396] Output: The data sent to the server.

[1397] Step 6:

[1398] The server parses the text data.

[1399] Specific operation: The server sends the received text data to a natural language processing engine (e.g., a generative AI model) for analysis.

[1400] Input: Text data.

[1401] Output: Data identifying the user's intent.

[1402] Step 7:

[1403] The server suggests destinations based on the user's intentions.

[1404] Specific operation: The server generates suggestions and responds to the user in an interactive format.

[1405] Input: Parsed text data.

[1406] Output: Destination suggestion data.

[1407] Step 8:

[1408] The server launches the navigation module.

[1409] Specific operation: After the destination is determined, the server launches the navigation module and calculates the optimal route to the destination.

[1410] Input: Confirmed destination information.

[1411] Output: Launch of navigation module.

[1412] Step 9:

[1413] The server obtains real-time traffic information.

[1414] Specific operation: The server accesses the traffic information API and obtains the latest traffic information (traffic congestion, accidents, construction, etc.).

[1415] Input: A navigation launch request.

[1416] Output: The obtained traffic information.

[1417] Step 10:

[1418] The server calculates the optimal route.

[1419] Specific operation: The server calculates the optimal route based on the acquired real-time traffic information.

[1420] Input: Real-time traffic information.

[1421] Output: Calculated optimal route data.

[1422] Step 11:

[1423] The server sends the route information to the device.

[1424] Specific operation: The server sends the calculated route information to the terminal.

[1425] Input: Optimal route data.

[1426] Output: Route information sent to the device.

[1427] Step 12:

[1428] The terminal issues instructions to the autonomous driving system.

[1429] Specific operation: The terminal issues instructions to the autonomous driving system based on the received route information, and the vehicle moves along the route.

[1430] Input: Route information.

[1431] Output: Instructions to the autonomous driving system.

[1432] Step 13:

[1433] The server obtains the destination information.

[1434] Specific operation: The server queries external databases or APIs to obtain detailed information about the destination (ratings, menu, opening hours, etc.).

[1435] Input: Destination information.

[1436] Output: The obtained destination information.

[1437] Step 14:

[1438] The server provides the destination details to the user.

[1439] Specific operation: The server provides the acquired destination information to the user through the dialogue engine.

[1440] Input: Destination information.

[1441] Output: Detailed information provided to the user.

[1442] (Application example 1)

[1443] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1444] In modern self-driving vehicles, there are systems that allow users to specify their destination via voice and provide the optimal route based on that destination. However, there are problems when users do not specify a specific destination, and there is a lack of systems that provide detailed destination information. There is also a need for dialogue-based destination suggestions that allow users to intuitively select their destination. Furthermore, natural dialogue using prompt sentences generated using generative AI models has not yet been realized. In this situation, a comprehensive and highly functional system is needed to improve user convenience.

[1445] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1446] In this invention, the server includes means for accepting voice input, means for converting voice to text, means for analyzing the text to identify a user's intent, means for suggesting a destination based on the user's intent, means for calculating an optimal route when the destination is confirmed, means for acquiring real-time traffic information, means for selecting an optimal route, means for providing the selected route information to the vehicle's autonomous driving system, means for acquiring detailed information about the identified destination from an external database and providing it to the user, and means for suggesting destinations in an interactive format using a generative AI model and generating prompt sentences to help the user make an appropriate selection. This allows the user to intuitively and efficiently determine their destination through voice operation and travel along the optimal route while obtaining detailed information.

[1447] "Means for accepting voice input" refers to means for capturing voice uttered by a user and inputting the voice data into the system.

[1448] The "means for converting voice to text" is a means for converting received voice data into text data in a specified language.

[1449] "Means for analyzing text to identify user intent" refers to means for analyzing converted text data and recognizing and identifying the user's intent.

[1450] The "means for suggesting a destination based on the user's intention" is a means for suggesting destination candidates that match the specified user's intention.

[1451] The "means for calculating the optimum route when the destination is determined" is a means for calculating the optimum route to reach the determined destination.

[1452] "Means for obtaining real-time traffic information" refers to means for obtaining real-time traffic data such as current traffic conditions, congestion, and accident information.

[1453] The "means for selecting an optimal route" is a means for selecting the most efficient route in consideration of the acquired real-time traffic information.

[1454] "Means for providing selected route information to the vehicle's automated driving system" refers to means for issuing driving instructions to the vehicle's automated driving system based on the selected route.

[1455] The "means for obtaining detailed information about the identified destination from an external database and providing it to the user" refers to a means for obtaining detailed information about the destination from an external database and providing it to the user.

[1456] "A means of using a generative AI model to suggest destinations in an interactive format and generate prompts to help the user make an appropriate choice" refers to a means of utilizing generative AI to suggest destinations in a natural interactive format and generate prompts to guide the user in making an appropriate choice.

[1457] This invention is a system for fully automated driving vehicles that uses voice control to determine destinations and efficiently provide optimal routes. This system supports everything from voice input to destination suggestions, route calculations, real-time traffic information, and detailed destination information.

[1458] Voice Input and Recognition Module

[1459] The terminal is a device that allows users to input destinations and travel instructions by voice. The terminal is equipped with a microphone that captures voice input and sends the data to a speech recognition module (e.g., Google Cloud Speech-to-Text API), which converts the voice data into text.

[1460] Text analysis and dialogue engine

[1461] The server analyzes the text data sent from the speech recognition module and identifies the user's intention. A dialogue engine (e.g., Dialogflow) is used for the analysis, and suggests destinations in a dialogue format. For example, if the user says, "I want to go to a nice restaurant," the server responds as follows:

[1462] "We'll introduce you to popular restaurants in the area. You can choose from Italian, Japanese, or Chinese cuisine."

[1463] Destination and navigation

[1464] Once the destination is determined, the server launches the navigation module, which uses the Google Maps API to calculate the optimal route to the destination, retrieves real-time traffic information (e.g., Google Maps Traffic API), selects the optimal route, and provides that information to the vehicle's autonomous driving system.

[1465] Real-time traffic information

[1466] The server accesses an external traffic information service API to obtain the latest traffic information, including road congestion, accidents, and construction information, and uses this information to calculate the optimal route in the navigation module.

[1467] Providing destination information

[1468] The server queries an external database or API (e.g., Yelp API) for details about the destination selected by the user and provides the information obtained to the user. For example, if the user requests "Tell me the details of this restaurant," the server provides the following information:

[1469] "This restaurant has a 4.5 rating, offers pasta and pizza on the menu, and is open from 11am to 10pm."

[1470] Leveraging generative AI models

[1471] The server uses a generative AI model to interactively suggest destinations and generate prompts to help users make appropriate choices. This makes it easier for users to decide on a destination through natural dialogue. Here are some examples of prompts:

[1472] "I want to go to a nice restaurant."

[1473] "We'll introduce you to popular restaurants in the area. You can choose from Italian, Japanese, or Chinese cuisine."

[1474] "If you select Italian, we will retrieve and provide detailed information about your destination to you."

[1475] This will enable users to intuitively and efficiently operate autonomous vehicles, providing a comfortable travel experience.

[1476] As described above, the system of the present invention comprehensively supports everything from voice input to destination setting, route guidance, and the provision of detailed information, and by utilizing a generative AI model in the process, it provides users with high convenience and intuitive operability.

[1477] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1478] Step 1:

[1479] Accepting voice input

[1480] The user inputs destinations and travel instructions into the device by voice, which is captured through the device's microphone and saved as audio data.

[1481] Input: User's voice command

[1482] Output: Captured audio data

[1483] Step 2:

[1484] Converting audio data to text

[1485] The device sends the captured voice data to a voice recognition module (e.g., Google Cloud Speech-to-Text API) and converts the voice data into text data.

[1486] Input: Audio data

[1487] Output: Converted text data

[1488] Step 3:

[1489] Text data analysis

[1490] The server analyzes the converted text data to identify the user's intent. A dialogue engine (e.g., Dialogflow) is used for the analysis, and the user's intent regarding destination and route is identified.

[1491] Input: Text data

[1492] Output: Identified user intent data

[1493] Step 4:

[1494] Destination suggestions

[1495] The server suggests destinations based on the user's intentions, and uses a generative AI model to interactively suggest potential destinations and generate prompts to help the user make the appropriate choice.

[1496] Input: User intent data

[1497] Output: Possible destinations and prompts

[1498] Step 5:

[1499] Destination determination and route calculation

[1500] Once the user has decided on a destination, the server uses the Google Maps API to calculate the optimal route to the destination.

[1501] Input: Selected destination

[1502] Output: Optimal route information

[1503] Step 6:

[1504] Obtaining real-time traffic information

[1505] The server uses external traffic information service APIs, such as Google Maps Traffic API, to obtain the latest traffic information, including road congestion, accidents, and construction information.

[1506] Input: None (Access to external traffic information services)

[1507] Output: Real-time traffic information

[1508] Step 7:

[1509] Selection of the optimal route

[1510] The server selects the optimal route taking into account real-time traffic information.

[1511] Input: Initial route calculation, real-time traffic information

[1512] Output: Optimal route reflecting traffic information

[1513] Step 8:

[1514] Providing route information

[1515] The server provides the selected route information to the vehicle's autonomous driving system and controls the vehicle's driving.

[1516] Input: Optimal route information

[1517] Output: Driving instructions to the vehicle's autonomous driving system

[1518] Step 9:

[1519] Providing destination information

[1520] The server queries an external database (e.g., Yelp API) for details about the identified destination and provides the retrieved information to the user.

[1521] Input: Confirmed destination

[1522] Output: Destination details (ratings, menu, opening hours, etc.)

[1523] This allows users to intuitively and efficiently determine their destination through voice control, and travel along the optimal route while obtaining detailed information.

[1524] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1525] The present invention provides a system for determining a destination using voice control in a fully automated driving vehicle, recognizing the user's emotions, and efficiently providing an optimal route. Specific embodiments of this system are described below.

[1526] 1. Speech Recognition Module

[1527] Device:

[1528] The user inputs destinations and travel instructions by voice. The device is equipped with a microphone that captures the voice input and saves it as voice data. The saved voice data is sent to a voice recognition system and converted into text data. This text data is then sent to the server.

[1529] 2. Emotion Engine

[1530] server:

[1531] The server analyzes the text data and voice sent from the voice recognition module to identify the user's emotions. Using an emotion engine, it recognizes the user's emotional state (e.g., joy, anger, sadness, etc.) and records the results. This makes it possible to understand the emotional state of the user when giving instructions and provide appropriate responses and destination suggestions.

[1532] For example, the emotion engine works as follows:

[1533] User: "I want to go to a nice restaurant." (Audio contains a tone of delight)

[1534] Server: "Here are some popular restaurants in the area. Would you like to choose Italian, Japanese, or Chinese?" (The emotion engine recognizes that the user is happy and responds in a positive tone.)

[1535] 3. Dialogue Engine

[1536] server:

[1537] The server analyzes the text data received from the speech recognition module to identify the user's intent. Through this analysis, the server understands the user's intent (command, question, suggestion, etc.) and determines the next step. The server also takes into account the results of the emotion engine to generate optimal destination suggestions and responses.

[1538] 4. Navigation Module

[1539] server:

[1540] When the server receives the destination information, the navigation module is activated and calculates the optimal route to the destination. The navigation module obtains the latest traffic information from the real-time traffic information module and selects the optimal route based on that information.

[1541] The navigation module sends calculated route information to the terminal and provides instructions to the vehicle's autonomous driving system.

[1542] 5. Real-time traffic information module

[1543] server:

[1544] The server accesses an external traffic information service API to obtain the latest traffic information, including road congestion, accidents, construction information, etc. The obtained information is provided to the navigation module and used to calculate the optimal route.

[1545] 6. Destination Information Module

[1546] server:

[1547] The server queries external databases and APIs to obtain detailed information about destinations that the user has expressed interest in. The information obtained (e.g., restaurant ratings, menus, opening hours, etc.) is provided to the user through a dialogue engine.

[1548] For example, if the user says "Tell me more about this restaurant," the server provides the following information:

[1549] Server: "This restaurant has a 4.5 rating. They have pasta and pizza on the menu. They're open from 11:00 AM to 10:00 PM."

[1550] summary

[1551] The system covers a range of processes, from voice input to emotion recognition, destination suggestions, route calculation, real-time traffic information, and detailed destination information. This allows users to not only intuitively operate an autonomous vehicle, but also to receive more detailed emotional services, providing a comfortable travel experience.

[1552] The processing flow will be explained below.

[1553] Step 1:

[1554] User:

[1555] The user says, "I want to go to a nice restaurant."

[1556] Step 2:

[1557] Device:

[1558] The microphone on the device captures the user's voice and saves it as audio data.

[1559] Step 3:

[1560] Device:

[1561] The saved voice data is sent to a voice recognition service and converted into text data.

[1562] Step 4:

[1563] Device:

[1564] The converted text data is sent to the server.

[1565] Step 5:

[1566] server:

[1567] The server analyzes the received text data and determines the user's intent. In this case, it understands that the user is looking for a restaurant.

[1568] Step 6:

[1569] server:

[1570] The emotion engine is used to analyze the user's emotions from the received voice data, for example, recognizing that the user is happy from the tone and accent of the voice.

[1571] Step 7:

[1572] server:

[1573] Based on the analysis results of the emotion engine, the dialogue engine asks the user a question: "We'll introduce you to some popular restaurants in the area. Would you like to choose from Italian, Japanese, or Chinese cuisine?" The message is sent to the device.

[1574] Step 8:

[1575] User:

[1576] A user says, "I want Italian food."

[1577] Step 9:

[1578] Device:

[1579] The user's voice is captured again and sent to a speech recognition service for conversion into text data.

[1580] Step 10:

[1581] Device:

[1582] The converted text data is sent to the server.

[1583] Step 11:

[1584] server:

[1585] The server analyzes the text data and identifies the user's request. In this case, it understands that the user wants an Italian restaurant.

[1586] Step 12:

[1587] server:

[1588] The server retrieves information about recommended Italian restaurants from an external database and makes suggestions to the user, for example, sending a message to the terminal saying, "Nearby Italian restaurants are Restaurant A, Restaurant B, and Restaurant C."

[1589] Step 13:

[1590] User:

[1591] A user says, "I want to go to Restaurant A."

[1592] Step 14:

[1593] Device:

[1594] The user's voice is captured and sent to a speech recognition service to be converted into text data.

[1595] Step 15:

[1596] Device:

[1597] The converted text data is sent to the server.

[1598] Step 16:

[1599] server:

[1600] The server analyzes the text data and determines that Restaurant A is the destination.

[1601] Step 17:

[1602] server:

[1603] The server launches the navigation module and calculates the optimal route to the destination, Restaurant A.

[1604] Step 18:

[1605] server:

[1606] The navigation module accesses the real-time traffic information module to obtain the latest traffic information, including congestion, traffic accidents, and construction information.

[1607] Step 19:

[1608] server:

[1609] The navigation module takes into account traffic information obtained and selects the optimal route.

[1610] Step 20:

[1611] server:

[1612] The selected route information is sent to the terminal.

[1613] Step 21:

[1614] Device:

[1615] The route information received by the terminal is reflected in the vehicle's autonomous driving system, and the vehicle begins driving automatically toward the destination.

[1616] Step 22:

[1617] User:

[1618] A user says, "Tell me more about this restaurant."

[1619] Step 23:

[1620] Device:

[1621] The user's voice is captured and sent to a speech recognition service to be converted into text data.

[1622] Step 24:

[1623] Device:

[1624] The converted text data is sent to the server.

[1625] Step 25:

[1626] server:

[1627] The server parses the text data and understands the user's request.

[1628] Step 26:

[1629] server:

[1630] The server accesses an external information service to obtain detailed restaurant information (ratings, menu, opening hours, etc.).

[1631] Step 27:

[1632] server:

[1633] The acquired detailed information is then provided to the user through a dialogue engine, for example, "This restaurant has a rating of 4.5. The menu includes pasta and pizza. It is open from 11:00 AM to 10:00 PM."

[1634] Example 2

[1635] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1636] Conventional autonomous driving systems require users to manually input their destination and set up routes, resulting in unintuitive operation. Furthermore, they do not take into account the user's emotional state, resulting in issues with not being able to provide optimal service or responses. Furthermore, they are not sufficient in selecting optimal routes that take real-time traffic information into account, leaving a need for improving the quality of the travel experience.

[1637] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1638] In this invention, the server includes means for accepting voice input, means for converting the accepted voice into text, means for analyzing the converted text and voice data to identify the user's emotional state, means for identifying the user's intention based on the analyzed text data, means for suggesting a destination based on the identified user's intention and emotional state, means for calculating an optimal route using the destination information when the destination is confirmed, means for acquiring real-time traffic information from an external traffic information service, means for selecting an optimal route taking into account the acquired real-time traffic information, and means for providing the selected route information to the vehicle's autonomous driving system. This allows the user to specify a destination in an intuitive manner, provides appropriate services and responses taking into account the user's emotional state, and enables optimal route selection based on real-time traffic information.

[1639] "Means for accepting voice input" refers to a device or software that has the function of recognizing voice uttered by a user and inputting it into the system.

[1640] "Means for converting received speech into text" refers to a technology or system for analyzing speech data and converting it into text data.

[1641] "Means for analyzing converted text and voice data to identify a user's emotional state" refers to technology or systems that recognize and identify a user's emotions (e.g., joy, anger, sadness, etc.) by analyzing text and voice data.

[1642] "Means for identifying a user's intent based on analyzed text data" refers to a technology or system that analyzes text data to understand a user's instructions or requests and identify their intent.

[1643] "Means for suggesting destinations based on identified user intentions and emotional state" refers to a technology or system for suggesting appropriate destinations taking into account a user's intentions and emotional state.

[1644] "Means for calculating the optimal route using information once the destination is determined" refers to technology or a system that calculates the optimal route to a destination based on the destination specified or agreed to by the user.

[1645] "Means for obtaining real-time traffic information from external traffic information services" refers to a technology or system that accesses external traffic information services and obtains the latest traffic situation data.

[1646] "Means for selecting the optimal route taking into account acquired real-time traffic information" refers to a technology or system that selects the most efficient route based on real-time traffic conditions.

[1647] "Means for providing selected route information to the vehicle's automated driving system" refers to a technology or system that communicates the selected route information to the vehicle's automated driving system and issues instructions.

[1648] This invention is a system for fully autonomous vehicles that uses voice control to determine destinations, recognizes the user's emotions, and efficiently provides the optimal route. This system is mainly composed of the following modules.

[1649] 1. Voice Input Module

[1650] Device:

[1651] The user inputs destinations and travel instructions by voice. The device is equipped with a microphone that captures the user's voice and saves it as audio data. The saved audio data is then sent to a server. This process uses a common hardware microphone and voice recognition software called a "voice recognition API" (for example, Google Cloud Speech-to-Text API).

[1652] For example, when a user says, "I want to go to a nice restaurant," the microphone on the device captures the voice and generates voice data, which is then sent to the server.

[1653] 2. Speech Recognition and Text Conversion

[1654] server:

[1655] The server passes the received voice data to a voice recognition engine (for example, Google Cloud Speech-to-Text API) and converts the voice data into text data, which is then stored on the server.

[1656] 3. Emotion Recognition Engine

[1657] server:

[1658] An emotion engine (e.g., IBM Watson Tone Analyzer) analyzes the converted text data and the original voice data to identify the user's emotional state (e.g., joy, anger, sadness, etc.).

[1659] For example, if a user says, "I want to go to a nice restaurant," the voice contains a tone of joy. The server uses an emotion engine to recognize the emotional state and determine that the user is happy.

[1660] 4. Dialogue Engine

[1661] server:

[1662] The server analyzes the text data received from the speech recognition module. A dialogue engine (e.g., OpenAI GPT-3) is used to identify the user's intent. Through the analysis, the user's intent (e.g., command, question, suggestion, etc.) is understood.

[1663] 5. Destination suggestions

[1664] server:

[1665] Destination suggestions are made based on the user's intent and emotional state. The server combines the identified user intent and emotional state to generate optimal destination suggestions and responses.

[1666] For example, if a user says, "I want to go to a nice restaurant," the server will suggest, "We'll introduce you to some popular restaurants in the area. Would you like to choose from Italian, Japanese, or Chinese cuisine?"

[1667] 6. Route calculation

[1668] server:

[1669] Once the destination information is confirmed, the navigation module (e.g., Google Maps API) is used to calculate the optimal route to the destination. The latest traffic information is obtained from the real-time traffic information module, and the optimal route is selected based on that information.

[1670] As a concrete example, if you select "Italian restaurant," the server will calculate the optimal route through the navigation module.

[1671] 7. Real-time traffic information

[1672] server:

[1673] The server obtains the latest traffic information from an external traffic information service API and provides it to the navigation module, which then selects the optimal route based on the real-time traffic information.

[1674] 8. Providing route information

[1675] server:

[1676] The calculated route information is sent to the terminal and instructions are sent to the autonomous driving system, which then guides the vehicle to its destination.

[1677] 9. Providing destination information

[1678] server:

[1679] Detailed information about the destinations the user has expressed interest in is retrieved from external databases and APIs and provided to the user, including suggested restaurant details such as ratings, menus, and opening hours.

[1680] For example, if a user says, "Tell me more about this restaurant," the server might provide information like, "This restaurant has a rating of 4.5. The menu includes pasta and pizza. It's open from 11:00 a.m. to 10:00 p.m."

[1681] Prompt Sentence Examples

[1682] User: "I want to go to a nice restaurant."

[1683] This system covers a range of processes, from voice input to emotion recognition, destination suggestions, route calculation, real-time traffic information, and detailed destination information, thereby providing users with intuitive and emotionally responsive services and a comfortable travel experience.

[1684] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1685] Step 1: Getting voice input

[1686] The user inputs destinations and instructions into the terminal by voice.

[1687] The device captures the user's voice with a microphone and saves it as audio data.

[1688] Input: User's voice commands

[1689] Output: Audio data

[1690] How it works: When a user says, "I want to go to a nice restaurant," the microphone captures the voice, which is converted into digital audio data and stored on the device. This audio data is then sent to the server.

[1691] Step 2: Speech recognition and text conversion

[1692] The server passes the received voice data to a voice recognition engine (Google Cloud Speech-to-Text API) and converts the voice data into text data.

[1693] Input: Audio data

[1694] Output: Text data

[1695] Specific operation: The voice data is sent to the server and passed to the speech recognition engine. This engine analyzes the voice data and converts it into text data such as "I want to go to a delicious restaurant."

[1696] Step 3: Emotion Recognition

[1697] The server passes the text and voice data to an emotion engine (IBM Watson Tone Analyzer) to analyze the user's emotions.

[1698] Input: Text data, audio data

[1699] Output: User's emotional state (e.g., joy)

[1700] How it works: Text and voice data are passed to the emotion engine, which then identifies the user's emotional state. After analysis, it recognizes that the voice contains a tone of joy.

[1701] Step 4: User Intent Analysis

[1702] The server passes the text data to a dialogue engine (OpenAI GPT-3) for analysis to identify the user's intent.

[1703] Input: Text data

[1704] Output: User intent (e.g., search for restaurants)

[1705] How it works: The text data is passed to the dialogue engine, which analyzes the user's intent. As a result of the analysis, it determines that the user wants to search for restaurants.

[1706] Step 5: Generate destination suggestions

[1707] The server generates destination suggestions based on the user's intentions and emotional state.

[1708] Input: User intent, emotional state

[1709] Output: Text data of destination suggestions

[1710] Specific operation: Based on the user's intent of "I want to go to a delicious restaurant" and the emotional state of "delight," the server generates a destination suggestion: "We'll introduce you to some popular restaurants in the area. Would you like to choose from Italian, Japanese, or Chinese cuisine?"

[1711] Step 6: Route calculation

[1712] The server passes the destination information to the navigation module (Google Maps API) and calculates the optimal route.

[1713] Input: Destination information

[1714] Output: Optimal route information

[1715] Specific operation: If the user selects "Italian restaurant," the server sends the destination information to the navigation module, which calculates the optimal route.

[1716] Step 7: Obtaining real-time traffic information

[1717] The server accesses an external traffic information service API to obtain the latest traffic information.

[1718] Input: None

[1719] Output: Real-time traffic information

[1720] Specific operation: The server accesses an external traffic information service and obtains real-time data such as current road conditions, congestion information, and accident information.

[1721] Step 8: Route optimization

[1722] The optimal route is recalculated based on real-time traffic information obtained by the server.

[1723] Input: Real-time traffic information, destination information

[1724] Output: Optimized route information

[1725] Specific operation: The acquired real-time traffic information is reflected in the navigation module, and the optimal route is recalculated to avoid delays and congestion.

[1726] Step 9: Provide route information

[1727] The server sends optimized route information to the device and issues instructions to the autonomous driving system.

[1728] Input: Optimal route information

[1729] Output: None

[1730] Specific operation: The optimized route information is sent to the terminal, and the autonomous driving system begins driving according to the route.

[1731] Step 10: Provide destination information

[1732] In response to user requests, the server retrieves and provides detailed destination information from external databases and APIs.

[1733] Input: User request

[1734] Output: Destination details

[1735] What happens: If a user says, "Tell me more about this restaurant," the server retrieves information from an external database and provides, "This restaurant has a rating of 4.5. The menu includes pasta and pizza. It's open from 11:00 AM to 10:00 PM."

[1736] (Application example 2)

[1737] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1738] When users use voice commands to operate systems such as self-driving vehicles and food delivery services, they need systems that can not only recognize their voices but also make suggestions that take into account the user's emotions and mood. Conventional systems have difficulty analyzing user emotions and providing optimal routes and services, making it difficult to provide personalized services to individual users.

[1739] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for accepting voice input, means for converting the accepted voice into text, means for recognizing the user's emotions from the analyzed text, means for suggesting destinations and services based on the user's emotions and intentions, means for calculating the optimal route and service provision using the information when the destination and service are determined, means for acquiring real-time information, means for selecting the optimal route and service taking the real-time information into consideration, and means for providing information on the selected route and service. This enables personalized suggestions that take the user's emotions and intentions into consideration.

[1740] "Voice input" refers to voice information that a user speaks to a system and that is received through an input device such as a microphone.

[1741] "Converting to text" means converting the received voice information into a string of characters using voice recognition technology.

[1742] "User intent" refers to the wishes, requests, commands, etc. that the user is trying to convey to the system through voice input.

[1743] "User emotion" refers to the emotional state of the user identified by voice analysis, and includes different emotions such as joy, sadness, anger, fatigue, etc.

[1744] "Suggestions" refers to recommendations of destinations and services made by the system based on the user's intentions and emotions.

[1745] An "optimal route" refers to an efficient and safe route to a destination calculated taking into account real-time information.

[1746] "Real-time information" includes current data such as traffic conditions and service usage status, and optimal routes and services are provided based on this information.

[1747] "Selection" means choosing the most appropriate option from multiple options.

[1748] "Provision" refers to the act of the system providing information or services to the user.

[1749] "Autonomous driving system" refers to a system that includes hardware and software for autonomously controlling a vehicle and driving it automatically.

[1750] The present invention provides a personalized system that combines user voice input and emotion recognition to enhance user experience. This system can be applied as a food delivery assistant. Specific embodiments of the system are described below.

[1751] 1. Voice Input Module

[1752] Device:

[1753] Users input their voice using a device such as a smartphone. The device is equipped with a microphone that captures the user's voice. The voice input is then converted into text data using speech recognition software (e.g., Python's speech_recognition library).

[1754] Examples:

[1755] User: "I want to eat some delicious Chinese food today."

[1756] 2. Emotion Recognition Module

[1757] server:

[1758] The text data sent from the voice input module is analyzed by an emotion recognition engine on the server, using, for example, Python's TextBlob library, to identify the user's emotional state (positive, negative, neutral).

[1759] Examples:

[1760] Text data: "I want to eat some delicious Chinese food today."

[1761] Sentiment analysis result: Negative (slightly tired tone)

[1762] 3. Suggestion Engine

[1763] server:

[1764] Based on the user's emotions and intentions identified by the emotion recognition module, the server suggests potential destinations and services. A suggestion engine runs and accesses external restaurant databases and APIs to retrieve appropriate options.

[1765] Examples:

[1766] Server: "You're looking for a Chinese restaurant in the area. It sounds like you're in the mood for a little relaxation. Would you like a quiet restaurant with a festive atmosphere?"

[1767] 4. Optimization Engine

[1768] server:

[1769] If the user accepts the suggestion, the server will obtain real-time information based on the request and calculate the optimal route and service provision, accessing the external traffic information service API to obtain real-time data.

[1770] 5. Service Provision Module

[1771] Device:

[1772] The device provides the information received from the server to the user, who can then select the suggested options or check the detailed information.

[1773] Examples:

[1774] Selected restaurant information: "This restaurant has a 4.5 rating. The menu includes pasta and pizza. It is open from 11:00 AM to 10:00 PM."

[1775] Prompt Sentence Examples

[1776] An example of input a user might make to the system is shown below.

[1777] User: "I want to eat some delicious Chinese food today."

[1778] System (Response): "You're looking for a Chinese restaurant nearby. It sounds like you're looking for a place to relax. Would you like a quiet restaurant with a festive atmosphere?"

[1779] In this way, the system of the present invention provides a more personalized service by simultaneously considering the user's voice input and emotions.

[1780] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1781] Step 1:

[1782] The user speaks, which is captured by the device's microphone. The device converts this speech data into text using a generative AI model (e.g., a speech recognition library). The converted text data is sent to the server.

[1783] input:

[1784] Audio data

[1785] output:

[1786] Text data

[1787] Step 2:

[1788] The server sends the received text data to an emotion recognition engine, which analyzes the emotional state of the text data using a generative AI model (e.g., Python's TextBlob library). The result of the emotion analysis (positive, negative, neutral) is used in the next step.

[1789] input:

[1790] Text data

[1791] output:

[1792] Emotion analysis results

[1793] Step 3:

[1794] The server uses a generative AI model that identifies the user's intent using the sentiment analysis results and text data as input. Based on the user's intent, destination and service candidates are generated. The generated destination and service candidates are stored on the server.

[1795] input:

[1796] Emotion analysis results

[1797] Text data

[1798] output:

[1799] Destination and service suggestions

[1800] Step 4:

[1801] The server proposes optimal options from the generated list of destinations and services based on the user's intentions and the results of emotion analysis. The proposed options are sent to the terminal and presented to the user.

[1802] input:

[1803] Destination and service suggestions

[1804] output:

[1805] suggestion

[1806] Step 5:

[1807] If the user accepts the suggestion, the device sends the selection to the server, which then accesses an external API to obtain real-time information (e.g., traffic conditions and store congestion status). Based on the obtained real-time information, the server calculates the optimal route and service provision.

[1808] input:

[1809] Accepting the proposal

[1810] output:

[1811] Real-time information

[1812] Step 6:

[1813] The server sends the calculated optimal route and service information to the device, which then provides this information to the user, allowing the user to view detailed information.

[1814] input:

[1815] Real-time information

[1816] output:

[1817] Optimal route and service information

[1818] Step 7:

[1819] The user can check the optimal route and service information provided and take action as needed, allowing the user to take optimal action based on personalized information.

[1820] input:

[1821] Optimal route and service information

[1822] output:

[1823] User Behavior

[1824] This series of processing steps allows users to receive personalized suggestions and services through voice input and emotion recognition.

[1825] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1826] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1827] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1828] [Fourth embodiment]

[1829] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1830] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1831] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1832] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1833] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1834] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1835] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1836] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1837] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1838] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1839] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1840] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1841] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1842] The present invention is a system for determining a destination using voice control in a fully automated driving vehicle and efficiently providing an optimal route. Specific embodiments of this system will be described below.

[1843] 1. Speech Recognition Module

[1844] Device:

[1845] The user inputs destinations and travel instructions by voice. The device is equipped with a microphone that captures the voice input and saves it as voice data. The saved voice data is sent to a voice recognition system and converted into text data. This text data is then sent to the server.

[1846] 2. Dialogue Engine

[1847] server:

[1848] The server receives the text data sent from the speech recognition module and analyzes it. This analysis identifies the user's intent and determines the next step. For example, if the user says, "I want to go to a nice restaurant," the server understands that the user is searching for a restaurant and then suggests suitable restaurant options.

[1849] For example, the server might proceed with the following dialogue:

[1850] User: "I want to go to a nice restaurant."

[1851] Server: "We'd like to introduce you to some popular restaurants in the area. Would you like to choose Italian, Japanese, or Chinese?"

[1852] 3. Navigation Module

[1853] server:

[1854] Once the destination is confirmed, the server launches the navigation module to calculate the optimal route to the destination. The navigation module obtains current traffic condition data from the real-time traffic information module and selects the optimal route based on this data.

[1855] The navigation module sends calculated route information to the terminal and provides instructions to the vehicle's autonomous driving system.

[1856] 4. Real-time traffic information module

[1857] server:

[1858] The server accesses an external traffic information service API to obtain the latest traffic information, including road congestion, accidents, construction information, etc. The obtained information is provided to the navigation module and used to calculate the optimal route.

[1859] 5. Destination Information Module

[1860] server:

[1861] The server queries external databases and APIs to obtain detailed information about destinations that the user has expressed interest in. The information obtained (e.g., restaurant ratings, menus, opening hours, etc.) is provided to the user through a dialogue engine.

[1862] For example, if a user requests "Tell me more about this restaurant," the server provides the following information:

[1863] Server: "This restaurant has a 4.5 rating. They have pasta and pizza on the menu. They're open from 11:00 AM to 10:00 PM."

[1864] summary

[1865] The system supports a series of processes, from voice input to destination suggestions, route calculation, real-time traffic information, and detailed destination information, allowing users to intuitively and efficiently operate an autonomous vehicle and providing a comfortable travel experience.

[1866] The processing flow will be explained below.

[1867] Step 1:

[1868] User:

[1869] The user says, "I want to go to a nice restaurant."

[1870] Step 2:

[1871] Device:

[1872] The microphone on the device captures the user's voice and saves it as audio data.

[1873] Step 3:

[1874] Device:

[1875] The saved voice data is sent to a voice recognition service and converted into text data.

[1876] Step 4:

[1877] Device:

[1878] The converted text data is sent to the server.

[1879] Step 5:

[1880] server:

[1881] The server analyzes the received text data and determines the user's intent. In this case, it understands that the user is looking for a restaurant.

[1882] Step 6:

[1883] server:

[1884] The dialogue engine asks the user a question: "We'll introduce you to some popular restaurants in the area. Would you like to choose Italian, Japanese, or Chinese?" The message is sent to the device.

[1885] Step 7:

[1886] User:

[1887] A user says, "I want Italian food."

[1888] Step 8:

[1889] Device:

[1890] The user's voice is captured again and sent to a speech recognition service for conversion into text data.

[1891] Step 9:

[1892] Device:

[1893] The converted text data is sent to the server.

[1894] Step 10:

[1895] server:

[1896] The server analyzes the text data and identifies the user's request. In this case, it understands that the user wants an Italian restaurant.

[1897] Step 11:

[1898] server:

[1899] The server retrieves information about recommended Italian restaurants from an external database and makes suggestions to the user, for example, sending a message to the terminal saying, "Nearby Italian restaurants are Restaurant A, Restaurant B, and Restaurant C."

[1900] Step 12:

[1901] User:

[1902] A user says, "I want to go to Restaurant A."

[1903] Step 13:

[1904] Device:

[1905] The user's voice is captured and sent to a speech recognition service to be converted into text data.

[1906] Step 14:

[1907] Device:

[1908] The converted text data is sent to the server.

[1909] Step 15:

[1910] server:

[1911] The server analyzes the text data and determines that Restaurant A is the destination.

[1912] Step 16:

[1913] server:

[1914] The server passes the destination information to the navigation module and calculates the optimal route to Restaurant A.

[1915] Step 17:

[1916] server:

[1917] The navigation module obtains traffic information from the real-time traffic information module and selects the optimal route taking into account traffic congestion.

[1918] Step 18:

[1919] server:

[1920] The selected route information is sent to the terminal.

[1921] Step 19:

[1922] Device:

[1923] The route information received by the terminal is reflected in the vehicle's autonomous driving system, and the vehicle begins driving automatically toward the destination.

[1924] Step 20:

[1925] server:

[1926] When a user requests "more information about this restaurant," the server retrieves information such as restaurant ratings, menus, and opening hours from an external information service and provides it to the user.

[1927] Example 1

[1928] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1929] Current autonomous vehicle systems do not fully realize the process in which a user can intuitively and efficiently specify a destination using voice input, and the autonomous vehicle selects the optimal route and automatically guides the user to the destination. Therefore, a system is needed that integrates a series of processes: determining a destination using voice control, reflecting real-time traffic information, and providing detailed destination information.

[1930] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1931] In this invention, the server includes means for accepting voice input, means for converting the accepted voice into text, means for identifying the user's intention by analyzing the converted text, means for suggesting a destination based on the user's intention, means for calculating an optimal route using the destination information when the destination is confirmed, means for acquiring real-time traffic information, means for selecting an optimal route taking the real-time traffic information into consideration, means for providing the selected route information to the vehicle's autonomous driving system, and means for querying an external database to acquire detailed destination information and providing it to the user. This allows the user to specify a destination through intuitive voice operation and use a system that integrates optimal route calculation and detailed information provision.

[1932] "Means for accepting voice input" refers to devices or software that capture voice instructions given by a user and convert them into digital data.

[1933] "Means for converting received speech into text" refers to a system that uses speech recognition technology to convert speech data into text data.

[1934] "Means for analyzing the converted text to identify the user's intent" refers to a system that uses natural language processing technology to analyze text data and identify the user's requests and wishes.

[1935] "Means for suggesting destinations based on the user's intentions" refers to a system that predicts the user's desired destination or route and makes suggestions accordingly.

[1936] "A means for calculating the optimal route using information once the destination is confirmed" refers to a system for calculating the shortest distance and the most time-efficient route based on confirmed destination information.

[1937] "Means of obtaining real-time traffic information" refers to APIs and database access means for obtaining current road conditions and traffic data from external services.

[1938] "Means for selecting the optimal route taking into account real-time traffic information" refers to algorithms or systems that determine the optimal route to a destination based on real-time traffic information.

[1939] "Means for providing selected route information to the vehicle's automated driving system" refers to a system that transmits calculated route data to the automated vehicle so that the vehicle can follow that route.

[1940] "Means of querying an external database to obtain detailed destination information and providing it to the user" refers to a system that queries an external database or API for detailed destination information (e.g., opening hours, ratings, menu, etc.) and provides it to the user.

[1941] The present invention provides a fully automated driving system that allows a user to specify a destination using voice input and efficiently provides an optimal route. Specific embodiments of the present invention will be described in detail below.

[1942] 1. Speech Recognition Module

[1943] Device:

[1944] The user inputs destinations and travel instructions by voice. The device is equipped with a microphone that captures the user's voice input. This voice data is sent to a voice recognition system (e.g., voice recognition API) and converted into text data. The converted text data is then sent to the server.

[1945] Examples:

[1946] When a user speaks into a microphone, "I want to go to a nice restaurant," the speech is converted into text.

[1947] 2. Dialogue Engine

[1948] server:

[1949] The server receives the text data sent from the speech recognition module and analyzes it using a natural language processing engine (e.g., a generative AI model). The analysis identifies the user's intent and generates appropriate suggestions as the next step.

[1950] Examples:

[1951] User: "I want to go to a nice restaurant."

[1952] Server: "We'd like to introduce you to some popular restaurants in the area. Would you like to choose Italian, Japanese, or Chinese?"

[1953] 3. Navigation Module

[1954] server:

[1955] Once the destination is confirmed, the server launches the navigation module (e.g., map API) to calculate the optimal route to the destination. This module obtains current traffic condition data from the real-time traffic information module and selects the optimal route. The calculation results are sent to the terminal, which issues instructions to the vehicle's autonomous driving system.

[1956] Examples:

[1957] After the user selects "Japanese food," the server calculates the optimal route to the Japanese restaurant and provides instructions to the autonomous driving system.

[1958] 4. Real-time traffic information module

[1959] server:

[1960] The server accesses an external traffic information service API (e.g., traffic information API) to obtain the latest traffic information. This information includes road congestion, accidents, construction information, etc. The obtained information is provided to the navigation module and used to calculate the optimal route.

[1961] Examples:

[1962] The server obtains traffic congestion information and calculates the optimal detour route.

[1963] 5. Destination Information Module

[1964] server:

[1965] Detailed information about destinations that the user has shown interest in is obtained by querying external databases or APIs (e.g., information provision APIs). The obtained information includes restaurant ratings, menus, opening hours, etc. This information is provided to the user through a dialogue engine.

[1966] Examples:

[1967] User: "Tell me more about this restaurant."

[1968] Server: "This restaurant has a 4.5 rating and offers pasta and pizza. It's open from 11:00 AM to 10:00 PM."

[1969] summary

[1970] The system supports a series of processes, from voice input to destination suggestions, route calculation, real-time traffic information, and detailed destination information, allowing users to intuitively and efficiently operate an autonomous vehicle and enjoy a comfortable travel experience.

[1971] Prompt Sentence Examples

[1972] "In a fully autonomous vehicle system, if a user says, 'I want to go to a nice restaurant,' explain how each module in the system processes the data and provides information to the user."

[1973] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1974] Step 1:

[1975] The user provides voice input.

[1976] Specific action: The user speaks into the microphone, "I want to go to a nice restaurant."

[1977] Input: User's voice command.

[1978] Output: Audio data.

[1979] Step 2:

[1980] The device captures the audio data.

[1981] Specific operation: The microphone installed on the device captures the user's voice and saves it as audio data.

[1982] Input: User's voice.

[1983] Output: The captured audio data.

[1984] Step 3:

[1985] The terminal transmits the voice data to the voice recognition system.

[1986] Specific operation: The device sends the captured voice data to the voice recognition API.

[1987] Input: Audio data.

[1988] Output: Data sent to the speech recognition API.

[1989] Step 4:

[1990] A speech recognition system converts the speech data into text data.

[1991] Specific operation: The speech recognition API analyzes the speech data and converts it into text data.

[1992] Input: Audio data.

[1993] Output: The converted text data.

[1994] Step 5:

[1995] The terminal transmits the text data to the server.

[1996] Specific operation: The terminal sends the converted text data to the server.

[1997] Input: Text data.

[1998] Output: The data sent to the server.

[1999] Step 6:

[2000] The server parses the text data.

[2001] Specific operation: The server sends the received text data to a natural language processing engine (e.g., a generative AI model) for analysis.

[2002] Input: Text data.

[2003] Output: Data identifying the user's intent.

[2004] Step 7:

[2005] The server suggests destinations based on the user's intentions.

[2006] Specific operation: The server generates suggestions and responds to the user in an interactive format.

[2007] Input: Parsed text data.

[2008] Output: Destination suggestion data.

[2009] Step 8:

[2010] The server launches the navigation module.

[2011] Specific operation: After the destination is determined, the server launches the navigation module and calculates the optimal route to the destination.

[2012] Input: Confirmed destination information.

[2013] Output: Launch of navigation module.

[2014] Step 9:

[2015] The server obtains real-time traffic information.

[2016] Specific operation: The server accesses the traffic information API and obtains the latest traffic information (traffic congestion, accidents, construction, etc.).

[2017] Input: A navigation launch request.

[2018] Output: The obtained traffic information.

[2019] Step 10:

[2020] The server calculates the optimal route.

[2021] Specific operation: The server calculates the optimal route based on the acquired real-time traffic information.

[2022] Input: Real-time traffic information.

[2023] Output: Calculated optimal route data.

[2024] Step 11:

[2025] The server sends the route information to the device.

[2026] Specific operation: The server sends the calculated route information to the terminal.

[2027] Input: Optimal route data.

[2028] Output: Route information sent to the device.

[2029] Step 12:

[2030] The terminal issues instructions to the autonomous driving system.

[2031] Specific operation: The terminal issues instructions to the autonomous driving system based on the received route information, and the vehicle moves along the route.

[2032] Input: Route information.

[2033] Output: Instructions to the autonomous driving system.

[2034] Step 13:

[2035] The server obtains the destination information.

[2036] Specific operation: The server queries external databases or APIs to obtain detailed information about the destination (ratings, menu, opening hours, etc.).

[2037] Input: Destination information.

[2038] Output: The obtained destination information.

[2039] Step 14:

[2040] The server provides the destination details to the user.

[2041] Specific operation: The server provides the acquired destination information to the user through the dialogue engine.

[2042] Input: Destination information.

[2043] Output: Detailed information provided to the user.

[2044] (Application example 1)

[2045] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2046] In modern self-driving vehicles, there are systems that allow users to specify their destination via voice and provide the optimal route based on that destination. However, there are problems when users do not specify a specific destination, and there is a lack of systems that provide detailed destination information. There is also a need for dialogue-based destination suggestions that allow users to intuitively select their destination. Furthermore, natural dialogue using prompt sentences generated using generative AI models has not yet been realized. In this situation, a comprehensive and highly functional system is needed to improve user convenience.

[2047] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2048] In this invention, the server includes means for accepting voice input, means for converting voice to text, means for analyzing the text to identify a user's intent, means for suggesting a destination based on the user's intent, means for calculating an optimal route when the destination is confirmed, means for acquiring real-time traffic information, means for selecting an optimal route, means for providing the selected route information to the vehicle's autonomous driving system, means for acquiring detailed information about the identified destination from an external database and providing it to the user, and means for suggesting destinations in an interactive format using a generative AI model and generating prompt sentences to help the user make an appropriate selection. This allows the user to intuitively and efficiently determine their destination through voice operation and travel along the optimal route while obtaining detailed information.

[2049] "Means for accepting voice input" refers to means for capturing voice uttered by a user and inputting the voice data into the system.

[2050] The "means for converting voice to text" is a means for converting received voice data into text data in a specified language.

[2051] "Means for analyzing text to identify user intent" refers to means for analyzing converted text data and recognizing and identifying the user's intent.

[2052] The "means for suggesting a destination based on the user's intention" is a means for suggesting destination candidates that match the specified user's intention.

[2053] The "means for calculating the optimum route when the destination is determined" is a means for calculating the optimum route to reach the determined destination.

[2054] "Means for obtaining real-time traffic information" refers to means for obtaining real-time traffic data such as current traffic conditions, congestion, and accident information.

[2055] The "means for selecting an optimal route" is a means for selecting the most efficient route in consideration of the acquired real-time traffic information.

[2056] "Means for providing selected route information to the vehicle's automated driving system" refers to means for issuing driving instructions to the vehicle's automated driving system based on the selected route.

[2057] The "means for obtaining detailed information about the identified destination from an external database and providing it to the user" refers to a means for obtaining detailed information about the destination from an external database and providing it to the user.

[2058] "A means of using a generative AI model to suggest destinations in an interactive format and generate prompts to help the user make an appropriate choice" refers to a means of utilizing generative AI to suggest destinations in a natural interactive format and generate prompts to guide the user in making an appropriate choice.

[2059] This invention is a system for fully automated driving vehicles that uses voice control to determine destinations and efficiently provide optimal routes. This system supports everything from voice input to destination suggestions, route calculations, real-time traffic information, and detailed destination information.

[2060] Voice Input and Recognition Module

[2061] The terminal is a device that allows users to input destinations and travel instructions by voice. The terminal is equipped with a microphone that captures voice input and sends the data to a speech recognition module (e.g., Google Cloud Speech-to-Text API), which converts the voice data into text.

[2062] Text analysis and dialogue engine

[2063] The server analyzes the text data sent from the speech recognition module and identifies the user's intention. A dialogue engine (e.g., Dialogflow) is used for the analysis, and suggests destinations in a dialogue format. For example, if the user says, "I want to go to a nice restaurant," the server responds as follows:

[2064] "We'll introduce you to popular restaurants in the area. You can choose from Italian, Japanese, or Chinese cuisine."

[2065] Destination and navigation

[2066] Once the destination is determined, the server launches the navigation module, which uses the Google Maps API to calculate the optimal route to the destination, retrieves real-time traffic information (e.g., Google Maps Traffic API), selects the optimal route, and provides that information to the vehicle's autonomous driving system.

[2067] Real-time traffic information

[2068] The server accesses an external traffic information service API to obtain the latest traffic information, including road congestion, accidents, and construction information, and uses this information to calculate the optimal route in the navigation module.

[2069] Providing destination information

[2070] The server queries an external database or API (e.g., Yelp API) for details about the destination selected by the user and provides the information obtained to the user. For example, if the user requests "Tell me the details of this restaurant," the server provides the following information:

[2071] "This restaurant has a 4.5 rating, offers pasta and pizza on the menu, and is open from 11am to 10pm."

[2072] Leveraging generative AI models

[2073] The server uses a generative AI model to interactively suggest destinations and generate prompts to help users make appropriate choices. This makes it easier for users to decide on a destination through natural dialogue. Here are some examples of prompts:

[2074] "I want to go to a nice restaurant."

[2075] "We'll introduce you to popular restaurants in the area. You can choose from Italian, Japanese, or Chinese cuisine."

[2076] "If you select Italian, we will retrieve and provide detailed information about your destination to you."

[2077] This will enable users to intuitively and efficiently operate autonomous vehicles, providing a comfortable travel experience.

[2078] As described above, the system of the present invention comprehensively supports everything from voice input to destination setting, route guidance, and the provision of detailed information, and by utilizing a generative AI model in the process, it provides users with high convenience and intuitive operability.

[2079] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2080] Step 1:

[2081] Accepting voice input

[2082] The user inputs destinations and travel instructions into the device by voice, which is captured through the device's microphone and saved as audio data.

[2083] Input: User's voice command

[2084] Output: Captured audio data

[2085] Step 2:

[2086] Converting audio data to text

[2087] The device sends the captured voice data to a voice recognition module (e.g., Google Cloud Speech-to-Text API) and converts the voice data into text data.

[2088] Input: Audio data

[2089] Output: Converted text data

[2090] Step 3:

[2091] Text data analysis

[2092] The server analyzes the converted text data to identify the user's intent. A dialogue engine (e.g., Dialogflow) is used for the analysis, and the user's intent regarding destination and route is identified.

[2093] Input: Text data

[2094] Output: Identified user intent data

[2095] Step 4:

[2096] Destination suggestions

[2097] The server suggests destinations based on the user's intentions, and uses a generative AI model to interactively suggest potential destinations and generate prompts to help the user make the appropriate choice.

[2098] Input: User intent data

[2099] Output: Possible destinations and prompts

[2100] Step 5:

[2101] Destination determination and route calculation

[2102] Once the user has decided on a destination, the server uses the Google Maps API to calculate the optimal route to the destination.

[2103] Input: Selected destination

[2104] Output: Optimal route information

[2105] Step 6:

[2106] Obtaining real-time traffic information

[2107] The server uses external traffic information service APIs, such as Google Maps Traffic API, to obtain the latest traffic information, including road congestion, accidents, and construction information.

[2108] Input: None (Access to external traffic information services)

[2109] Output: Real-time traffic information

[2110] Step 7:

[2111] Selection of the optimal route

[2112] The server selects the optimal route taking into account real-time traffic information.

[2113] Input: Initial route calculation, real-time traffic information

[2114] Output: Optimal route reflecting traffic information

[2115] Step 8:

[2116] Providing route information

[2117] The server provides the selected route information to the vehicle's autonomous driving system and controls the vehicle's driving.

[2118] Input: Optimal route information

[2119] Output: Driving instructions to the vehicle's autonomous driving system

[2120] Step 9:

[2121] Providing destination information

[2122] The server queries an external database (e.g., Yelp API) for details about the identified destination and provides the retrieved information to the user.

[2123] Input: Confirmed destination

[2124] Output: Destination details (ratings, menu, opening hours, etc.)

[2125] This allows users to intuitively and efficiently determine their destination through voice control, and travel along the optimal route while obtaining detailed information.

[2126] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2127] The present invention provides a system for determining a destination using voice control in a fully automated driving vehicle, recognizing the user's emotions, and efficiently providing an optimal route. Specific embodiments of this system are described below.

[2128] 1. Speech Recognition Module

[2129] Device:

[2130] The user inputs destinations and travel instructions by voice. The device is equipped with a microphone that captures the voice input and saves it as voice data. The saved voice data is sent to a voice recognition system and converted into text data. This text data is then sent to the server.

[2131] 2. Emotion Engine

[2132] server:

[2133] The server analyzes the text data and voice sent from the voice recognition module to identify the user's emotions. Using an emotion engine, it recognizes the user's emotional state (e.g., joy, anger, sadness, etc.) and records the results. This makes it possible to understand the emotional state of the user when giving instructions and provide appropriate responses and destination suggestions.

[2134] For example, the emotion engine works as follows:

[2135] User: "I want to go to a nice restaurant." (Audio contains a tone of delight)

[2136] Server: "Here are some popular restaurants in the area. Would you like to choose Italian, Japanese, or Chinese?" (The emotion engine recognizes that the user is happy and responds in a positive tone.)

[2137] 3. Dialogue Engine

[2138] server:

[2139] The server analyzes the text data received from the speech recognition module to identify the user's intent. Through this analysis, the server understands the user's intent (command, question, suggestion, etc.) and determines the next step. The server also takes into account the results of the emotion engine to generate optimal destination suggestions and responses.

[2140] 4. Navigation Module

[2141] server:

[2142] When the server receives the destination information, the navigation module is activated and calculates the optimal route to the destination. The navigation module obtains the latest traffic information from the real-time traffic information module and selects the optimal route based on that information.

[2143] The navigation module sends calculated route information to the terminal and provides instructions to the vehicle's autonomous driving system.

[2144] 5. Real-time traffic information module

[2145] server:

[2146] The server accesses an external traffic information service API to obtain the latest traffic information, including road congestion, accidents, construction information, etc. The obtained information is provided to the navigation module and used to calculate the optimal route.

[2147] 6. Destination Information Module

[2148] server:

[2149] The server queries external databases and APIs to obtain detailed information about destinations that the user has expressed interest in. The information obtained (e.g., restaurant ratings, menus, opening hours, etc.) is provided to the user through a dialogue engine.

[2150] For example, if the user says "Tell me more about this restaurant," the server provides the following information:

[2151] Server: "This restaurant has a 4.5 rating. They have pasta and pizza on the menu. They're open from 11:00 AM to 10:00 PM."

[2152] summary

[2153] The system covers a range of processes, from voice input to emotion recognition, destination suggestions, route calculation, real-time traffic information, and detailed destination information. This allows users to not only intuitively operate an autonomous vehicle, but also to receive more detailed emotional services, providing a comfortable travel experience.

[2154] The processing flow will be explained below.

[2155] Step 1:

[2156] User:

[2157] The user says, "I want to go to a nice restaurant."

[2158] Step 2:

[2159] Device:

[2160] The microphone on the device captures the user's voice and saves it as audio data.

[2161] Step 3:

[2162] Device:

[2163] The saved voice data is sent to a voice recognition service and converted into text data.

[2164] Step 4:

[2165] Device:

[2166] The converted text data is sent to the server.

[2167] Step 5:

[2168] server:

[2169] The server analyzes the received text data and determines the user's intent. In this case, it understands that the user is looking for a restaurant.

[2170] Step 6:

[2171] server:

[2172] The emotion engine is used to analyze the user's emotions from the received voice data, for example, recognizing that the user is happy from the tone and accent of the voice.

[2173] Step 7:

[2174] server:

[2175] Based on the analysis results of the emotion engine, the dialogue engine asks the user a question: "We'll introduce you to some popular restaurants in the area. Would you like to choose from Italian, Japanese, or Chinese cuisine?" The message is sent to the device.

[2176] Step 8:

[2177] User:

[2178] A user says, "I want Italian food."

[2179] Step 9:

[2180] Device:

[2181] The user's voice is captured again and sent to a speech recognition service for conversion into text data.

[2182] Step 10:

[2183] Device:

[2184] The converted text data is sent to the server.

[2185] Step 11:

[2186] server:

[2187] The server analyzes the text data and identifies the user's request. In this case, it understands that the user wants an Italian restaurant.

[2188] Step 12:

[2189] server:

[2190] The server retrieves information about recommended Italian restaurants from an external database and makes suggestions to the user, for example, sending a message to the terminal saying, "Nearby Italian restaurants are Restaurant A, Restaurant B, and Restaurant C."

[2191] Step 13:

[2192] User:

[2193] A user says, "I want to go to Restaurant A."

[2194] Step 14:

[2195] Device:

[2196] The user's voice is captured and sent to a speech recognition service to be converted into text data.

[2197] Step 15:

[2198] Device:

[2199] The converted text data is sent to the server.

[2200] Step 16:

[2201] server:

[2202] The server analyzes the text data and determines that Restaurant A is the destination.

[2203] Step 17:

[2204] server:

[2205] The server launches the navigation module and calculates the optimal route to the destination, Restaurant A.

[2206] Step 18:

[2207] server:

[2208] The navigation module accesses the real-time traffic information module to obtain the latest traffic information, including congestion, traffic accidents, and construction information.

[2209] Step 19:

[2210] server:

[2211] The navigation module takes into account traffic information obtained and selects the optimal route.

[2212] Step 20:

[2213] server:

[2214] The selected route information is sent to the terminal.

[2215] Step 21:

[2216] Device:

[2217] The route information received by the terminal is reflected in the vehicle's autonomous driving system, and the vehicle begins driving automatically toward the destination.

[2218] Step 22:

[2219] User:

[2220] A user says, "Tell me more about this restaurant."

[2221] Step 23:

[2222] Device:

[2223] The user's voice is captured and sent to a speech recognition service to be converted into text data.

[2224] Step 24:

[2225] Device:

[2226] The converted text data is sent to the server.

[2227] Step 25:

[2228] server:

[2229] The server parses the text data and understands the user's request.

[2230] Step 26:

[2231] server:

[2232] The server accesses an external information service to obtain detailed restaurant information (ratings, menu, opening hours, etc.).

[2233] Step 27:

[2234] server:

[2235] The acquired detailed information is then provided to the user through a dialogue engine, for example, "This restaurant has a rating of 4.5. The menu includes pasta and pizza. It is open from 11:00 AM to 10:00 PM."

[2236] Example 2

[2237] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2238] Conventional autonomous driving systems require users to manually input their destination and set up routes, resulting in unintuitive operation. Furthermore, they do not take into account the user's emotional state, resulting in issues with not being able to provide optimal service or responses. Furthermore, they are not sufficient in selecting optimal routes that take real-time traffic information into account, leaving a need for improving the quality of the travel experience.

[2239] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2240] In this invention, the server includes means for accepting voice input, means for converting the accepted voice into text, means for analyzing the converted text and voice data to identify the user's emotional state, means for identifying the user's intention based on the analyzed text data, means for suggesting a destination based on the identified user's intention and emotional state, means for calculating an optimal route using the destination information when the destination is confirmed, means for acquiring real-time traffic information from an external traffic information service, means for selecting an optimal route taking into account the acquired real-time traffic information, and means for providing the selected route information to the vehicle's autonomous driving system. This allows the user to specify a destination in an intuitive manner, provides appropriate services and responses taking into account the user's emotional state, and enables optimal route selection based on real-time traffic information.

[2241] "Means for accepting voice input" refers to a device or software that has the function of recognizing voice uttered by a user and inputting it into the system.

[2242] "Means for converting received speech into text" refers to a technology or system for analyzing speech data and converting it into text data.

[2243] "Means for analyzing converted text and voice data to identify a user's emotional state" refers to technology or systems that recognize and identify a user's emotions (e.g., joy, anger, sadness, etc.) by analyzing text and voice data.

[2244] "Means for identifying a user's intent based on analyzed text data" refers to a technology or system that analyzes text data to understand a user's instructions or requests and identify their intent.

[2245] "Means for suggesting destinations based on identified user intentions and emotional state" refers to a technology or system for suggesting appropriate destinations taking into account a user's intentions and emotional state.

[2246] "Means for calculating the optimal route using information once the destination is determined" refers to technology or a system that calculates the optimal route to a destination based on the destination specified or agreed to by the user.

[2247] "Means for obtaining real-time traffic information from external traffic information services" refers to a technology or system that accesses external traffic information services and obtains the latest traffic situation data.

[2248] "Means for selecting the optimal route taking into account acquired real-time traffic information" refers to a technology or system that selects the most efficient route based on real-time traffic conditions.

[2249] "Means for providing selected route information to the vehicle's automated driving system" refers to a technology or system that communicates the selected route information to the vehicle's automated driving system and issues instructions.

[2250] This invention is a system for fully autonomous vehicles that uses voice control to determine destinations, recognizes the user's emotions, and efficiently provides the optimal route. This system is mainly composed of the following modules.

[2251] 1. Voice Input Module

[2252] Device:

[2253] The user inputs destinations and travel instructions by voice. The device is equipped with a microphone that captures the user's voice and saves it as audio data. The saved audio data is then sent to a server. This process uses a common hardware microphone and voice recognition software called a "voice recognition API" (for example, Google Cloud Speech-to-Text API).

[2254] For example, when a user says, "I want to go to a nice restaurant," the microphone on the device captures the voice and generates voice data, which is then sent to the server.

[2255] 2. Speech Recognition and Text Conversion

[2256] server:

[2257] The server passes the received voice data to a voice recognition engine (for example, Google Cloud Speech-to-Text API) and converts the voice data into text data, which is then stored on the server.

[2258] 3. Emotion Recognition Engine

[2259] server:

[2260] An emotion engine (e.g., IBM Watson Tone Analyzer) analyzes the converted text data and the original voice data to identify the user's emotional state (e.g., joy, anger, sadness, etc.).

[2261] For example, if a user says, "I want to go to a nice restaurant," the voice contains a tone of joy. The server uses an emotion engine to recognize the emotional state and determine that the user is happy.

[2262] 4. Dialogue Engine

[2263] server:

[2264] The server analyzes the text data received from the speech recognition module. A dialogue engine (e.g., OpenAI GPT-3) is used to identify the user's intent. Through the analysis, the user's intent (e.g., command, question, suggestion, etc.) is understood.

[2265] 5. Destination suggestions

[2266] server:

[2267] Destination suggestions are made based on the user's intent and emotional state. The server combines the identified user intent and emotional state to generate optimal destination suggestions and responses.

[2268] For example, if a user says, "I want to go to a nice restaurant," the server will suggest, "We'll introduce you to some popular restaurants in the area. Would you like to choose from Italian, Japanese, or Chinese cuisine?"

[2269] 6. Route calculation

[2270] server:

[2271] Once the destination information is confirmed, the navigation module (e.g., Google Maps API) is used to calculate the optimal route to the destination. The latest traffic information is obtained from the real-time traffic information module, and the optimal route is selected based on that information.

[2272] As a concrete example, if you select "Italian restaurant," the server will calculate the optimal route through the navigation module.

[2273] 7. Real-time traffic information

[2274] server:

[2275] The server obtains the latest traffic information from an external traffic information service API and provides it to the navigation module, which then selects the optimal route based on the real-time traffic information.

[2276] 8. Providing route information

[2277] server:

[2278] The calculated route information is sent to the terminal and instructions are sent to the autonomous driving system, which then guides the vehicle to its destination.

[2279] 9. Providing destination information

[2280] server:

[2281] Detailed information about the destinations the user has expressed interest in is retrieved from external databases and APIs and provided to the user, including suggested restaurant details such as ratings, menus, and opening hours.

[2282] For example, if a user says, "Tell me more about this restaurant," the server might provide information like, "This restaurant has a rating of 4.5. The menu includes pasta and pizza. It's open from 11:00 a.m. to 10:00 p.m."

[2283] Prompt Sentence Examples

[2284] User: "I want to go to a nice restaurant."

[2285] This system covers a range of processes, from voice input to emotion recognition, destination suggestions, route calculation, real-time traffic information, and detailed destination information, thereby providing users with intuitive and emotionally responsive services and a comfortable travel experience.

[2286] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2287] Step 1: Getting voice input

[2288] The user inputs destinations and instructions into the terminal by voice.

[2289] The device captures the user's voice with a microphone and saves it as audio data.

[2290] Input: User's voice commands

[2291] Output: Audio data

[2292] How it works: When a user says, "I want to go to a nice restaurant," the microphone captures the voice, which is converted into digital audio data and stored on the device. This audio data is then sent to the server.

[2293] Step 2: Speech recognition and text conversion

[2294] The server passes the received voice data to a voice recognition engine (Google Cloud Speech-to-Text API) and converts the voice data into text data.

[2295] Input: Audio data

[2296] Output: Text data

[2297] Specific operation: The voice data is sent to the server and passed to the speech recognition engine. This engine analyzes the voice data and converts it into text data such as "I want to go to a delicious restaurant."

[2298] Step 3: Emotion Recognition

[2299] The server passes the text and voice data to an emotion engine (IBM Watson Tone Analyzer) to analyze the user's emotions.

[2300] Input: Text data, audio data

[2301] Output: User's emotional state (e.g., joy)

[2302] How it works: Text and voice data are passed to the emotion engine, which then identifies the user's emotional state. After analysis, it recognizes that the voice contains a tone of joy.

[2303] Step 4: User Intent Analysis

[2304] The server passes the text data to a dialogue engine (OpenAI GPT-3) for analysis to identify the user's intent.

[2305] Input: Text data

[2306] Output: User intent (e.g., search for restaurants)

[2307] How it works: The text data is passed to the dialogue engine, which analyzes the user's intent. As a result of the analysis, it determines that the user wants to search for restaurants.

[2308] Step 5: Generate destination suggestions

[2309] The server generates destination suggestions based on the user's intentions and emotional state.

[2310] Input: User intent, emotional state

[2311] Output: Text data of destination suggestions

[2312] Specific operation: Based on the user's intent of "I want to go to a delicious restaurant" and the emotional state of "delight," the server generates a destination suggestion: "We'll introduce you to some popular restaurants in the area. Would you like to choose from Italian, Japanese, or Chinese cuisine?"

[2313] Step 6: Route calculation

[2314] The server passes the destination information to the navigation module (Google Maps API) and calculates the optimal route.

[2315] Input: Destination information

[2316] Output: Optimal route information

[2317] Specific operation: If the user selects "Italian restaurant," the server sends the destination information to the navigation module, which calculates the optimal route.

[2318] Step 7: Obtaining real-time traffic information

[2319] The server accesses an external traffic information service API to obtain the latest traffic information.

[2320] Input: None

[2321] Output: Real-time traffic information

[2322] Specific operation: The server accesses an external traffic information service and obtains real-time data such as current road conditions, congestion information, and accident information.

[2323] Step 8: Route optimization

[2324] The optimal route is recalculated based on real-time traffic information obtained by the server.

[2325] Input: Real-time traffic information, destination information

[2326] Output: Optimized route information

[2327] Specific operation: The acquired real-time traffic information is reflected in the navigation module, and the optimal route is recalculated to avoid delays and congestion.

[2328] Step 9: Provide route information

[2329] The server sends optimized route information to the device and issues instructions to the autonomous driving system.

[2330] Input: Optimal route information

[2331] Output: None

[2332] Specific operation: The optimized route information is sent to the terminal, and the autonomous driving system begins driving according to the route.

[2333] Step 10: Provide destination information

[2334] In response to user requests, the server retrieves and provides detailed destination information from external databases and APIs.

[2335] Input: User request

[2336] Output: Destination details

[2337] What happens: If a user says, "Tell me more about this restaurant," the server retrieves information from an external database and provides, "This restaurant has a rating of 4.5. The menu includes pasta and pizza. It's open from 11:00 AM to 10:00 PM."

[2338] (Application example 2)

[2339] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2340] When users use voice commands to operate systems such as self-driving vehicles and food delivery services, they need systems that can not only recognize their voices but also make suggestions that take into account the user's emotions and mood. Conventional systems have difficulty analyzing user emotions and providing optimal routes and services, making it difficult to provide personalized services to individual users.

[2341] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for accepting voice input, means for converting the accepted voice into text, means for recognizing the user's emotions from the analyzed text, means for suggesting destinations and services based on the user's emotions and intentions, means for calculating the optimal route and service provision using the information when the destination and service are determined, means for acquiring real-time information, means for selecting the optimal route and service taking the real-time information into consideration, and means for providing information on the selected route and service. This enables personalized suggestions that take the user's emotions and intentions into consideration.

[2342] "Voice input" refers to voice information that a user speaks to a system and that is received through an input device such as a microphone.

[2343] "Converting to text" means converting the received voice information into a string of characters using voice recognition technology.

[2344] "User intent" refers to the wishes, requests, commands, etc. that the user is trying to convey to the system through voice input.

[2345] "User emotion" refers to the emotional state of the user identified by voice analysis, and includes different emotions such as joy, sadness, anger, fatigue, etc.

[2346] "Suggestions" refers to recommendations of destinations and services made by the system based on the user's intentions and emotions.

[2347] An "optimal route" refers to an efficient and safe route to a destination calculated taking into account real-time information.

[2348] "Real-time information" includes current data such as traffic conditions and service usage status, and optimal routes and services are provided based on this information.

[2349] "Selection" means choosing the most appropriate option from multiple options.

[2350] "Provision" refers to the act of the system providing information or services to the user.

[2351] "Autonomous driving system" refers to a system that includes hardware and software for autonomously controlling a vehicle and driving it automatically.

[2352] The present invention provides a personalized system that combines user voice input and emotion recognition to enhance user experience. This system can be applied as a food delivery assistant. Specific embodiments of the system are described below.

[2353] 1. Voice Input Module

[2354] Device:

[2355] Users input their voice using a device such as a smartphone. The device is equipped with a microphone that captures the user's voice. The voice input is then converted into text data using speech recognition software (e.g., Python's speech_recognition library).

[2356] Examples:

[2357] User: "I want to eat some delicious Chinese food today."

[2358] 2. Emotion Recognition Module

[2359] server:

[2360] The text data sent from the voice input module is analyzed by an emotion recognition engine on the server, using, for example, Python's TextBlob library, to identify the user's emotional state (positive, negative, neutral).

[2361] Examples:

[2362] Text data: "I want to eat some delicious Chinese food today."

[2363] Sentiment analysis result: Negative (slightly tired tone)

[2364] 3. Suggestion Engine

[2365] server:

[2366] Based on the user's emotions and intentions identified by the emotion recognition module, the server suggests potential destinations and services. A suggestion engine runs and accesses external restaurant databases and APIs to retrieve appropriate options.

[2367] Examples:

[2368] Server: "You're looking for a Chinese restaurant in the area. It sounds like you're in the mood for a little relaxation. Would you like a quiet restaurant with a festive atmosphere?"

[2369] 4. Optimization Engine

[2370] server:

[2371] If the user accepts the suggestion, the server will obtain real-time information based on the request and calculate the optimal route and service provision, accessing the external traffic information service API to obtain real-time data.

[2372] 5. Service Provision Module

[2373] Device:

[2374] The device provides the information received from the server to the user, who can then select the suggested options or check the detailed information.

[2375] Examples:

[2376] Selected restaurant information: "This restaurant has a 4.5 rating. The menu includes pasta and pizza. It is open from 11:00 AM to 10:00 PM."

[2377] Prompt Sentence Examples

[2378] An example of input a user might make to the system is shown below.

[2379] User: "I want to eat some delicious Chinese food today."

[2380] System (Response): "You're looking for a Chinese restaurant nearby. It sounds like you're looking for a place to relax. Would you like a quiet restaurant with a festive atmosphere?"

[2381] In this way, the system of the present invention provides a more personalized service by simultaneously considering the user's voice input and emotions.

[2382] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2383] Step 1:

[2384] The user speaks, which is captured by the device's microphone. The device converts this speech data into text using a generative AI model (e.g., a speech recognition library). The converted text data is sent to the server.

[2385] input:

[2386] Audio data

[2387] output:

[2388] Text data

[2389] Step 2:

[2390] The server sends the received text data to an emotion recognition engine, which analyzes the emotional state of the text data using a generative AI model (e.g., Python's TextBlob library). The result of the emotion analysis (positive, negative, neutral) is used in the next step.

[2391] input:

[2392] Text data

[2393] output:

[2394] Emotion analysis results

[2395] Step 3:

[2396] The server uses a generative AI model that identifies the user's intent using the sentiment analysis results and text data as input. Based on the user's intent, destination and service candidates are generated. The generated destination and service candidates are stored on the server.

[2397] input:

[2398] Emotion analysis results

[2399] Text data

[2400] output:

[2401] Destination and service suggestions

[2402] Step 4:

[2403] The server proposes optimal options from the generated list of destinations and services based on the user's intentions and the results of emotion analysis. The proposed options are sent to the terminal and presented to the user.

[2404] input:

[2405] Destination and service suggestions

[2406] output:

[2407] suggestion

[2408] Step 5:

[2409] If the user accepts the suggestion, the device sends the selection to the server, which then accesses an external API to obtain real-time information (e.g., traffic conditions and store congestion status). Based on the obtained real-time information, the server calculates the optimal route and service provision.

[2410] input:

[2411] Accepting the proposal

[2412] output:

[2413] Real-time information

[2414] Step 6:

[2415] The server sends the calculated optimal route and service information to the device, which then provides this information to the user, allowing the user to view detailed information.

[2416] input:

[2417] Real-time information

[2418] output:

[2419] Optimal route and service information

[2420] Step 7:

[2421] The user can check the optimal route and service information provided and take action as needed, allowing the user to take optimal action based on personalized information.

[2422] input:

[2423] Optimal route and service information

[2424] output:

[2425] User Behavior

[2426] This series of processing steps allows users to receive personalized suggestions and services through voice input and emotion recognition.

[2427] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2428] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2429] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2430] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2431] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2432] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2433] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2434] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2435] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2436] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2437] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2438] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2439] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2440] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2441] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2442] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2443] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2444] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2445] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2446] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2447] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2448] The following is further disclosed regarding the above embodiment.

[2449] (Claim 1)

[2450] means for accepting voice input;

[2451] means for converting the received speech into text;

[2452] A means for analyzing the converted text to determine user intent; and

[2453] A means of suggesting destinations based on the user's intent;

[2454] Once the destination is determined, a means for calculating the optimal route using that information;

[2455] a means for obtaining real-time traffic information;

[2456] A means of selecting the optimal route taking into account real-time traffic information;

[2457] a means for providing the selected route information to an automated driving system of the vehicle;

[2458] A system including:

[2459] (Claim 2)

[2460] 10. The system of claim 1, further comprising means for interactively suggesting a destination if the user does not specify a destination.

[2461] (Claim 3)

[2462] 10. The system of claim 1, further comprising means for obtaining and providing detailed destination information to the user.

[2463] "Example 1"

[2464] (Claim 1)

[2465] means for accepting voice input;

[2466] means for converting the received speech into text;

[2467] means for analyzing the converted text to determine user intent;

[2468] A means for suggesting destinations based on the user's intentions;

[2469] Once the destination is determined, a means for calculating the optimal route using that information;

[2470] a means for obtaining real-time traffic information;

[2471] A means of selecting the optimal route taking into account real-time traffic information;

[2472] a means for providing the selected route information to an automated driving system of the vehicle;

[2473] means for querying an external database to obtain detailed destination information and providing it to the user;

[2474] A system including:

[2475] (Claim 2)

[2476] 10. The system of claim 1, further comprising means for interactively suggesting a destination if the user has not specified a destination.

[2477] (Claim 3)

[2478] 10. The system of claim 1, further comprising means for analyzing a dialogue with the user using a dialogue engine and generating an appropriate response.

[2479] "Application Example 1"

[2480] (Claim 1)

[2481] means for accepting voice input;

[2482] means for converting the received speech into text;

[2483] A means for analyzing the converted text to determine user intent; and

[2484] A means of suggesting destinations based on the user's intent;

[2485] Once the destination is determined, a means for calculating the optimal route using that information;

[2486] a means for obtaining real-time traffic information;

[2487] A means of selecting the optimal route taking into account real-time traffic information;

[2488] a means for providing the selected route information to an automated driving system of the vehicle;

[2489] a means for retrieving detailed information about the identified destination from an external database and providing it to the user;

[2490] A means of using a generative AI model to interactively suggest destinations and generate prompts to help the user make appropriate choices;

[2491] A system including:

[2492] (Claim 2)

[2493] 10. The system of claim 1, further comprising means for interactively suggesting a destination if the user does not specify a destination.

[2494] (Claim 3)

[2495] 10. The system of claim 1, further comprising means for obtaining and providing detailed destination information to the user.

[2496] "Example 2: Combining Emotion Engines"

[2497] (Claim 1)

[2498] means for accepting voice input;

[2499] means for converting the received speech into text;

[2500] means for analyzing the converted text and speech data to identify the user's emotional state;

[2501] means for identifying user intent based on the analyzed text data;

[2502] means for suggesting destinations based on the identified user intent and emotional state;

[2503] Once the destination is determined, a means for calculating the optimal route using that information;

[2504] a means for obtaining real-time traffic information from an external traffic information service;

[2505] A means for selecting an optimal route taking into account the acquired real-time traffic information;

[2506] a means for providing the selected route information to an automated driving system of the vehicle;

[2507] A system including:

[2508] (Claim 2)

[2509] 10. The system of claim 1, further comprising means for interactively suggesting a destination if the user has not specified a destination.

[2510] (Claim 3)

[2511] 10. The system of claim 1, further comprising means for retrieving detailed destination information from an external database and providing it to the user.

[2512] "Application example 2 when combining emotion engines"

[2513] (Claim 1)

[2514] means for accepting voice input;

[2515] means for converting the received speech into text;

[2516] A means for analyzing the converted text to determine user intent; and

[2517] a means for recognizing user sentiment from the analyzed text;

[2518] A means to suggest destinations and services based on the user's emotions and intentions,

[2519] Once the destination and service are determined, a means of calculating the optimal route and service provision using that information;

[2520] a means for obtaining real-time information;

[2521] A means of selecting the best route and service taking into account real-time information;

[2522] a means for providing selected route and service information;

[2523] A system including:

[2524] (Claim 2)

[2525] 10. The system of claim 1, further comprising means for interactively suggesting destinations and services if the user has not specified a destination or service.

[2526] (Claim 3)

[2527] 10. The system of claim 1, further comprising means for obtaining and providing detailed destination and service information to the user. [Explanation of symbols]

[2528] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for accepting voice input; means for converting the received speech into text; A means for analyzing the converted text to determine user intent; and A means of suggesting destinations based on the user's intent; Once the destination is determined, a means for calculating the optimal route using that information; a means for obtaining real-time traffic information; A means of selecting the optimal route taking into account real-time traffic information; a means for providing the selected route information to an automated driving system of the vehicle; A system including:

2. 10. The system of claim 1, further comprising means for interactively suggesting a destination if the user has not specified a destination.

3. 10. The system of claim 1, further comprising means for obtaining and providing destination details to the user.

Images