system

Abstract

We provide the system. [Solution] The system includes means for generating artificial intelligence installed in the vehicle to recognize and analyze the user's voice commands, means for communicating with a server to obtain the latest map data and traffic information and calculate the optimal route, and means for performing autonomous driving based on the calculated route information. It also further includes means for monitoring the battery level, suggesting charging facilities, recalculating the route to those facilities if charging is needed, and automatically processing payment upon arrival at the charging facility. Furthermore, the system includes means for recognizing music and video playback commands, obtaining streaming data from the user's smartphone and playing it, and means for coordinating with the National Police Agency and traffic control systems to receive traffic signal information and emergency information and reflect it in autonomous driving.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including: receiving a user utterance; adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot; encoding the prompt; and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] There are many problems in the current transportation system, such as the driving burden on drivers, traffic accidents, environmental load, soaring gasoline prices, lack of system cooperation, and the trouble at charging stations. In addition, in order to maximize the convenience of electric vehicles, updating of map data, real-time acquisition of traffic information, and improvement of the accuracy of autonomous driving are required. Furthermore, providing entertainment during operation and automating the charging process are also important issues.

Means for Solving the Problems

[0005] The present invention is a system that includes means for generating artificial intelligence installed in a vehicle to recognize and analyze user voice commands, means for communicating with a server to obtain the latest map data and traffic information and calculate the optimal route, and means for performing autonomous driving based on the calculated route information. It further includes means for monitoring the battery level, suggesting charging facilities, recalculating the route to those facilities if charging is necessary, and means for automatically processing payment upon arrival at a charging facility. Furthermore, it includes means for recognizing music and video playback commands, obtaining streaming data from the user's smartphone and playing it, and means for coordinating with the National Police Agency and traffic control systems to receive traffic signal information and emergency information and reflect it in autonomous driving, thereby solving various problems.

[0006] Understood. Below are definitions for the key terms included in the patent claims.

[0007] "Generative artificial intelligence" refers to artificial intelligence that recognizes and analyzes user voice commands and generates the necessary instructions.

[0008] A "server" is a device located on the cloud or network that communicates with vehicles to provide map data and traffic information.

[0009] "Map data" refers to data that includes the latest road information and location information.

[0010] "Traffic information" refers to real-time information on road congestion and accidents.

[0011] The "optimal route" is the most efficient path to reach your destination.

[0012] "Autonomous driving" refers to a function where artificial intelligence controls the vehicle and automates the driving process.

[0013] "Battery level" is an indicator that shows the remaining charge of the vehicle's battery.

[0014] "Charging facilities" refer to the infrastructure for charging electric vehicles.

[0015] "Music and video playback instructions" refer to the instructions given to the generative artificial intelligence by the user to play music and videos in the vehicle.

[0016] "Streaming data" refers to data for playing data in real time via the Internet.

[0017] "Traffic control system" refers to a system for managing traffic signals and traffic conditions in real time.

[0018] "Traffic signal information" refers to information regarding the lighting status and timing of traffic signals.

[0019] "Emergency information" refers to information that occurs during emergencies such as accidents and disasters.

[0020] Now, the definitions of all important words are complete.

Brief Description of Drawings

[0021] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6]It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Modes for Carrying Out the Invention

[0022] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0023] First, the language used in the following description will be explained.

[0024] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), and APU (Accelerated Processing Unit).

[0025] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0026] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0027] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0028] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0029] [First Embodiment]

[0030] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0031] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0032] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0033] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0034] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0035] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0036] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0037] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0038] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0039] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0040] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0041] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0042] This invention relates to a vehicle system that integrates autonomous driving and generative artificial intelligence. This system provides the optimal route based on user voice input and achieves autonomous driving using real-time updated map data and traffic information. Furthermore, it provides high levels of convenience and safety to the user through charging station guidance and payment processing, entertainment provision, and coordination with police and traffic control systems.

[0043] Program processing and function description

[0044] Speech recognition and analysis

[0045] User: When getting into the car, specify the destination to the generating AI by voice. For example, say, "I want to go to Yokohama."

[0046] Terminal: Generative artificial intelligence (generative AI) installed in the vehicle receives voice commands, and a voice recognition module analyzes them to extract the destination "Yokohama." The extracted information is then re-notified to the user by voice for confirmation.

[0047] Route calculation and information retrieval

[0048] Terminal (vehicle): Sends destination information to the server. For example, it requests a route to Yokohama.

[0049] Server: Calculates the optimal route based on the latest map data and real-time traffic information, and sends the result back to the vehicle.

[0050] Autonomous driving

[0051] Terminal (vehicle): Based on the received route information, it starts autonomous driving. During operation, it is equipped with various sensors and cameras, and the generating AI analyzes this information to dynamically correct the route.

[0052] Entertainment provider

[0053] User: Instructs the generating AI to "play music" while the vehicle is in operation.

[0054] Terminal (vehicle): Analyzes voice commands, retrieves streaming data from the user's smartphone, and plays it back on the in-vehicle audio system.

[0055] Information and payment for charging facilities

[0056] Terminal (vehicle): Monitors battery level and suggests charging facilities as needed. For example, it might notify you, "Charging is required at the next rest stop."

[0057] User: If you approve the proposal, the route to the nearest charging station will begin recalculating.

[0058] Terminal (vehicle): Upon arrival at a charging station, the system automatically processes payment and begins charging. Once all processing is complete, the user is notified and the vehicle resumes operation.

[0059] Integration with traffic control systems

[0060] Server: Connects in real time with the National Police Agency and traffic control systems to receive traffic signal timings and emergency information.

[0061] Terminal (vehicle): Controls autonomous driving based on received traffic signal information. Also, takes appropriate action in response to emergency information.

[0062] Specific example

[0063] For example, if a user is traveling from Shinjuku, Tokyo to Yokohama, the following scenarios are possible.

[0064] The user gets into the vehicle and gives voice instructions to the AI ​​generator, saying, "Take me to Yokohama, please."

[0065] The terminal (vehicle) analyzes the voice commands and requests route calculation from the server.

[0066] The server calculates the optimal route and sends it to the vehicle, and the generating AI starts autonomous driving.

[0067] When a user requests "play music" during the ride, the AI ​​generates music data from the smartphone and plays it.

[0068] The terminal (vehicle) monitors the battery level and suggests charging facilities if charging is needed.

[0069] Once the user accepts the proposal and arrives at the charging station, the generating AI automatically processes the payment and begins charging.

[0070] The server provides traffic control information, and the generating AI adjusts the route in real time.

[0071] Finally, upon arriving in Yokohama, the generating AI terminates the operation and notifies the user.

[0072] In this way, the present invention can significantly improve user convenience, safety, and efficiency.

[0073] The following describes the processing flow.

[0074] Step 1:

[0075] When a user boards the vehicle, the terminal automatically activates its AI. A passenger sensor detects the user's presence, and a Bluetooth module scans the user's smartphone to attempt a connection. Once a Bluetooth connection is established, the user is notified.

[0076] Step 2:

[0077] The user gives voice instructions to the generating AI about their destination. For example, they might say, "I want to go to Yokohama." The device uses a voice recognition module to analyze the user's voice and extract the destination "Yokohama." The generating AI then verbally notifies the user of the extracted information for confirmation.

[0078] Step 3:

[0079] The terminal sends destination information to the server. The server retrieves the latest map information from the map database, collects real-time traffic information, and calculates the optimal route. The calculation result is then sent back to the terminal.

[0080] Step 4:

[0081] The system initiates autonomous driving based on route information received by the terminal. The generating AI starts the engine and begins operation. During operation, sensors and cameras constantly monitor the surrounding environment, and the generating AI dynamically adjusts the route.

[0082] Step 5:

[0083] The user gives a command during operation, such as "Please play music." The terminal analyzes the voice command and streams music data from the user's smartphone. The audio system activates and plays the requested music.

[0084] Step 6:

[0085] The device monitors the battery level and, if it determines that charging is needed, suggests the most suitable charging location to the user. It notifies the user, "You need to charge at the next rest stop." Once the user accepts the suggestion, the device recalculates the route to the nearest charging location.

[0086] Step 7:

[0087] When the device arrives at the charging station, it automatically initiates the charging process. It processes payment in conjunction with the charging station's system and confirms that charging has started correctly. After charging is complete, it notifies the user and resumes operation.

[0088] Step 8:

[0089] The server connects to the National Police Agency and the traffic control system, receiving real-time information on traffic signal timings and emergency situations. The terminal then incorporates the received traffic control information into the generating AI, ensuring safe and efficient autonomous driving continues.

[0090] Step 9:

[0091] Upon arrival at the destination, the terminal notifies the user that they have arrived at their destination. The generating AI stops autonomous driving and turns off the engine. When the user leaves the vehicle, the passenger sensor detects this and automatically shuts down the generating AI and other systems.

[0092] (Example 1)

[0093] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0094] Modern automotive systems need to integrate many functions, including real-time autonomous driving adjustments based on traffic signals and emergency information, highly accurate route calculation, charging station guidance and automated payment processing, and consistent entertainment delivery. However, existing systems struggle to achieve these efficiently and safely. Furthermore, there is a need to ensure driving safety while maximizing user convenience.

[0095] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0096] In this invention, the server includes a generator artificial intelligence installed in the vehicle, means for recognizing and analyzing user voice commands, means for communicating with the server to obtain the latest map data and traffic information and calculate the optimal route, means for performing autonomous driving based on the calculated route information, means for obtaining and playing streaming data from the user's smartphone to provide entertainment during operation, means for monitoring the surrounding environment in real time and dynamically adjusting the route and speed, and means for coordinating with the National Police Agency and traffic control systems to receive traffic signal information and emergency information and reflect it in autonomous driving. This makes it possible to significantly improve user convenience, safety, and efficiency.

[0097] "Generative artificial intelligence" is a system that uses advanced algorithms, including natural language processing and speech recognition technologies, to analyze user instructions in real time and control autonomous driving or provide entertainment.

[0098] "Means for recognizing and analyzing user voice instructions" refers to a voice recognition module that analyzes voice acquired by microphones inside the vehicle and extracts the user's intended instructions.

[0099] A "server" refers to a computer system that receives requests from vehicles, calculates the optimal route based on map data and traffic information, and transmits that information to the vehicle.

[0100] "Up-to-date map data and traffic information" refers to road maps and traffic condition data obtained in real time from map service providers and traffic information services.

[0101] "Means for calculating the optimal route" refers to an algorithm in which a server calculates the most efficient route under specified conditions based on the vehicle's current location and destination information.

[0102] "Means of autonomous driving" refers to a system that autonomously controls a vehicle based on calculated route information to operate it to its destination.

[0103] "Means of providing entertainment" refers to a system that acquires streaming data from the user's smartphone and plays it back through the vehicle's audio system.

[0104] "A means of monitoring the surrounding environment in real time and dynamically adjusting the route and speed" refers to a function that uses sensors such as LiDAR and cameras to detect the surrounding situation, and then generates AI to analyze that information and adjust the route and speed in real time.

[0105] "Means for receiving traffic signal information and emergency information and reflecting it in autonomous driving" refers to a system that uses signal timing and emergency information obtained in cooperation with the National Police Agency and traffic control systems to reflect it in autonomous driving control.

[0106] "Means of monitoring battery level" refers to a system that constantly monitors the vehicle's battery level and notifies the user when charging is necessary.

[0107] "Means of suggesting charging facilities" refers to a system that searches for the nearest charging station and guides the user when the battery level drops.

[0108] "Method for recalculating the route to the charging facility" refers to an algorithm that, when charging is needed, causes the server to recalculate the optimal route from the current location to the nearest charging facility.

[0109] "Automatic payment processing" refers to a function that automatically completes payment using pre-registered credit card information or a digital wallet upon arrival at a charging station.

[0110] This invention relates to a system that enables autonomous driving within a vehicle and integrates multiple functions, including entertainment provision, guidance and payment processing for charging facilities, and real-time driving adjustments based on traffic signals and emergency information. This system functions with generative artificial intelligence (generative AI model) at its core.

[0111] System Configuration

[0112] hardware

[0113] Terminal: Computer and sensor devices (cameras, LiDAR, radar, microphones, etc.) installed in the vehicle.

[0114] Server: A remote computer system that interacts with map databases and traffic information APIs.

[0115] User device: A mobile device such as a smartphone or tablet.

[0116] software

[0117] Generative AI models: speech recognition modules, route calculation algorithms, entertainment playback systems, payment processing systems.

[0118] Communication protocol: A standard internet protocol used for communication (e.g., HTTP / HTTPS).

[0119] Processing in the autonomous driving system

[0120] Speech recognition and analysis

[0121] The user boards the vehicle and gives a voice command for their destination. For example: "I want to go to Yokohama." The device's microphone captures the voice, and a voice recognition module analyzes it. The analysis result is generated as text and sent to the user for confirmation.

[0122] Root calculation

[0123] The terminal sends destination information to the server, which calculates the optimal route based on the latest map data and real-time traffic information. The calculation result is sent back to the terminal, and the terminal starts autonomous driving.

[0124] Autonomous driving

[0125] The terminal controls the vehicle based on the received route information. Various sensors (camera, LiDAR, radar) monitor the surrounding environment in real time and dynamically adjust the route and speed. The generated AI analyzes this information and sends appropriate operational instructions to the vehicle control system.

[0126] Entertainment provider

[0127] During operation, the user instructs the AI ​​to "play music." The terminal analyzes the voice command, retrieves streaming data from the user's device, and plays it through the in-car audio system.

[0128] Information and payment for charging facilities

[0129] The device monitors the battery level and suggests charging locations as needed. For example, it might notify the user, "You need to charge at the next rest stop." Once the user approves the suggestion, the device recalculates the route to the charging location, automatically processes payment upon arrival, and begins charging.

[0130] Integration with traffic control systems

[0131] The server connects in real time with the National Police Agency and traffic control systems to receive traffic signal timings and emergency information. Based on this information, the terminal updates the control plan for autonomous driving, ensuring safe and efficient operation.

[0132] Specific example

[0133] For example, if a user instructs the system to "go to Yokohama," the terminal sends this information to the server. The server calculates the route and sends the result back to the terminal. Once autonomous driving begins, if the user instructs the system to "play music" during the journey, the terminal retrieves music from the user's device and plays it. It suggests charging stations as needed, and resumes operation once charging is complete. Based on traffic information, necessary route changes are reflected in real time.

[0134] Examples of prompt messages include "I want to go to Yokohama," "Please play music," and "I need to charge my phone at the next rest stop."

[0135] In this way, integrated systems can significantly improve user convenience, security, and efficiency.

[0136] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0137] Step 1:

[0138] The user boards the vehicle and gives voice instructions for their destination. The input is voice, and the output is voice data. For example, the user might say, "I want to go to Yokohama."

[0139] Step 2:

[0140] The device captures audio data through its microphone. The input is the user's voice, and the output is digitized audio data. This audio data is sent to the speech recognition module of the generating AI.

[0141] Step 3:

[0142] The terminal's voice recognition module analyzes the voice data and converts it into text information. The input is digitized voice data, and the output is text-based instruction information. Specifically, it extracts the destination "Yokohama" as text.

[0143] Step 4:

[0144] The device then re-notifies the user of the analyzed text information. The input is transcribed instruction information, and the output is an audio notification. Specifically, it plays a confirmation message saying, "Your destination is Yokohama. Is this correct?"

[0145] Step 5:

[0146] The terminal sends text information to the server and requests the optimal route based on the latest map data and traffic information. The input is destination information (text), and the output is request data. Specifically, the data is transmitted using the Internet Protocol.

[0147] Step 6:

[0148] The server retrieves the latest data from the map database and traffic information API and calculates the optimal route. The input is the request data, and the output is the optimal route information. Specifically, it uses an algorithm to calculate the route.

[0149] Step 7:

[0150] The server returns the calculation results to the terminal. The input is the calculated route information, and the output is the returned data. Specifically, it sends data including the coordinates of each point along the route, the total distance, the time required, and any points to note.

[0151] Step 8:

[0152] The terminal initiates autonomous driving based on the received route information. The input is the optimal route information, and the output is vehicle control instructions. Specifically, it sends instructions to the vehicle's control system and starts autonomous driving.

[0153] Step 9:

[0154] The terminal monitors the surrounding environment in real time using sensors mounted on the vehicle (camera, LiDAR, radar), and the generated AI analyzes this information to dynamically adjust the route and speed. The input is sensor data, and the output is vehicle control instructions. Specific operations include obstacle detection and lane keeping.

[0155] Step 10:

[0156] During operation, the user gives a voice command saying, "Please play music." The input is voice, and the output is voice data. Specifically, the user says the command, "Please play music."

[0157] Step 11:

[0158] The terminal uses a streaming service API to retrieve streaming data from the user's device and play it back on the in-car audio system. The input is a voice command, and the output is audio data. Specifically, it retrieves streaming data and starts playback.

[0159] Step 12:

[0160] The device monitors the battery level and suggests charging locations if charging is needed. The input is battery data, and the output is notification data. Specifically, it sends a notification to the user stating, "Charging is needed at the next rest stop."

[0161] Step 13:

[0162] Once the user approves the charging suggestion, the device recalculates the route to the nearest charging station. The input is the user's approval information, and the output is the recalculated route information. Specifically, it sends a recalculation request to the server.

[0163] Step 14:

[0164] Upon arrival at a charging station, the device automatically processes payment and begins charging. The input is information about arrival at the charging station, and the output is payment confirmation and charging start notification. Specifically, it completes the payment using pre-registered payment information.

[0165] Step 15:

[0166] The server connects in real time with the National Police Agency and traffic control systems to receive traffic signal timings and emergency information. Inputs are traffic signal data and emergency information, while outputs are control instruction data. Specifically, it collects traffic information and sends appropriate instructions to terminals.

[0167] Step 16:

[0168] The terminal updates the autonomous driving control plan based on received traffic signal and emergency information. Inputs are traffic signal data and emergency information, while output is the updated control instructions. Specifically, it adjusts speed in accordance with signal timing and takes appropriate action based on emergency information.

[0169] Through this series of steps, the system of the present invention can improve user convenience, safety, and efficiency.

[0170] (Application Example 1)

[0171] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0172] Conventional autonomous vehicle systems have not adequately improved user convenience and safety. Specifically, there is a demand for seamless travel, such as users specifying destinations by voice and enjoying entertainment, as well as driving that reflects real-time traffic information, but systems that can achieve this are still limited. In addition, the procedures at charging stations are cumbersome, and automation of charging is desired. Therefore, there is a need for a system that enhances overall convenience and safety in autonomous driving.

[0173] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0174] In this invention, the server includes means for recognizing and analyzing voice instructions, means for communicating with the server to obtain the latest map data and traffic information and calculate the optimal route, and means for performing autonomous driving based on the calculated route information. This enables seamless destination setting, optimal route calculation, and autonomous driving control based on the user's voice instructions. Furthermore, by including functions such as entertainment playback means, battery level monitoring and charging facility guidance, automatic payment means, and cooperation with traffic control systems, user convenience and safety can be greatly improved. In addition, high-quality entertainment can be provided by utilizing voice instruction analysis using prompts and streaming playback functions.

[0175] "Means for recognizing and analyzing voice commands" refers to devices and software that understand voice commands entered by a user, analyze their content, and translate that into specific actions.

[0176] "Means of communicating with a server to obtain the latest map data and traffic information and calculate the optimal route" refers to equipment and software that communicates with an external server to obtain map data and traffic information that are updated in real time, and calculates the optimal driving route based on that data.

[0177] "Means for performing automated driving based on calculated route information" refers to devices and programs for automatically driving a vehicle based on acquired and calculated route information.

[0178] "Means for playing entertainment content" refers to devices and software used to play entertainment (music, videos, etc.) as instructed by the user.

[0179] "A means of monitoring battery levels, suggesting charging facilities, and recalculating routes to charging facilities as needed" refers to a system that constantly monitors the vehicle's battery level, suggests the nearest charging facility when necessary, and calculates a new route to that facility.

[0180] "Means of automatically processing payment upon arrival at a charging facility" refers to equipment and software that automatically completes payment and starts the charging process upon arrival at a charging facility, without requiring manual intervention from the user.

[0181] "Means of receiving traffic signal information and emergency information in cooperation with the National Police Agency and traffic control systems and reflecting it in autonomous driving" refers to a system that receives signal information and emergency information from the National Police Agency and traffic control systems in real time and adjusts the control of autonomous driving based on that information.

[0182] "Means of acquiring information from smart devices and utilizing it while driving" refers to a system that utilizes information acquired from the user's smartphone or other smart devices while driving, and reflects it in vehicle operation and entertainment provision.

[0183] "Means for analyzing and playing voice commands via prompts" refers to devices and software that analyze prompts given by a user via voice (e.g., a command to play music) and provide entertainment based on those commands.

[0184] "Means of acquiring and playing streaming data from a smartphone" refers to equipment and software that acquire data (such as music and videos) delivered in streaming format from a user's smartphone and play it back on an in-car system.

[0185] This invention relates to a vehicle system that integrates autonomous driving and generative AI. This system provides the optimal route based on user voice input and achieves autonomous driving using real-time updated map data and traffic information. Furthermore, it offers high levels of convenience and safety to the user through features such as charging station guidance and payment processing, entertainment provision, and coordination with police and traffic control systems.

[0186] Speech recognition and analysis

[0187] The user gets into the vehicle and specifies the destination to the generating AI by voice, saying, "Please take me to Yokohama." The voice recognition module installed in the vehicle receives this voice instruction, analyzes it, and extracts the destination "Yokohama." The generating AI then verbally notifies the user of the extracted information for confirmation. The voice recognition technology used here is the Google Speech Recognition API.

[0188] Route calculation and information retrieval

[0189] The terminal (vehicle) sends destination information to the server, which calculates the optimal route based on the latest map data and real-time traffic information. The calculation result is sent back to the terminal, and the terminal starts autonomous driving. A real-time map API is used in this process.

[0190] Autonomous driving

[0191] The terminal (vehicle) begins autonomous driving based on calculated route information. During operation, various sensors and cameras are installed, and the generating AI analyzes this information to dynamically correct the route. A high-precision GPS module is used in the hardware.

[0192] Entertainment provider

[0193] When a user commands "play music" while driving, the device analyzes the voice command, retrieves streaming data from the user's smartphone, and plays it through the car's audio system. This allows users to enjoy entertainment while driving.

[0194] Information and payment for charging facilities

[0195] The terminal monitors the battery level and suggests charging stations as needed. Once the user approves the suggestion, it recalculates the route to the nearest charging station. Upon arrival at the charging station, the terminal automatically processes the payment and begins charging. Once all processes are complete, the terminal notifies the user and resumes operation.

[0196] Integration with traffic control systems

[0197] The server connects in real time with the National Police Agency and traffic control systems to receive traffic signal timings and emergency information. The terminal uses this information to control autonomous driving and take appropriate action in response to emergency information.

[0198] Specific example

[0199] If a user is traveling from Shinjuku, Tokyo to Yokohama, the following prompt will be used:

[0200] "Launch SmartDrive Assistant"

[0201] "Please take me to Yokohama."

[0202] "Tell me the nearest charging station."

[0203] "Play the music."

[0204] In this way, the present invention can significantly improve user convenience, safety, and efficiency.

[0205] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0206] Step 1:

[0207] The user gets into the vehicle and enters a voice command specifying the destination. For example, they might say, "Please take me to Yokohama." A generative AI model receives this voice command and analyzes it with a speech recognition module. The input is the user's voice command, and the output is the analyzed text data. Through this process, the destination (in this case, "Yokohama") is extracted.

[0208] Step 2:

[0209] The terminal (vehicle) sends the analyzed destination information to the server. The input is destination text data, and the output is the result of the transmission to the server. The server uses real-time map data and traffic information to calculate the optimal route. This process generates the optimal route information to the destination.

[0210] Step 3:

[0211] The server returns the calculated optimal route information to the terminal (vehicle). The input is the route information calculated on the server, and the output is the result received by the terminal. The terminal receives this route information and starts autonomous driving. Through this process, the vehicle automatically travels towards its destination.

[0212] Step 4:

[0213] When a user gives a command such as "Play music" during operation, the generative AI model analyzes this voice command. The input is the voice command, and the output is the analyzed text data. This process recognizes the user's entertainment request.

[0214] Step 5:

[0215] The device retrieves streaming data from the user's smartphone based on the analyzed entertainment instructions. The input is the text data of the entertainment instructions, and the output is the retrieved streaming data. This process makes music and videos playable.

[0216] Step 6:

[0217] The device monitors the battery level and suggests charging facilities as needed. The input is battery level data, and the output is a charging suggestion notification to the user. When the user accepts the suggestion, the route to the nearest charging facility is recalculated. This process generates new route information to the charging facility.

[0218] Step 7:

[0219] Upon arrival at a charging station, the terminal automatically processes payment and begins charging. Inputs are data indicating arrival at the charging station and payment information, while outputs are the payment processing result and a charging start notification. This process allows users to complete charging without any effort on their part.

[0220] Step 8:

[0221] The server acquires information in real time from the National Police Agency and traffic control systems and transmits it to terminals. Inputs include traffic signal information and emergency information, while output is the results of the transmission to the terminals. The terminals use this information to control autonomous driving and respond to emergencies. This process ensures safe and efficient operation.

[0222] In this way, vehicle systems can significantly improve user convenience and safety through advanced autonomous driving and entertainment provision, automated charging and payment, and real-time integration of traffic information.

[0223] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0224] This invention relates to a vehicle system that integrates autonomous driving, generative artificial intelligence, and an emotion engine. This system provides the optimal route based on user voice input and achieves autonomous driving using the latest map data and traffic information. Furthermore, it offers high levels of convenience and safety to the user by providing charging station guidance and payment processing, entertainment, coordinating with police and traffic control systems, and recognizing the user's emotions.

[0225] Program processing and function description

[0226] Speech recognition and analysis

[0227] User: When boarding the vehicle, you can use voice commands to tell the generated artificial intelligence your destination. For example, you might say, "I want to go to Yokohama."

[0228] Terminal: The voice recognition module analyzes the user's voice and extracts destination information. The extracted information is then re-notified to the user via voice for confirmation.

[0229] Route calculation and information retrieval

[0230] Terminal: Sends destination information to the server. The server retrieves the latest map information from the map database and calculates the optimal route based on real-time traffic information. The calculation result is then sent back to the vehicle.

[0231] Autonomous driving

[0232] Terminal: Based on the received route information, it initiates autonomous driving. The generating AI starts the engine and begins operation. During operation, sensors and cameras constantly monitor the surrounding environment, and the generating AI dynamically adjusts the route.

[0233] Entertainment provider

[0234] User: Instructs the generating AI to "play music" while the vehicle is in operation.

[0235] Terminal: Analyzes voice commands, retrieves streaming data from the user's smartphone, and plays it. The in-car audio system activates and plays the desired music.

[0236] Information and payment for charging facilities

[0237] Device: Monitors battery level and suggests charging locations as needed. For example, it might notify the user, "You need to charge at the next rest stop." Once the user approves the suggestion, the device recalculates the route to the nearest charging location.

[0238] Terminal: Upon arrival at a charging station, the terminal automatically initiates the charging process. It processes payment in conjunction with the charging station's system and confirms that charging is started correctly. After charging is complete, it notifies the user and resumes operation.

[0239] Integration with traffic control systems

[0240] Server: Connects to the National Police Agency and traffic control systems to receive real-time information on traffic signal timings and emergency information.

[0241] Terminal: By reflecting the received traffic control information into the generating AI, it enables safe and efficient autonomous driving to continue.

[0242] Emotion recognition and response

[0243] The device features an emotion engine that analyzes the user's facial expressions and voice tone. The emotion engine recognizes the user's emotions and provides that information to the generating AI.

[0244] Terminal: Selects the appropriate driving mode based on the user's emotions. For example, if the user is feeling stressed, it will select a relaxing driving mode.

[0245] Terminal: Evaluates the user's stress level and fatigue, and suggests breaks as needed. Based on the evaluation results, it recalculates the route to rest points and suggests it to the user.

[0246] Device: Selects and suggests entertainment based on the user's mood. For example, if the user is in a good mood, it suggests rhythmic music. Conversely, if the user is sad, it suggests relaxing music.

[0247] Specific example

[0248] For example, if a user is traveling from Shinjuku, Tokyo to Yokohama, the following scenario will be executed.

[0249] The user gets into the vehicle and gives voice instructions to the AI ​​generator, saying, "Please take me to Yokohama."

[0250] The terminal analyzes the voice command and requests route calculation from the server.

[0251] The server calculates the optimal route, and the generating AI starts the automated driving process.

[0252] When a user requests "play music" during the ride, the AI ​​generates music data from the smartphone and plays it.

[0253] The device monitors the battery level and suggests charging facilities if charging is needed.

[0254] Once the user accepts the proposal and arrives at the charging station, the generating AI automatically processes the payment and starts charging.

[0255] The server provides traffic control information, and the generating AI adjusts the route in real time.

[0256] The device analyzes the user's facial expressions and voice, and if it determines that the user is tired, it suggests taking a break and guides them to an appropriate resting place.

[0257] Upon arrival in Yokohama, the AI ​​generates the data and terminates the operation, notifying the user.

[0258] In this way, the present invention can significantly improve user convenience, safety, and efficiency.

[0259] The following describes the processing flow.

[0260] Step 1:

[0261] When a user boards the vehicle, the terminal automatically activates its AI. A passenger sensor detects the user's presence, and a Bluetooth module scans the user's smartphone to attempt a connection. Once a Bluetooth connection is established, the user is notified.

[0262] Step 2:

[0263] The user gives voice instructions to the AI ​​generating the destination. For example, they might say, "I want to go to Yokohama." The device uses a voice recognition module to analyze the user's voice and extract the destination "Yokohama." The extracted information is then re-notified to the user via voice for confirmation.

[0264] Step 3:

[0265] The terminal sends destination information to the server. The server retrieves the latest map information from the map database, collects real-time traffic information, and calculates the optimal route. The calculation result is then sent back to the terminal.

[0266] Step 4:

[0267] The system initiates autonomous driving based on route information received by the terminal. The generating AI starts the engine and begins operation. During operation, various sensors and cameras constantly monitor the surrounding environment, and the generating AI dynamically adjusts the route.

[0268] Step 5:

[0269] The user gives a voice command, such as "I want to play music," while driving. The terminal analyzes the voice command and streams music data from the user's smartphone. The in-car audio system activates and plays the requested music.

[0270] Step 6:

[0271] The device monitors the battery level and, if it determines that charging is needed, suggests the most suitable charging location to the user. It notifies the user, "You need to charge at the next rest stop." Once the user accepts the suggestion, the device recalculates the route to the nearest charging location.

[0272] Step 7:

[0273] When the device arrives at the charging station, it automatically initiates the charging process. It processes payment in conjunction with the charging station's system and confirms that charging has started correctly. After charging is complete, it notifies the user and resumes operation.

[0274] Step 8:

[0275] The server connects to the National Police Agency and the traffic control system, receiving real-time information on traffic signal timings and emergency situations. The terminal then incorporates the received traffic control information into the generating AI, ensuring safe and efficient autonomous driving continues.

[0276] Step 9:

[0277] The device uses an emotion engine that analyzes the user's facial expressions and voice tone to recognize their emotions. If the user is stressed, the emotion engine provides that information to the generating AI, which then selects a relaxing driving mode.

[0278] Step 10:

[0279] The system assesses the user's stress level and fatigue, and suggests breaks as needed. Once the user approves the suggestion, the device recalculates the route to an appropriate rest stop and notifies the user.

[0280] Step 11:

[0281] The terminal makes a selection of entertainment according to the user's mood. For example, if the user is in a good mood, rhythmic music is proposed, and conversely, if the user is sad, relaxing music is proposed.

[0282] Step 12:

[0283] When arriving at the destination, the terminal notifies the user that "Arrived at the destination". The generative AI stops the autonomous driving and turns off the engine. When the user leaves the vehicle, the occupancy sensor detects it and automatically shuts down the generative AI and other systems.

[0284] (Example 2)

[0285] Next, Example 2 will be described. In the following description, the data processing device 12 is referred to as a "server", and the smart device 14 is referred to as a "terminal".

[0286] Currently, many autonomous driving systems existing in the market can recognize the user's voice instructions, calculate the route, and perform autonomous driving, but lack the function of recognizing the user's emotions and adjusting the driving mode. In addition, various functions for enhancing the user's convenience, such as the proposal of charging facilities and automatic payment, the real-time reflection of traffic signal information, and the entertainment function, are required. Moreover, due to the lack of technology for integrally executing these functions, improving the user experience is an issue.

[0287] The identification processing performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server has a generating artificial intelligence installed in the vehicle and includes means for recognizing and analyzing the user's voice instructions, means for communicating with the server to acquire the latest map data and traffic information and calculate the optimal route, means for performing automatic driving based on the calculated route information, means for analyzing the user's facial expressions and voice tone to recognize emotions and providing that information to the generating artificial intelligence, and means for selecting a driving mode based on the user's emotions. This makes it possible to adjust driving according to the user's emotions, significantly improving driving safety and comfort. Furthermore, by integrating and executing multiple functions such as suggesting charging facilities and automatic payment, reflecting real-time traffic signal information, and providing entertainment, user convenience and the quality of the experience can be enhanced.

[0288] "Generative artificial intelligence installed in a vehicle" refers to artificial intelligence installed in the vehicle's computer system that recognizes and analyzes user voice commands and performs autonomous driving and other vehicle operation management.

[0289] "User voice commands" refers to the act of a user giving instructions by voice regarding vehicle operation, setting a destination, etc.

[0290] "Means for recognizing and analyzing voice commands" refers to the collective term for hardware and software that capture, analyze, and understand user voice commands.

[0291] A "server" refers to a remote computer system that receives data transmitted from a vehicle, processes it, and sends the results back to the vehicle.

[0292] "Latest map data" refers to data that includes road and topographic information updated in real time using a geographic information system.

[0293] "Traffic information" refers to information about current road conditions, including dynamic data such as congestion, traffic accidents, and road construction.

[0294] "Means for calculating the optimal route" refers to algorithms and software used to calculate the most efficient and safe route to the user's destination.

[0295] "Means of autonomous driving" refers to an integrated system of hardware and software that enables autonomous operation of a vehicle.

[0296] "Means of recognizing emotions by analyzing a user's facial expressions and voice tone" refers to hardware and software that analyze the characteristics of a user's face and voice to determine their emotional state.

[0297] "Means for selecting a driving mode based on user emotions" refers to a control system that selects the optimal driving mode according to the recognized emotions of the user.

[0298] "Means of monitoring battery level" refers to sensors and software that continuously monitor the vehicle's battery status and provide necessary information.

[0299] "A means of suggesting charging facilities and recalculating the route to those facilities when charging is needed" refers to technology that, when the battery level drops, suggests the nearest charging facility to the user and calculates and suggests the optimal route to that facility.

[0300] "A means of automatically processing payment upon arrival at a charging facility" refers to a system that automatically completes payment when charging begins at a charging station.

[0301] "Means for recognizing music and video playback commands and acquiring streaming data from the user's smartphone for playback" refers to technology that recognizes a user's command to play music or videos, acquires the necessary data from the smartphone, and plays the content.

[0302] The means of "cooperating with the police department and traffic control systems, receiving traffic signal information and emergency information, and reflecting them in autonomous driving" refers to the technology for exchanging data with the police and traffic control centers in real time, receiving information on traffic signal changes and emergencies, and applying it to the autonomous driving system.

[0303] The present invention relates to a vehicle system that integrates autonomous driving, generative artificial intelligence, and an emotion engine. This system provides an optimal route based on the user's voice input and realizes autonomous driving using the latest map data and traffic information. In addition, it provides high convenience and safety for users by guiding charging stations and payment processing for improved convenience, providing entertainment, cooperating with the police and traffic control systems, and further recognizing the user's emotions.

[0304] Voice recognition and analysis

[0305] The user gets into the vehicle and gives a voice instruction such as "Please take me to Yokohama".

[0306] The terminal (in-vehicle system) captures the voice with a microphone installed in the vehicle and sends it to a voice recognition module (e.g., Google Cloud Speech-to-Text).

[0307] The terminal uses the voice recognition module to convert the voice data into text and extracts the destination information of "Yokohama". This information is confirmed with the user by voice.

[0308] Route calculation and acquisition of traffic information

[0309] The terminal sends the confirmed destination information to the server. An API request is made to the server's URL and request data including the destination is sent.

[0310] The server obtains the latest map information and current traffic situation information from a map database (e.g., Google Maps API).

[0311] The server calculates the optimal route based on the information it has acquired, and this calculation uses real-time traffic data. The calculation result is returned to the terminal as an API response.

[0312] Autonomous driving

[0313] The terminal passes the route information it receives to a generating AI (e.g., Tesla's AutoPilot).

[0314] The terminal uses generated AI to start the engine and begin autonomous driving. During operation, it uses a 100-directional sensor and cameras to monitor the surrounding environment in real time.

[0315] The terminal uses generated AI to analyze data from sensors and cameras, and adjusts the route and driving actions as needed.

[0316] Entertainment provider

[0317] The user can give a voice command to "play music" while the vehicle is in operation.

[0318] The device analyzes the voice command and uses the API of a music streaming service (e.g., Spotify) to retrieve music data from the smartphone.

[0319] The device activates the in-car audio system and plays the acquired music data.

[0320] Information on charging facilities and payment processing

[0321] The device constantly monitors the battery level, and when the battery level drops below 20%, it notifies the user that "charging is needed at the next rest stop."

[0322] The user approves the notification.

[0323] The device sends another API request to the server to search for the nearest charging station.

[0324] The server provides information on the nearest charging station, recalculates the route, and sends it back to the terminal.

[0325] Upon arrival at the charging station, the system automatically initiates the charging process and handles payment in conjunction with the charging station's system. Once charging is complete, the user is notified by voice, and the vehicle resumes operation.

[0326] Integration with traffic control systems

[0327] The server connects in real time with the National Police Agency and traffic control systems to receive traffic signal timings and emergency information.

[0328] The server sends the received information to the terminal.

[0329] The terminal uses AI generation to dynamically reflect traffic control information, ensuring safe operation continues.

[0330] Emotion recognition and adaptive responses

[0331] The device is equipped with an emotion engine (e.g., Microsoft Azure Cognitive Services) that analyzes the user's facial expressions and voice tone.

[0332] The device analyzes the user's emotions from voice and camera data and recognizes that they are experiencing stress.

[0333] The device uses AI generation to select a relaxing driving mode based on the user's emotional information.

[0334] If the device detects that the user is stressed, it will suggest and play relaxing music. It will also suggest taking a break if it detects fatigue.

[0335] If the user agrees to the suggestion, the device calculates and guides them to the optimal resting place.

[0336] Specific examples and examples of prompt sentences for generative AI models

[0337] For example, if a user is traveling from Shinjuku, Tokyo to Yokohama:

[0338] The user gives a voice command saying, "Please take me to Yokohama."

[0339] The terminal analyzes the voice command and requests route calculation from the server.

[0340] The server calculates the optimal route and sends it back to the terminal.

[0341] The device initiates autonomous driving, monitoring the surroundings in real time while driving.

[0342] When the user requests that the device play music, the device will play music from the smartphone.

[0343] The device monitors the battery level and suggests charging facilities as needed. If the user agrees, it will guide them to the charging facility and automatically process payment and begin charging upon arrival.

[0344] The server transmits traffic control information to the terminal, which then adjusts its driving to ensure safety.

[0345] The device analyzes the user's emotions and, if necessary, provides guidance on relaxing driving modes or taking breaks.

[0346] Specific examples of prompt statements for generative AI models:

[0347] "Calculate the optimal route to our destination, Yokohama, and ensure safe and efficient driving."

[0348] "Please explain the procedure for analyzing user voice commands and providing entertainment."

[0349] "If the current battery level is below 20%, please explain how to suggest the nearest charging station and process the payment automatically."

[0350] "Please explain the process for recognizing user emotions and making suggestions to reduce stress."

[0351] The specific embodiments of the present invention have now been described. This embodiment significantly improves user safety and convenience.

[0352] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0353] Step 1: Receiving and analyzing voice input

[0354] Input: The user gives a voice command inside the vehicle saying, "Please take me to Yokohama."

[0355] Terminal: A microphone inside the vehicle captures the user's voice and sends the audio data to a speech recognition module (e.g., Google Cloud Speech-to-Text). The speech recognition module analyzes the audio data and extracts destination information such as "Yokohama".

[0356] Output: Destination information is returned to the terminal as text data.

[0357] Operation: The device confirms the voice analysis results with the user via voice. It asks the user, "Is Yokohama your destination?"

[0358] Step 2: Route calculation and acquisition of traffic information

[0359] Input: The terminal is given information about the destination, "Yokohama".

[0360] Terminal: Sends destination information to the server. The data, including the destination, is sent to the server as an API request.

[0361] Server: Retrieves the latest map information and real-time traffic information from a map database (e.g., Google Maps API). This information is used to calculate the optimal route.

[0362] Output: The calculation result is sent back to the terminal as route information.

[0363] Operation: The terminal receives route information calculated by the server and uses it in subsequent steps.

[0364] Step 3: Start autonomous driving

[0365] Input: Route information is provided to the terminal.

[0366] Terminal: Passes the received route information to the generating AI (e.g., Tesla's AutoPilot).

[0367] Terminal: The generating AI starts the engine and begins autonomous driving. It monitors the surrounding environment in real time using a 100-directional sensor and camera.

[0368] Output: The start state for autonomous driving mode is generated.

[0369] Operation: The generating AI analyzes data from sensors and cameras and adjusts the route and driving actions in real time.

[0370] Step 4: Providing Entertainment

[0371] Input: The user gives a voice command saying, "Please play music."

[0372] Terminal: Analyzes voice commands and retrieves music data from the user's smartphone using the API of a music streaming service (e.g., Spotify).

[0373] Terminal: Activates the in-car audio system and plays the acquired music data.

[0374] Output: Music playback begins.

[0375] Operation: The device streams music data based on voice commands and plays it on the audio system.

[0376] Step 5: Guidance to charging facilities and payment processing

[0377] Input: The device monitors the battery level, and when the battery level falls below 20%, it triggers a response.

[0378] Device: Notifies the user that "charging is required at the next rest stop."

[0379] User: Accept notification.

[0380] Terminal: Sends another API request to the server to search for the nearest charging station.

[0381] Server: Provides information on the nearest charging station, recalculates the route, and sends it back to the terminal.

[0382] Terminal: Upon arrival at the charging station, it automatically initiates the charging process and handles payment processing in conjunction with the charging station's system.

[0383] Output: Charging and payment processing are complete.

[0384] Operation: The device starts the charging process, notifies the user when it is complete, and resumes operation.

[0385] Step 6: Integration with traffic control systems

[0386] Input: The server receives real-time traffic signal information and emergency information from the National Police Agency and traffic control systems.

[0387] Server: Sends received information to the terminal.

[0388] Terminal: The generating AI dynamically reflects traffic control information, ensuring safe operation continues.

[0389] Output: The operational status is updated based on traffic control information.

[0390] Operation: The generating AI reflects traffic signal information in real time and adjusts driving accordingly.

[0391] Step 7: Emotion Recognition and Adaptive Response

[0392] Input: The user's facial expressions and voice tone are captured by the device.

[0393] Terminal: An emotion engine (e.g., Microsoft Azure Cognitive Services) analyzes the user's facial expressions and voice tone to recognize emotions.

[0394] Terminal: Provides emotional information to the AI ​​that generates it.

[0395] Generative AI: Selects a relaxing driving mode based on the user's emotions. Suggests and plays relaxing music as needed.

[0396] Output: The system initiates driving modes and provides entertainment tailored to the user's mood.

[0397] Operation: The generated AI selects a driving mode based on the user's emotional information and adjusts the driving to promote relaxation. It suggests breaks as needed and guides the user to the optimal rest stop.

[0398] (Application Example 2)

[0399] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0400] In modern autonomous vehicles, it is crucial to accurately recognize user voice commands and provide optimal routes using up-to-date map data and traffic information. However, in addition to this, it is also necessary to handle charging station guidance and payment processing, appropriate driving modes and entertainment based on user emotion recognition, and the reflection of real-time traffic control information. Conventional systems lack these comprehensive functions, resulting in challenges in ensuring sufficient user convenience and safety.

[0401] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a generating artificial intelligence installed in the vehicle and means for recognizing and analyzing the user's voice instructions, means for communicating with the server to acquire the latest map data and traffic information and calculate the optimal route, means for performing autonomous driving based on the calculated route information, means for having an emotion recognition engine to recognize emotions from the user's facial expressions and voice tone and provide driving modes and entertainment based on that, and means for guiding users to charging facilities and processing payments. This makes it possible to significantly improve user convenience and safety by providing the optimal route based on the user's voice instructions, recognizing and adapting to the situation during autonomous driving, guiding users to charging stations and processing payments, and providing entertainment that responds to emotions.

[0402] "Generative artificial intelligence" is an artificial intelligence that recognizes and analyzes user voice commands and generates the information necessary for decision-making.

[0403] "Map data" refers to data that represents geographical information such as roads, buildings, and topography in a digital format.

[0404] "Traffic information" refers to data that shows the current traffic conditions, including real-time information such as traffic congestion and accidents.

[0405] The "optimal route" is the most efficient path between a designated starting point and a destination, taking into account factors such as travel time, distance, and traffic conditions.

[0406] "Autonomous driving" is a technology that allows vehicles to move autonomously without driver intervention.

[0407] An "emotion recognition engine" is a technology that analyzes a user's facial expressions and voice tone to identify their emotional state.

[0408] "Entertainment provision" refers to the function of playing music, videos, and other entertainment content for users.

[0409] The "Charging Station Guide" is a function that displays the nearest suitable charging station to the user based on the vehicle's battery level.

[0410] "Payment processing" is the process by which the system pays for the services or goods provided on behalf of the user.

[0411] This invention relates to a system for autonomous driving vehicles that integrates generative artificial intelligence, an emotion recognition engine, real-time traffic information, and entertainment provision functions.

[0412] First, when a user gets into the vehicle, the AI ​​recognizes and analyzes the user's voice instructions. For example, if the user says, "I want to go to Yokohama," the voice is converted to text using the Google Cloud Speech-to-Text API, and "Yokohama" as the destination is extracted.

[0413] Next, this destination information is sent to a server, which uses the Google Maps API to calculate the optimal route based on the latest map data and real-time traffic information. The calculation results are sent back to the vehicle, and autonomous driving begins. Autonomous driving is performed using each vehicle manufacturer's autonomous driving API.

[0414] While driving, the generated AI uses an emotion recognition engine powered by the Affectiva SDK to analyze the user's facial expressions and voice tone, identifying the user's emotional state. For example, if the user is tired, the system will suggest a rest stop. This improves user safety and comfort.

[0415] Furthermore, when the battery level drops, the system automatically suggests a charging station. Once the user approves, it uses the Google Maps API to recalculate the optimal route to the charging station, and payment is automatically processed upon arrival.

[0416] In providing entertainment, when a user requests "play music," the AI ​​generates music and videos using the Spotify API or YouTube Data API and plays them through the in-car audio system.

[0417] Furthermore, the system collaborates with the National Police Agency and traffic control systems to acquire traffic signal and emergency information in real time, and adjusts autonomous driving based on this information. This enables safe operation that is in line with traffic conditions.

[0418] Finally, the following are concrete examples and examples of prompt messages.

[0419] Specific example:

[0420] For example, if a user is traveling from Shinjuku, Tokyo to Yokohama, the following scenario will be executed: The user gets into the vehicle and gives a voice command to the generating AI, "Take me to Yokohama, please." The voice is then converted to text using the Google Cloud Speech-to-Text API. Next, the optimal route is calculated using the Google Maps API, and autonomous driving begins. During the journey, if the user requests, "Please play music," the generating AI will play music using the Spotify API. If the battery level drops, the nearest charging station will be indicated, and payment will be processed automatically. In addition, the route is adjusted in real time based on traffic control information.

[0421] Example of a prompt:

[0422] When a user says, "Take me to Yokohama," the voice recognition module analyzes the voice, calculates the optimal route using the Google Maps API, and starts autonomous driving. If the user says, "Play music" during the ride, music will be played using the Spotify API. If the battery is low, the nearest charging station will be guided, and payment processing will be initiated. The emotion recognition engine analyzes the user's emotions and provides entertainment suggestions based on that.

[0423] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0424] Step 1:

[0425] The user boards the vehicle and gives voice commands to the generating artificial intelligence. For example, they might say, "I want to go to Yokohama." This voice input is acquired through the microphone inside the vehicle.

[0426] Input: Voice Instructions

[0427] Output: Audio data

[0428] Step 2:

[0429] The device uses the Google Cloud Speech-to-Text API to convert the user's voice commands into text. The destination is extracted based on the converted text data. This analysis result is recorded in a database.

[0430] Input: Audio data

[0431] Output: Text data (e.g., "I want to go to Yokohama")

[0432] Step 3:

[0433] The device sends the acquired text data to the server. The server uses the Google Maps API to obtain the latest map data and real-time traffic information based on the destination, and calculates the optimal route based on that data.

[0434] Input: Text data (destination information)

[0435] Output: Map data, traffic information, optimal route

[0436] Step 4:

[0437] The server sends the calculated optimal route information to a terminal inside the vehicle. Based on the received route information, the terminal issues instructions to the autonomous driving system and starts autonomous driving. During autonomous driving, sensor data (cameras, LiDAR, etc.) is constantly monitored, and the generating AI dynamically adjusts the route.

[0438] Input: Optimal route information

[0439] Output: Vehicle operation instructions, start of autonomous driving.

[0440] Step 5:

[0441] During operation, the terminal uses the Affectiva SDK to analyze the user's facial expressions and voice tone acquired from the in-vehicle camera and microphone. The emotion recognition engine recognizes the user's emotions, and based on that information, the system provides appropriate driving modes and entertainment.

[0442] Input: Facial expression data, voice tone

[0443] Output: Emotional state, driving mode instructions, entertainment suggestions

[0444] Step 6:

[0445] As an example of entertainment provision, when a user requests "play music," the device uses the Spotify API or YouTube Data API to retrieve music or videos and plays them through the in-car audio system.

[0446] Input: Voice command (entertainment request)

[0447] Output: Streaming data, entertainment playback

[0448] Step 7:

[0449] When the battery level drops, the device monitors the battery in real time and suggests the nearest charging station if necessary. Once the user approves the suggestion, the device uses the Google Maps API to calculate the optimal route to the charging facility and sends that information to the server.

[0450] Input: Battery level

[0451] Output: Suggested charging stations, optimal route information (to charging stations)

[0452] Step 8:

[0453] When the user arrives at the charging station, the terminal automatically processes the payment in conjunction with the charging station's system and begins charging. Once charging is complete, the terminal notifies the user and resumes operation.

[0454] Input: Arrival information for charging facilities

[0455] Output: Payment processing execution, charging start instruction, completion notification

[0456] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0457] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0458] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0459] [Second Embodiment]

[0460] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0461] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0462] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0463] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0464] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0465] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0466] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0467] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0468] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0469] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0470] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0471] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0472] This invention relates to a vehicle system that integrates autonomous driving and generative artificial intelligence. This system provides the optimal route based on user voice input and achieves autonomous driving using real-time updated map data and traffic information. Furthermore, it provides high levels of convenience and safety to the user through charging station guidance and payment processing, entertainment provision, and coordination with police and traffic control systems.

[0473] Program processing and function description

[0474] Speech recognition and analysis

[0475] User: When getting into the car, specify the destination to the generating AI by voice. For example, say, "I want to go to Yokohama."

[0476] Terminal: Generative artificial intelligence (generative AI) installed in the vehicle receives voice commands, and a voice recognition module analyzes them to extract the destination "Yokohama." The extracted information is then re-notified to the user by voice for confirmation.

[0477] Route calculation and information retrieval

[0478] Terminal (vehicle): Sends destination information to the server. For example, it requests a route to Yokohama.

[0479] Server: Calculates the optimal route based on the latest map data and real-time traffic information, and sends the result back to the vehicle.

[0480] Autonomous driving

[0481] Terminal (vehicle): Based on the received route information, it starts autonomous driving. During operation, it is equipped with various sensors and cameras, and the generating AI analyzes this information to dynamically correct the route.

[0482] Entertainment provider

[0483] User: Instructs the generating AI to "play music" while the vehicle is in operation.

[0484] Terminal (vehicle): Analyzes voice commands, retrieves streaming data from the user's smartphone, and plays it back on the in-vehicle audio system.

[0485] Information and payment for charging facilities

[0486] Terminal (vehicle): Monitors battery level and suggests charging facilities as needed. For example, it might notify you, "Charging is required at the next rest stop."

[0487] User: If you approve the proposal, the route to the nearest charging station will begin recalculating.

[0488] Terminal (vehicle): Upon arrival at a charging station, the system automatically processes payment and begins charging. Once all processing is complete, the user is notified and the vehicle resumes operation.

[0489] Integration with traffic control systems

[0490] Server: Connects in real time with the National Police Agency and traffic control systems to receive traffic signal timings and emergency information.

[0491] Terminal (vehicle): Controls autonomous driving based on received traffic signal information. Also, takes appropriate action in response to emergency information.

[0492] Specific example

[0493] For example, if a user is traveling from Shinjuku, Tokyo to Yokohama, the following scenarios are possible.

[0494] The user gets into the vehicle and gives voice instructions to the AI ​​generator, saying, "Take me to Yokohama, please."

[0495] The terminal (vehicle) analyzes the voice commands and requests route calculation from the server.

[0496] The server calculates the optimal route and sends it to the vehicle, and the generating AI starts autonomous driving.

[0497] When a user requests "play music" during the ride, the AI ​​generates music data from the smartphone and plays it.

[0498] The terminal (vehicle) monitors the battery level and suggests charging facilities if charging is needed.

[0499] Once the user accepts the proposal and arrives at the charging station, the generating AI automatically processes the payment and begins charging.

[0500] The server provides traffic control information, and the generating AI adjusts the route in real time.

[0501] Finally, upon arriving in Yokohama, the generating AI terminates the operation and notifies the user.

[0502] In this way, the present invention can significantly improve user convenience, safety, and efficiency.

[0503] The following describes the processing flow.

[0504] Step 1:

[0505] When a user boards the vehicle, the terminal automatically activates its AI. A passenger sensor detects the user's presence, and a Bluetooth module scans the user's smartphone to attempt a connection. Once a Bluetooth connection is established, the user is notified.

[0506] Step 2:

[0507] The user gives voice instructions to the generating AI about their destination. For example, they might say, "I want to go to Yokohama." The device uses a voice recognition module to analyze the user's voice and extract the destination "Yokohama." The generating AI then verbally notifies the user of the extracted information for confirmation.

[0508] Step 3:

[0509] The terminal sends destination information to the server. The server retrieves the latest map information from the map database, collects real-time traffic information, and calculates the optimal route. The calculation result is then sent back to the terminal.

[0510] Step 4:

[0511] The system initiates autonomous driving based on route information received by the terminal. The generating AI starts the engine and begins operation. During operation, sensors and cameras constantly monitor the surrounding environment, and the generating AI dynamically adjusts the route.

[0512] Step 5:

[0513] The user gives a command during operation, such as "Please play music." The terminal analyzes the voice command and streams music data from the user's smartphone. The audio system activates and plays the requested music.

[0514] Step 6:

[0515] The device monitors the battery level and, if it determines that charging is needed, suggests the most suitable charging location to the user. It notifies the user, "You need to charge at the next rest stop." Once the user accepts the suggestion, the device recalculates the route to the nearest charging location.

[0516] Step 7:

[0517] When the device arrives at the charging station, it automatically initiates the charging process. It processes payment in conjunction with the charging station's system and confirms that charging has started correctly. After charging is complete, it notifies the user and resumes operation.

[0518] Step 8:

[0519] The server connects to the National Police Agency and the traffic control system, receiving real-time information on traffic signal timings and emergency situations. The terminal then incorporates the received traffic control information into the generating AI, ensuring safe and efficient autonomous driving continues.

[0520] Step 9:

[0521] Upon arrival at the destination, the terminal notifies the user that they have arrived at their destination. The generating AI stops autonomous driving and turns off the engine. When the user leaves the vehicle, the passenger sensor detects this and automatically shuts down the generating AI and other systems.

[0522] (Example 1)

[0523] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0524] Modern automotive systems need to integrate many functions, including real-time autonomous driving adjustments based on traffic signals and emergency information, highly accurate route calculation, charging station guidance and automated payment processing, and consistent entertainment delivery. However, existing systems struggle to achieve these efficiently and safely. Furthermore, there is a need to ensure driving safety while maximizing user convenience.

[0525] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0526] In this invention, the server includes a generator artificial intelligence installed in the vehicle, means for recognizing and analyzing user voice commands, means for communicating with the server to obtain the latest map data and traffic information and calculate the optimal route, means for performing autonomous driving based on the calculated route information, means for obtaining and playing streaming data from the user's smartphone to provide entertainment during operation, means for monitoring the surrounding environment in real time and dynamically adjusting the route and speed, and means for coordinating with the National Police Agency and traffic control systems to receive traffic signal information and emergency information and reflect it in autonomous driving. This makes it possible to significantly improve user convenience, safety, and efficiency.

[0527] "Generative artificial intelligence" is a system that uses advanced algorithms, including natural language processing and speech recognition technologies, to analyze user instructions in real time and control autonomous driving or provide entertainment.

[0528] "Means for recognizing and analyzing user voice instructions" refers to a voice recognition module that analyzes voice acquired by microphones inside the vehicle and extracts the user's intended instructions.

[0529] A "server" refers to a computer system that receives requests from vehicles, calculates the optimal route based on map data and traffic information, and transmits that information to the vehicle.

[0530] "Up-to-date map data and traffic information" refers to road maps and traffic condition data obtained in real time from map service providers and traffic information services.

[0531] "Means for calculating the optimal route" refers to an algorithm in which a server calculates the most efficient route under specified conditions based on the vehicle's current location and destination information.

[0532] "Means of autonomous driving" refers to a system that autonomously controls a vehicle based on calculated route information to operate it to its destination.

[0533] "Means of providing entertainment" refers to a system that acquires streaming data from the user's smartphone and plays it back through the vehicle's audio system.

[0534] "A means of monitoring the surrounding environment in real time and dynamically adjusting the route and speed" refers to a function that uses sensors such as LiDAR and cameras to detect the surrounding situation, and then generates AI to analyze that information and adjust the route and speed in real time.

[0535] "Means for receiving traffic signal information and emergency information and reflecting it in autonomous driving" refers to a system that uses signal timing and emergency information obtained in cooperation with the National Police Agency and traffic control systems to reflect it in autonomous driving control.

[0536] "Means of monitoring battery level" refers to a system that constantly monitors the vehicle's battery level and notifies the user when charging is necessary.

[0537] "Means of suggesting charging facilities" refers to a system that searches for the nearest charging station and guides the user when the battery level drops.

[0538] "Method for recalculating the route to the charging facility" refers to an algorithm that, when charging is needed, causes the server to recalculate the optimal route from the current location to the nearest charging facility.

[0539] "Automatic payment processing" refers to a function that automatically completes payment using pre-registered credit card information or a digital wallet upon arrival at a charging station.

[0540] This invention relates to a system that enables autonomous driving within a vehicle and integrates multiple functions, including entertainment provision, guidance and payment processing for charging facilities, and real-time driving adjustments based on traffic signals and emergency information. This system functions with generative artificial intelligence (generative AI model) at its core.

[0541] System Configuration

[0542] hardware

[0543] Terminal: Computer and sensor devices (cameras, LiDAR, radar, microphones, etc.) installed in the vehicle.

[0544] Server: A remote computer system that interacts with map databases and traffic information APIs.

[0545] User device: A mobile device such as a smartphone or tablet.

[0546] software

[0547] Generative AI models: speech recognition modules, route calculation algorithms, entertainment playback systems, payment processing systems.

[0548] Communication protocol: A standard internet protocol used for communication (e.g., HTTP / HTTPS).

[0549] Processing in the autonomous driving system

[0550] Speech recognition and analysis

[0551] The user boards the vehicle and gives a voice command for their destination. For example: "I want to go to Yokohama." The device's microphone captures the voice, and a voice recognition module analyzes it. The analysis result is generated as text and sent to the user for confirmation.

[0552] Root calculation

[0553] The terminal sends destination information to the server, which calculates the optimal route based on the latest map data and real-time traffic information. The calculation result is sent back to the terminal, and the terminal starts autonomous driving.

[0554] Autonomous driving

[0555] The terminal controls the vehicle based on the received route information. Various sensors (camera, LiDAR, radar) monitor the surrounding environment in real time and dynamically adjust the route and speed. The generated AI analyzes this information and sends appropriate operational instructions to the vehicle control system.

[0556] Entertainment provider

[0557] During operation, the user instructs the AI ​​to "play music." The terminal analyzes the voice command, retrieves streaming data from the user's device, and plays it through the in-car audio system.

[0558] Information and payment for charging facilities

[0559] The device monitors the battery level and suggests charging locations as needed. For example, it might notify the user, "You need to charge at the next rest stop." Once the user approves the suggestion, the device recalculates the route to the charging location, automatically processes payment upon arrival, and begins charging.

[0560] Integration with traffic control systems

[0561] The server connects in real time with the National Police Agency and traffic control systems to receive traffic signal timings and emergency information. Based on this information, the terminal updates the control plan for autonomous driving, ensuring safe and efficient operation.

[0562] Specific example

[0563] For example, if a user instructs the system to "go to Yokohama," the terminal sends this information to the server. The server calculates the route and sends the result back to the terminal. Once autonomous driving begins, if the user instructs the system to "play music" during the journey, the terminal retrieves music from the user's device and plays it. It suggests charging stations as needed, and resumes operation once charging is complete. Based on traffic information, necessary route changes are reflected in real time.

[0564] Examples of prompt messages include "I want to go to Yokohama," "Please play music," and "I need to charge my phone at the next rest stop."

[0565] In this way, integrated systems can significantly improve user convenience, security, and efficiency.

[0566] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0567] Step 1:

[0568] The user boards the vehicle and gives voice instructions for their destination. The input is voice, and the output is voice data. For example, the user might say, "I want to go to Yokohama."

[0569] Step 2:

[0570] The device captures audio data through its microphone. The input is the user's voice, and the output is digitized audio data. This audio data is sent to the speech recognition module of the generating AI.

[0571] Step 3:

[0572] The terminal's voice recognition module analyzes the voice data and converts it into text information. The input is digitized voice data, and the output is text-based instruction information. Specifically, it extracts the destination "Yokohama" as text.

[0573] Step 4:

[0574] The device then re-notifies the user of the analyzed text information. The input is transcribed instruction information, and the output is an audio notification. Specifically, it plays a confirmation message saying, "Your destination is Yokohama. Is this correct?"

[0575] Step 5:

[0576] The terminal sends text information to the server and requests the optimal route based on the latest map data and traffic information. The input is destination information (text), and the output is request data. Specifically, the data is transmitted using the Internet Protocol.

[0577] Step 6:

[0578] The server retrieves the latest data from the map database and traffic information API and calculates the optimal route. The input is the request data, and the output is the optimal route information. Specifically, it uses an algorithm to calculate the route.

[0579] Step 7:

[0580] The server returns the calculation results to the terminal. The input is the calculated route information, and the output is the returned data. Specifically, it sends data including the coordinates of each point along the route, the total distance, the time required, and any points to note.

[0581] Step 8:

[0582] The terminal initiates autonomous driving based on the received route information. The input is the optimal route information, and the output is vehicle control instructions. Specifically, it sends instructions to the vehicle's control system and starts autonomous driving.

[0583] Step 9:

[0584] The terminal monitors the surrounding environment in real time using sensors mounted on the vehicle (camera, LiDAR, radar), and the generated AI analyzes this information to dynamically adjust the route and speed. The input is sensor data, and the output is vehicle control instructions. Specific operations include obstacle detection and lane keeping.

[0585] Step 10:

[0586] During operation, the user gives a voice command saying, "Please play music." The input is voice, and the output is voice data. Specifically, the user says the command, "Please play music."

[0587] Step 11:

[0588] The terminal uses a streaming service API to retrieve streaming data from the user's device and play it back on the in-car audio system. The input is a voice command, and the output is audio data. Specifically, it retrieves streaming data and starts playback.

[0589] Step 12:

[0590] The device monitors the battery level and suggests charging locations if charging is needed. The input is battery data, and the output is notification data. Specifically, it sends a notification to the user stating, "Charging is needed at the next rest stop."

[0591] Step 13:

[0592] Once the user approves the charging suggestion, the device recalculates the route to the nearest charging station. The input is the user's approval information, and the output is the recalculated route information. Specifically, it sends a recalculation request to the server.

[0593] Step 14:

[0594] Upon arrival at a charging station, the device automatically processes payment and begins charging. The input is information about arrival at the charging station, and the output is payment confirmation and charging start notification. Specifically, it completes the payment using pre-registered payment information.

[0595] Step 15:

[0596] The server connects in real time with the National Police Agency and traffic control systems to receive traffic signal timings and emergency information. Inputs are traffic signal data and emergency information, while outputs are control instruction data. Specifically, it collects traffic information and sends appropriate instructions to terminals.

[0597] Step 16:

[0598] The terminal updates the autonomous driving control plan based on received traffic signal and emergency information. Inputs are traffic signal data and emergency information, while output is the updated control instructions. Specifically, it adjusts speed in accordance with signal timing and takes appropriate action based on emergency information.

[0599] Through this series of steps, the system of the present invention can improve user convenience, safety, and efficiency.

[0600] (Application Example 1)

[0601] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0602] Conventional autonomous vehicle systems have not adequately improved user convenience and safety. Specifically, there is a demand for seamless travel, such as users specifying destinations by voice and enjoying entertainment, as well as driving that reflects real-time traffic information, but systems that can achieve this are still limited. In addition, the procedures at charging stations are cumbersome, and automation of charging is desired. Therefore, there is a need for a system that enhances overall convenience and safety in autonomous driving.

[0603] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0604] In this invention, the server includes means for recognizing and analyzing voice instructions, means for communicating with the server to obtain the latest map data and traffic information and calculate the optimal route, and means for performing autonomous driving based on the calculated route information. This enables seamless destination setting, optimal route calculation, and autonomous driving control based on the user's voice instructions. Furthermore, by including functions such as entertainment playback means, battery level monitoring and charging facility guidance, automatic payment means, and cooperation with traffic control systems, user convenience and safety can be greatly improved. In addition, high-quality entertainment can be provided by utilizing voice instruction analysis using prompts and streaming playback functions.

[0605] "Means for recognizing and analyzing voice commands" refers to devices and software that understand voice commands entered by a user, analyze their content, and translate that into specific actions.

[0606] "Means of communicating with a server to obtain the latest map data and traffic information and calculate the optimal route" refers to equipment and software that communicates with an external server to obtain map data and traffic information that are updated in real time, and calculates the optimal driving route based on that data.

[0607] "Means for performing automated driving based on calculated route information" refers to devices and programs for automatically driving a vehicle based on acquired and calculated route information.

[0608] "Means for playing entertainment content" refers to devices and software used to play entertainment (music, videos, etc.) as instructed by the user.

[0609] "A means of monitoring battery levels, suggesting charging facilities, and recalculating routes to charging facilities as needed" refers to a system that constantly monitors the vehicle's battery level, suggests the nearest charging facility when necessary, and calculates a new route to that facility.

[0610] "Means of automatically processing payment upon arrival at a charging facility" refers to equipment and software that automatically completes payment and starts the charging process upon arrival at a charging facility, without requiring manual intervention from the user.

[0611] "Means of receiving traffic signal information and emergency information in cooperation with the National Police Agency and traffic control systems and reflecting it in autonomous driving" refers to a system that receives signal information and emergency information from the National Police Agency and traffic control systems in real time and adjusts the control of autonomous driving based on that information.

[0612] "Means of acquiring information from smart devices and utilizing it while driving" refers to a system that utilizes information acquired from the user's smartphone or other smart devices while driving, and reflects it in vehicle operation and entertainment provision.

[0613] "Means for analyzing and playing voice commands via prompts" refers to devices and software that analyze prompts given by a user via voice (e.g., a command to play music) and provide entertainment based on those commands.

[0614] "Means of acquiring and playing streaming data from a smartphone" refers to equipment and software that acquire data (such as music and videos) delivered in streaming format from a user's smartphone and play it back on an in-car system.

[0615] This invention relates to a vehicle system that integrates autonomous driving and generative AI. This system provides the optimal route based on user voice input and achieves autonomous driving using real-time updated map data and traffic information. Furthermore, it offers high levels of convenience and safety to the user through features such as charging station guidance and payment processing, entertainment provision, and coordination with police and traffic control systems.

[0616] Speech recognition and analysis

[0617] The user gets into the vehicle and specifies the destination to the generating AI by voice, saying, "Please take me to Yokohama." The voice recognition module installed in the vehicle receives this voice instruction, analyzes it, and extracts the destination "Yokohama." The generating AI then verbally notifies the user of the extracted information for confirmation. The voice recognition technology used here is the Google Speech Recognition API.

[0618] Route calculation and information retrieval

[0619] The terminal (vehicle) sends destination information to the server, which calculates the optimal route based on the latest map data and real-time traffic information. The calculation result is sent back to the terminal, and the terminal starts autonomous driving. A real-time map API is used in this process.

[0620] Autonomous driving

[0621] The terminal (vehicle) begins autonomous driving based on calculated route information. During operation, various sensors and cameras are installed, and the generating AI analyzes this information to dynamically correct the route. A high-precision GPS module is used in the hardware.

[0622] Entertainment provider

[0623] When a user commands "play music" while driving, the device analyzes the voice command, retrieves streaming data from the user's smartphone, and plays it through the car's audio system. This allows users to enjoy entertainment while driving.

[0624] Information and payment for charging facilities

[0625] The terminal monitors the battery level and suggests charging stations as needed. Once the user approves the suggestion, it recalculates the route to the nearest charging station. Upon arrival at the charging station, the terminal automatically processes the payment and begins charging. Once all processes are complete, the terminal notifies the user and resumes operation.

[0626] Integration with traffic control systems

[0627] The server connects in real time with the National Police Agency and traffic control systems to receive traffic signal timings and emergency information. The terminal uses this information to control autonomous driving and take appropriate action in response to emergency information.

[0628] Specific example

[0629] If a user is traveling from Shinjuku, Tokyo to Yokohama, the following prompt will be used:

[0630] "Launch SmartDrive Assistant"

[0631] "Please take me to Yokohama."

[0632] "Tell me the nearest charging station."

[0633] "Play the music."

[0634] In this way, the present invention can significantly improve user convenience, safety, and efficiency.

[0635] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0636] Step 1:

[0637] The user gets into the vehicle and enters a voice command specifying the destination. For example, they might say, "Please take me to Yokohama." A generative AI model receives this voice command and analyzes it with a speech recognition module. The input is the user's voice command, and the output is the analyzed text data. Through this process, the destination (in this case, "Yokohama") is extracted.

[0638] Step 2:

[0639] The terminal (vehicle) sends the analyzed destination information to the server. The input is destination text data, and the output is the result of the transmission to the server. The server uses real-time map data and traffic information to calculate the optimal route. This process generates the optimal route information to the destination.

[0640] Step 3:

[0641] The server returns the calculated optimal route information to the terminal (vehicle). The input is the route information calculated on the server, and the output is the result received by the terminal. The terminal receives this route information and starts autonomous driving. Through this process, the vehicle automatically travels towards its destination.

[0642] Step 4:

[0643] When a user gives a command such as "Play music" during operation, the generative AI model analyzes this voice command. The input is the voice command, and the output is the analyzed text data. This process recognizes the user's entertainment request.

[0644] Step 5:

[0645] The device retrieves streaming data from the user's smartphone based on the analyzed entertainment instructions. The input is the text data of the entertainment instructions, and the output is the retrieved streaming data. This process makes music and videos playable.

[0646] Step 6:

[0647] The device monitors the battery level and suggests charging facilities as needed. The input is battery level data, and the output is a charging suggestion notification to the user. When the user accepts the suggestion, the route to the nearest charging facility is recalculated. This process generates new route information to the charging facility.

[0648] Step 7:

[0649] Upon arrival at a charging station, the terminal automatically processes payment and begins charging. Inputs are data indicating arrival at the charging station and payment information, while outputs are the payment processing result and a charging start notification. This process allows users to complete charging without any effort on their part.

[0650] Step 8:

[0651] The server acquires information in real time from the National Police Agency and traffic control systems and transmits it to terminals. Inputs include traffic signal information and emergency information, while output is the results of the transmission to the terminals. The terminals use this information to control autonomous driving and respond to emergencies. This process ensures safe and efficient operation.

[0652] In this way, vehicle systems can significantly improve user convenience and safety through advanced autonomous driving and entertainment provision, automated charging and payment, and real-time integration of traffic information.

[0653] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0654] This invention relates to a vehicle system that integrates autonomous driving, generative artificial intelligence, and an emotion engine. This system provides the optimal route based on user voice input and achieves autonomous driving using the latest map data and traffic information. Furthermore, it offers high levels of convenience and safety to the user by providing charging station guidance and payment processing, entertainment, coordinating with police and traffic control systems, and recognizing the user's emotions.

[0655] Program processing and function description

[0656] Speech recognition and analysis

[0657] User: When boarding the vehicle, you can use voice commands to tell the generated artificial intelligence your destination. For example, you might say, "I want to go to Yokohama."

[0658] Terminal: The voice recognition module analyzes the user's voice and extracts destination information. The extracted information is then re-notified to the user via voice for confirmation.

[0659] Route calculation and information retrieval

[0660] Terminal: Sends destination information to the server. The server retrieves the latest map information from the map database and calculates the optimal route based on real-time traffic information. The calculation result is then sent back to the vehicle.

[0661] Autonomous driving

[0662] Terminal: Based on the received route information, it initiates autonomous driving. The generating AI starts the engine and begins operation. During operation, sensors and cameras constantly monitor the surrounding environment, and the generating AI dynamically adjusts the route.

[0663] Entertainment provider

[0664] User: Instructs the generating AI to "play music" while the vehicle is in operation.

[0665] Terminal: Analyzes voice commands, retrieves streaming data from the user's smartphone, and plays it. The in-car audio system activates and plays the desired music.

[0666] Information and payment for charging facilities

[0667] Device: Monitors battery level and suggests charging locations as needed. For example, it might notify the user, "You need to charge at the next rest stop." Once the user approves the suggestion, the device recalculates the route to the nearest charging location.

[0668] Terminal: Upon arrival at a charging station, the terminal automatically initiates the charging process. It processes payment in conjunction with the charging station's system and confirms that charging is started correctly. After charging is complete, it notifies the user and resumes operation.

[0669] Integration with traffic control systems

[0670] Server: Connects to the National Police Agency and traffic control systems to receive real-time information on traffic signal timings and emergency information.

[0671] Terminal: By reflecting the received traffic control information into the generating AI, it enables safe and efficient autonomous driving to continue.

[0672] Emotion recognition and response

[0673] The device features an emotion engine that analyzes the user's facial expressions and voice tone. The emotion engine recognizes the user's emotions and provides that information to the generating AI.

[0674] Terminal: Selects the appropriate driving mode based on the user's emotions. For example, if the user is feeling stressed, it will select a relaxing driving mode.

[0675] Terminal: Evaluates the user's stress level and fatigue, and suggests breaks as needed. Based on the evaluation results, it recalculates the route to rest points and suggests it to the user.

[0676] Device: Selects and suggests entertainment based on the user's mood. For example, if the user is in a good mood, it suggests rhythmic music. Conversely, if the user is sad, it suggests relaxing music.

[0677] Specific example

[0678] For example, if a user is traveling from Shinjuku, Tokyo to Yokohama, the following scenario will be executed.

[0679] The user gets into the vehicle and gives voice instructions to the AI ​​generator, saying, "Please take me to Yokohama."

[0680] The terminal analyzes the voice command and requests route calculation from the server.

[0681] The server calculates the optimal route, and the generating AI starts the automated driving process.

[0682] When a user requests "play music" during the ride, the AI ​​generates music data from the smartphone and plays it.

[0683] The device monitors the battery level and suggests charging facilities if charging is needed.

[0684] Once the user accepts the proposal and arrives at the charging station, the generating AI automatically processes the payment and starts charging.

[0685] The server provides traffic control information, and the generating AI adjusts the route in real time.

[0686] The device analyzes the user's facial expressions and voice, and if it determines that the user is tired, it suggests taking a break and guides them to an appropriate resting place.

[0687] Upon arrival in Yokohama, the AI ​​generates the data and terminates the operation, notifying the user.

[0688] In this way, the present invention can significantly improve user convenience, safety, and efficiency.

[0689] The following describes the processing flow.

[0690] Step 1:

[0691] When a user boards the vehicle, the terminal automatically activates its AI. A passenger sensor detects the user's presence, and a Bluetooth module scans the user's smartphone to attempt a connection. Once a Bluetooth connection is established, the user is notified.

[0692] Step 2:

[0693] The user gives voice instructions to the AI ​​generating the destination. For example, they might say, "I want to go to Yokohama." The device uses a voice recognition module to analyze the user's voice and extract the destination "Yokohama." The extracted information is then re-notified to the user via voice for confirmation.

[0694] Step 3:

[0695] The terminal sends destination information to the server. The server retrieves the latest map information from the map database, collects real-time traffic information, and calculates the optimal route. The calculation result is then sent back to the terminal.

[0696] Step 4:

[0697] The system initiates autonomous driving based on route information received by the terminal. The generating AI starts the engine and begins operation. During operation, various sensors and cameras constantly monitor the surrounding environment, and the generating AI dynamically adjusts the route.

[0698] Step 5:

[0699] The user gives a voice command, such as "I want to play music," while driving. The terminal analyzes the voice command and streams music data from the user's smartphone. The in-car audio system activates and plays the requested music.

[0700] Step 6:

[0701] The device monitors the battery level and, if it determines that charging is needed, suggests the most suitable charging location to the user. It notifies the user, "You need to charge at the next rest stop." Once the user accepts the suggestion, the device recalculates the route to the nearest charging location.

[0702] Step 7:

[0703] When the device arrives at the charging station, it automatically initiates the charging process. It processes payment in conjunction with the charging station's system and confirms that charging has started correctly. After charging is complete, it notifies the user and resumes operation.

[0704] Step 8:

[0705] The server connects to the National Police Agency and the traffic control system, receiving real-time information on traffic signal timings and emergency situations. The terminal then incorporates the received traffic control information into the generating AI, ensuring safe and efficient autonomous driving continues.

[0706] Step 9:

[0707] The device uses an emotion engine that analyzes the user's facial expressions and voice tone to recognize their emotions. If the user is stressed, the emotion engine provides that information to the generating AI, which then selects a relaxing driving mode.

[0708] Step 10:

[0709] The system assesses the user's stress level and fatigue, and suggests breaks as needed. Once the user approves the suggestion, the device recalculates the route to an appropriate rest stop and notifies the user.

[0710] Step 11:

[0711] The device selects entertainment based on the user's mood. For example, if the user is in a good mood, it suggests rhythmic music; conversely, if the user is sad, it suggests relaxing music.

[0712] Step 12:

[0713] Upon arrival at the destination, the terminal notifies the user that they have arrived at their destination. The generating AI stops autonomous driving and turns off the engine. When the user leaves the vehicle, the passenger sensor detects this and automatically shuts down the generating AI and other systems.

[0714] (Example 2)

[0715] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0716] Many autonomous driving systems currently on the market can recognize user voice commands, calculate routes, and drive autonomously, but they lack the ability to recognize user emotions and adjust driving modes accordingly. Furthermore, there is a need for a variety of functions to enhance user convenience, such as charging station suggestions and automatic payment, real-time traffic signal information, and entertainment features. In addition, the lack of technology to integrate and execute these functions makes improving the user experience a challenge.

[0717] The identification processing performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server has a generating artificial intelligence installed in the vehicle and includes means for recognizing and analyzing the user's voice instructions, means for communicating with the server to acquire the latest map data and traffic information and calculate the optimal route, means for performing automatic driving based on the calculated route information, means for analyzing the user's facial expressions and voice tone to recognize emotions and providing that information to the generating artificial intelligence, and means for selecting a driving mode based on the user's emotions. This makes it possible to adjust driving according to the user's emotions, significantly improving driving safety and comfort. Furthermore, by integrating and executing multiple functions such as suggesting charging facilities and automatic payment, reflecting real-time traffic signal information, and providing entertainment, user convenience and the quality of the experience can be enhanced.

[0718] "Generative artificial intelligence installed in a vehicle" refers to artificial intelligence installed in the vehicle's computer system that recognizes and analyzes user voice commands and performs autonomous driving and other vehicle operation management.

[0719] "User voice commands" refers to the act of a user giving instructions by voice regarding vehicle operation, setting a destination, etc.

[0720] "Means for recognizing and analyzing voice commands" refers to the collective term for hardware and software that capture, analyze, and understand user voice commands.

[0721] A "server" refers to a remote computer system that receives data transmitted from a vehicle, processes it, and sends the results back to the vehicle.

[0722] "Latest map data" refers to data that includes road and topographic information updated in real time using a geographic information system.

[0723] "Traffic information" refers to information about current road conditions, including dynamic data such as congestion, traffic accidents, and road construction.

[0724] "Means for calculating the optimal route" refers to algorithms and software used to calculate the most efficient and safe route to the user's destination.

[0725] "Means of autonomous driving" refers to an integrated system of hardware and software that enables autonomous operation of a vehicle.

[0726] "Means of recognizing emotions by analyzing a user's facial expressions and voice tone" refers to hardware and software that analyze the characteristics of a user's face and voice to determine their emotional state.

[0727] "Means for selecting a driving mode based on user emotions" refers to a control system that selects the optimal driving mode according to the recognized emotions of the user.

[0728] "Means of monitoring battery level" refers to sensors and software that continuously monitor the vehicle's battery status and provide necessary information.

[0729] "A means of suggesting charging facilities and recalculating the route to those facilities when charging is needed" refers to technology that, when the battery level drops, suggests the nearest charging facility to the user and calculates and suggests the optimal route to that facility.

[0730] "A means of automatically processing payment upon arrival at a charging facility" refers to a system that automatically completes payment when charging begins at a charging station.

[0731] "Means for recognizing music and video playback commands and acquiring streaming data from the user's smartphone for playback" refers to technology that recognizes a user's command to play music or videos, acquires the necessary data from the smartphone, and plays the content.

[0732] "Means of receiving traffic signal information and emergency information in cooperation with the National Police Agency and traffic control systems and reflecting it in autonomous driving" refers to technology that exchanges data in real time with the police and traffic control centers, receives information on changes in traffic signals and emergencies, and applies it to autonomous driving systems.

[0733] This invention relates to a vehicle system that integrates autonomous driving, generative artificial intelligence, and an emotion engine. This system provides the optimal route based on the user's voice input and achieves autonomous driving using the latest map data and traffic information. It also provides a high level of convenience and safety for the user by guiding them to charging stations and processing payments, providing entertainment, coordinating with police and traffic control systems, and recognizing user emotions.

[0734] Speech recognition and analysis

[0735] The user boards the vehicle and gives voice commands such as, "Please take me to Yokohama."

[0736] The terminal (in-vehicle system) captures audio using a microphone mounted on the vehicle and sends it to a speech recognition module (e.g., Google Cloud Speech-to-Text).

[0737] The device uses a speech recognition module to convert the voice data into text and extracts the destination information, "Yokohama." This information is then confirmed by the user via voice.

[0738] Route calculation and traffic information acquisition

[0739] The device sends the confirmed destination information to the server. It executes an API request to the server's URL and sends request data including the destination.

[0740] The server retrieves the latest map information and current traffic information from a map database (e.g., Google Maps API).

[0741] The server calculates the optimal route based on the information it has acquired, and this calculation uses real-time traffic data. The calculation result is returned to the terminal as an API response.

[0742] Autonomous driving

[0743] The terminal passes the route information it receives to a generating AI (e.g., Tesla's AutoPilot).

[0744] The terminal uses generated AI to start the engine and begin autonomous driving. During operation, it uses a 100-directional sensor and cameras to monitor the surrounding environment in real time.

[0745] The terminal uses generated AI to analyze data from sensors and cameras, and adjusts the route and driving actions as needed.

[0746] Entertainment provider

[0747] The user can give a voice command to "play music" while the vehicle is in operation.

[0748] The device analyzes the voice command and uses the API of a music streaming service (e.g., Spotify) to retrieve music data from the smartphone.

[0749] The device activates the in-car audio system and plays the acquired music data.

[0750] Information on charging facilities and payment processing

[0751] The device constantly monitors the battery level, and when the battery level drops below 20%, it notifies the user that "charging is needed at the next rest stop."

[0752] The user approves the notification.

[0753] The device sends another API request to the server to search for the nearest charging station.

[0754] The server provides information on the nearest charging station, recalculates the route, and sends it back to the terminal.

[0755] Upon arrival at the charging station, the system automatically initiates the charging process and handles payment in conjunction with the charging station's system. Once charging is complete, the user is notified by voice, and the vehicle resumes operation.

[0756] Integration with traffic control systems

[0757] The server connects in real time with the National Police Agency and traffic control systems to receive traffic signal timings and emergency information.

[0758] The server sends the received information to the terminal.

[0759] The terminal uses AI generation to dynamically reflect traffic control information, ensuring safe operation continues.

[0760] Emotion recognition and adaptive responses

[0761] The device is equipped with an emotion engine (e.g., Microsoft Azure Cognitive Services) that analyzes the user's facial expressions and voice tone.

[0762] The device analyzes the user's emotions from voice and camera data and recognizes that they are experiencing stress.

[0763] The device uses AI generation to select a relaxing driving mode based on the user's emotional information.

[0764] If the device detects that the user is stressed, it will suggest and play relaxing music. It will also suggest taking a break if it detects fatigue.

[0765] If the user agrees to the suggestion, the device calculates and guides them to the optimal resting place.

[0766] Specific examples and examples of prompt sentences for generative AI models

[0767] For example, if a user is traveling from Shinjuku, Tokyo to Yokohama:

[0768] The user gives a voice command saying, "Please take me to Yokohama."

[0769] The terminal analyzes the voice command and requests route calculation from the server.

[0770] The server calculates the optimal route and sends it back to the terminal.

[0771] The device initiates autonomous driving, monitoring the surroundings in real time while driving.

[0772] When the user requests that the device play music, the device will play music from the smartphone.

[0773] The device monitors the battery level and suggests charging facilities as needed. If the user agrees, it will guide them to the charging facility and automatically process payment and begin charging upon arrival.

[0774] The server transmits traffic control information to the terminal, which then adjusts its driving to ensure safety.

[0775] The device analyzes the user's emotions and, if necessary, provides guidance on relaxing driving modes or taking breaks.

[0776] Specific examples of prompt statements for generative AI models:

[0777] "Calculate the optimal route to our destination, Yokohama, and ensure safe and efficient driving."

[0778] "Please explain the procedure for analyzing user voice commands and providing entertainment."

[0779] "If the current battery level is below 20%, please explain how to suggest the nearest charging station and process the payment automatically."

[0780] "Please explain the process for recognizing user emotions and making suggestions to reduce stress."

[0781] The specific embodiments of the present invention have now been described. This embodiment significantly improves user safety and convenience.

[0782] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0783] Step 1: Receiving and analyzing voice input

[0784] Input: The user gives a voice command inside the vehicle saying, "Please take me to Yokohama."

[0785] Terminal: A microphone inside the vehicle captures the user's voice and sends the audio data to a speech recognition module (e.g., Google Cloud Speech-to-Text). The speech recognition module analyzes the audio data and extracts destination information such as "Yokohama".

[0786] Output: Destination information is returned to the terminal as text data.

[0787] Operation: The device confirms the voice analysis results with the user via voice. It asks the user, "Is Yokohama your destination?"

[0788] Step 2: Route calculation and acquisition of traffic information

[0789] Input: The terminal is given information about the destination, "Yokohama".

[0790] Terminal: Sends destination information to the server. The data, including the destination, is sent to the server as an API request.

[0791] Server: Retrieves the latest map information and real-time traffic information from a map database (e.g., Google Maps API). This information is used to calculate the optimal route.

[0792] Output: The calculation result is sent back to the terminal as route information.

[0793] Operation: The terminal receives route information calculated by the server and uses it in subsequent steps.

[0794] Step 3: Start autonomous driving

[0795] Input: Route information is provided to the terminal.

[0796] Terminal: Passes the received route information to the generating AI (e.g., Tesla's AutoPilot).

[0797] Terminal: The generating AI starts the engine and begins autonomous driving. It monitors the surrounding environment in real time using a 100-directional sensor and camera.

[0798] Output: The start state for autonomous driving mode is generated.

[0799] Operation: The generating AI analyzes data from sensors and cameras and adjusts the route and driving actions in real time.

[0800] Step 4: Providing Entertainment

[0801] Input: The user gives a voice command saying, "Please play music."

[0802] Terminal: Analyzes voice commands and retrieves music data from the user's smartphone using the API of a music streaming service (e.g., Spotify).

[0803] Terminal: Activates the in-car audio system and plays the acquired music data.

[0804] Output: Music playback begins.

[0805] Operation: The device streams music data based on voice commands and plays it on the audio system.

[0806] Step 5: Guidance to charging facilities and payment processing

[0807] Input: The device monitors the battery level, and when the battery level falls below 20%, it triggers a response.

[0808] Device: Notifies the user that "charging is required at the next rest stop."

[0809] User: Accept notification.

[0810] Terminal: Sends another API request to the server to search for the nearest charging station.

[0811] Server: Provides information on the nearest charging station, recalculates the route, and sends it back to the terminal.

[0812] Terminal: Upon arrival at the charging station, it automatically initiates the charging process and handles payment processing in conjunction with the charging station's system.

[0813] Output: Charging and payment processing are complete.

[0814] Operation: The device starts the charging process, notifies the user when it is complete, and resumes operation.

[0815] Step 6: Integration with traffic control systems

[0816] Input: The server receives real-time traffic signal information and emergency information from the National Police Agency and traffic control systems.

[0817] Server: Sends received information to the terminal.

[0818] Terminal: The generating AI dynamically reflects traffic control information, ensuring safe operation continues.

[0819] Output: The operational status is updated based on traffic control information.

[0820] Operation: The generating AI reflects traffic signal information in real time and adjusts driving accordingly.

[0821] Step 7: Emotion Recognition and Adaptive Response

[0822] Input: The user's facial expressions and voice tone are captured by the device.

[0823] Terminal: An emotion engine (e.g., Microsoft Azure Cognitive Services) analyzes the user's facial expressions and voice tone to recognize emotions.

[0824] Terminal: Provides emotional information to the AI ​​that generates it.

[0825] Generative AI: Selects a relaxing driving mode based on the user's emotions. Suggests and plays relaxing music as needed.

[0826] Output: The system initiates driving modes and provides entertainment tailored to the user's mood.

[0827] Operation: The generated AI selects a driving mode based on the user's emotional information and adjusts the driving to promote relaxation. It suggests breaks as needed and guides the user to the optimal rest stop.

[0828] (Application Example 2)

[0829] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0830] In modern autonomous vehicles, it is crucial to accurately recognize user voice commands and provide optimal routes using up-to-date map data and traffic information. However, in addition to this, it is also necessary to handle charging station guidance and payment processing, appropriate driving modes and entertainment based on user emotion recognition, and the reflection of real-time traffic control information. Conventional systems lack these comprehensive functions, resulting in challenges in ensuring sufficient user convenience and safety.

[0831] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a generating artificial intelligence installed in the vehicle and means for recognizing and analyzing the user's voice instructions, means for communicating with the server to acquire the latest map data and traffic information and calculate the optimal route, means for performing autonomous driving based on the calculated route information, means for having an emotion recognition engine to recognize emotions from the user's facial expressions and voice tone and provide driving modes and entertainment based on that, and means for guiding users to charging facilities and processing payments. This makes it possible to significantly improve user convenience and safety by providing the optimal route based on the user's voice instructions, recognizing and adapting to the situation during autonomous driving, guiding users to charging stations and processing payments, and providing entertainment that responds to emotions.

[0832] "Generative artificial intelligence" is an artificial intelligence that recognizes and analyzes user voice commands and generates the information necessary for decision-making.

[0833] "Map data" refers to data that represents geographical information such as roads, buildings, and topography in a digital format.

[0834] "Traffic information" refers to data that shows the current traffic conditions, including real-time information such as traffic congestion and accidents.

[0835] The "optimal route" is the most efficient path between a designated starting point and a destination, taking into account factors such as travel time, distance, and traffic conditions.

[0836] "Autonomous driving" is a technology that allows vehicles to move autonomously without driver intervention.

[0837] An "emotion recognition engine" is a technology that analyzes a user's facial expressions and voice tone to identify their emotional state.

[0838] "Entertainment provision" refers to the function of playing music, videos, and other entertainment content for users.

[0839] The "Charging Station Guide" is a function that displays the nearest suitable charging station to the user based on the vehicle's battery level.

[0840] "Payment processing" is the process by which the system pays for the services or goods provided on behalf of the user.

[0841] This invention relates to a system for autonomous driving vehicles that integrates generative artificial intelligence, an emotion recognition engine, real-time traffic information, and entertainment provision functions.

[0842] First, when a user gets into the vehicle, the AI ​​recognizes and analyzes the user's voice instructions. For example, if the user says, "I want to go to Yokohama," the voice is converted to text using the Google Cloud Speech-to-Text API, and "Yokohama" as the destination is extracted.

[0843] Next, this destination information is sent to a server, which uses the Google Maps API to calculate the optimal route based on the latest map data and real-time traffic information. The calculation results are sent back to the vehicle, and autonomous driving begins. Autonomous driving is performed using each vehicle manufacturer's autonomous driving API.

[0844] While driving, the generated AI uses an emotion recognition engine powered by the Affectiva SDK to analyze the user's facial expressions and voice tone, identifying the user's emotional state. For example, if the user is tired, the system will suggest a rest stop. This improves user safety and comfort.

[0845] Furthermore, when the battery level drops, the system automatically suggests a charging station. Once the user approves, it uses the Google Maps API to recalculate the optimal route to the charging station, and payment is automatically processed upon arrival.

[0846] In providing entertainment, when a user requests "play music," the AI ​​generates music and videos using the Spotify API or YouTube Data API and plays them through the in-car audio system.

[0847] Furthermore, the system collaborates with the National Police Agency and traffic control systems to acquire traffic signal and emergency information in real time, and adjusts autonomous driving based on this information. This enables safe operation that is in line with traffic conditions.

[0848] Finally, the following are concrete examples and examples of prompt messages.

[0849] Specific example:

[0850] For example, if a user is traveling from Shinjuku, Tokyo to Yokohama, the following scenario will be executed: The user gets into the vehicle and gives a voice command to the generating AI, "Take me to Yokohama, please." The voice is then converted to text using the Google Cloud Speech-to-Text API. Next, the optimal route is calculated using the Google Maps API, and autonomous driving begins. During the journey, if the user requests, "Please play music," the generating AI will play music using the Spotify API. If the battery level drops, the nearest charging station will be indicated, and payment will be processed automatically. In addition, the route is adjusted in real time based on traffic control information.

[0851] Example of a prompt:

[0852] When a user says, "Take me to Yokohama," the voice recognition module analyzes the voice, calculates the optimal route using the Google Maps API, and starts autonomous driving. If the user says, "Play music" during the ride, music will be played using the Spotify API. If the battery is low, the nearest charging station will be guided, and payment processing will be initiated. The emotion recognition engine analyzes the user's emotions and provides entertainment suggestions based on that.

[0853] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0854] Step 1:

[0855] The user boards the vehicle and gives voice commands to the generating artificial intelligence. For example, they might say, "I want to go to Yokohama." This voice input is acquired through the microphone inside the vehicle.

[0856] Input: Voice Instructions

[0857] Output: Audio data

[0858] Step 2:

[0859] The device uses the Google Cloud Speech-to-Text API to convert the user's voice commands into text. The destination is extracted based on the converted text data. This analysis result is recorded in a database.

[0860] Input: Audio data

[0861] Output: Text data (e.g., "I want to go to Yokohama")

[0862] Step 3:

[0863] The device sends the acquired text data to the server. The server uses the Google Maps API to obtain the latest map data and real-time traffic information based on the destination, and calculates the optimal route based on that data.

[0864] Input: Text data (destination information)

[0865] Output: Map data, traffic information, optimal route

[0866] Step 4:

[0867] The server sends the calculated optimal route information to a terminal inside the vehicle. Based on the received route information, the terminal issues instructions to the autonomous driving system and starts autonomous driving. During autonomous driving, sensor data (cameras, LiDAR, etc.) is constantly monitored, and the generating AI dynamically adjusts the route.

[0868] Input: Optimal route information

[0869] Output: Vehicle operation instructions, start of autonomous driving.

[0870] Step 5:

[0871] During operation, the terminal uses the Affectiva SDK to analyze the user's facial expressions and voice tone acquired from the in-vehicle camera and microphone. The emotion recognition engine recognizes the user's emotions, and based on that information, the system provides appropriate driving modes and entertainment.

[0872] Input: Facial expression data, voice tone

[0873] Output: Emotional state, driving mode instructions, entertainment suggestions

[0874] Step 6:

[0875] As an example of entertainment provision, when a user requests "play music," the device uses the Spotify API or YouTube Data API to retrieve music or videos and plays them through the in-car audio system.

[0876] Input: Voice command (entertainment request)

[0877] Output: Streaming data, entertainment playback

[0878] Step 7:

[0879] When the battery level drops, the device monitors the battery in real time and suggests the nearest charging station if necessary. Once the user approves the suggestion, the device uses the Google Maps API to calculate the optimal route to the charging facility and sends that information to the server.

[0880] Input: Battery level

[0881] Output: Suggested charging stations, optimal route information (to charging stations)

[0882] Step 8:

[0883] When the user arrives at the charging station, the terminal automatically processes the payment in conjunction with the charging station's system and begins charging. Once charging is complete, the terminal notifies the user and resumes operation.

[0884] Input: Arrival information for charging facilities

[0885] Output: Payment processing execution, charging start instruction, completion notification

[0886] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0887] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0888] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0889] [Third Embodiment]

[0890] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0891] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0892] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0893] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0894] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0895] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0896] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0897] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0898] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0899] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0900] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0901] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0902] This invention relates to a vehicle system that integrates autonomous driving and generative artificial intelligence. This system provides the optimal route based on user voice input and achieves autonomous driving using real-time updated map data and traffic information. Furthermore, it provides high levels of convenience and safety to the user through charging station guidance and payment processing, entertainment provision, and coordination with police and traffic control systems.

[0903] Program processing and function description

[0904] Speech recognition and analysis

[0905] User: When getting into the car, specify the destination to the generating AI by voice. For example, say, "I want to go to Yokohama."

[0906] Terminal: Generative artificial intelligence (generative AI) installed in the vehicle receives voice commands, and a voice recognition module analyzes them to extract the destination "Yokohama." The extracted information is then re-notified to the user by voice for confirmation.

[0907] Route calculation and information retrieval

[0908] Terminal (vehicle): Sends destination information to the server. For example, it requests a route to Yokohama.

[0909] Server: Calculates the optimal route based on the latest map data and real-time traffic information, and sends the result back to the vehicle.

[0910] Autonomous driving

[0911] Terminal (vehicle): Based on the received route information, it starts autonomous driving. During operation, it is equipped with various sensors and cameras, and the generating AI analyzes this information to dynamically correct the route.

[0912] Entertainment provider

[0913] User: Instructs the generating AI to "play music" while the vehicle is in operation.

[0914] Terminal (vehicle): Analyzes voice commands, retrieves streaming data from the user's smartphone, and plays it back on the in-vehicle audio system.

[0915] Information and payment for charging facilities

[0916] Terminal (vehicle): Monitors battery level and suggests charging facilities as needed. For example, it might notify you, "Charging is required at the next rest stop."

[0917] User: If you approve the proposal, the route to the nearest charging station will begin recalculating.

[0918] Terminal (vehicle): Upon arrival at a charging station, the system automatically processes payment and begins charging. Once all processing is complete, the user is notified and the vehicle resumes operation.

[0919] Integration with traffic control systems

[0920] Server: Connects in real time with the National Police Agency and traffic control systems to receive traffic signal timings and emergency information.

[0921] Terminal (vehicle): Controls autonomous driving based on received traffic signal information. Also, takes appropriate action in response to emergency information.

[0922] Specific example

[0923] For example, if a user is traveling from Shinjuku, Tokyo to Yokohama, the following scenarios are possible.

[0924] The user gets into the vehicle and gives voice instructions to the AI ​​generator, saying, "Take me to Yokohama, please."

[0925] The terminal (vehicle) analyzes the voice commands and requests route calculation from the server.

[0926] The server calculates the optimal route and sends it to the vehicle, and the generating AI starts autonomous driving.

[0927] When a user requests "play music" during the ride, the AI ​​generates music data from the smartphone and plays it.

[0928] The terminal (vehicle) monitors the battery level and suggests charging facilities if charging is needed.

[0929] Once the user accepts the proposal and arrives at the charging station, the generating AI automatically processes the payment and begins charging.

[0930] The server provides traffic control information, and the generating AI adjusts the route in real time.

[0931] Finally, upon arriving in Yokohama, the generating AI terminates the operation and notifies the user.

[0932] In this way, the present invention can significantly improve user convenience, safety, and efficiency.

[0933] The following describes the processing flow.

[0934] Step 1:

[0935] When a user boards the vehicle, the terminal automatically activates its AI. A passenger sensor detects the user's presence, and a Bluetooth module scans the user's smartphone to attempt a connection. Once a Bluetooth connection is established, the user is notified.

[0936] Step 2:

[0937] The user gives voice instructions to the generating AI about their destination. For example, they might say, "I want to go to Yokohama." The device uses a voice recognition module to analyze the user's voice and extract the destination "Yokohama." The generating AI then verbally notifies the user of the extracted information for confirmation.

[0938] Step 3:

[0939] The terminal sends destination information to the server. The server retrieves the latest map information from the map database, collects real-time traffic information, and calculates the optimal route. The calculation result is then sent back to the terminal.

[0940] Step 4:

[0941] The system initiates autonomous driving based on route information received by the terminal. The generating AI starts the engine and begins operation. During operation, sensors and cameras constantly monitor the surrounding environment, and the generating AI dynamically adjusts the route.

[0942] Step 5:

[0943] The user gives a command during operation, such as "Please play music." The terminal analyzes the voice command and streams music data from the user's smartphone. The audio system activates and plays the requested music.

[0944] Step 6:

[0945] The device monitors the battery level and, if it determines that charging is needed, suggests the most suitable charging location to the user. It notifies the user, "You need to charge at the next rest stop." Once the user accepts the suggestion, the device recalculates the route to the nearest charging location.

[0946] Step 7:

[0947] When the device arrives at the charging station, it automatically initiates the charging process. It processes payment in conjunction with the charging station's system and confirms that charging has started correctly. After charging is complete, it notifies the user and resumes operation.

[0948] Step 8:

[0949] The server connects to the National Police Agency and the traffic control system, receiving real-time information on traffic signal timings and emergency situations. The terminal then incorporates the received traffic control information into the generating AI, ensuring safe and efficient autonomous driving continues.

[0950] Step 9:

[0951] Upon arrival at the destination, the terminal notifies the user that they have arrived at their destination. The generating AI stops autonomous driving and turns off the engine. When the user leaves the vehicle, the passenger sensor detects this and automatically shuts down the generating AI and other systems.

[0952] (Example 1)

[0953] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0954] Modern automotive systems need to integrate many functions, including real-time autonomous driving adjustments based on traffic signals and emergency information, highly accurate route calculation, charging station guidance and automated payment processing, and consistent entertainment delivery. However, existing systems struggle to achieve these efficiently and safely. Furthermore, there is a need to ensure driving safety while maximizing user convenience.

[0955] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0956] In this invention, the server includes a generator artificial intelligence installed in the vehicle, means for recognizing and analyzing user voice commands, means for communicating with the server to obtain the latest map data and traffic information and calculate the optimal route, means for performing autonomous driving based on the calculated route information, means for obtaining and playing streaming data from the user's smartphone to provide entertainment during operation, means for monitoring the surrounding environment in real time and dynamically adjusting the route and speed, and means for coordinating with the National Police Agency and traffic control systems to receive traffic signal information and emergency information and reflect it in autonomous driving. This makes it possible to significantly improve user convenience, safety, and efficiency.

[0957] "Generative artificial intelligence" is a system that uses advanced algorithms, including natural language processing and speech recognition technologies, to analyze user instructions in real time and control autonomous driving or provide entertainment.

[0958] "Means for recognizing and analyzing user voice instructions" refers to a voice recognition module that analyzes voice acquired by microphones inside the vehicle and extracts the user's intended instructions.

[0959] A "server" refers to a computer system that receives requests from vehicles, calculates the optimal route based on map data and traffic information, and transmits that information to the vehicle.

[0960] "Up-to-date map data and traffic information" refers to road maps and traffic condition data obtained in real time from map service providers and traffic information services.

[0961] "Means for calculating the optimal route" refers to an algorithm in which a server calculates the most efficient route under specified conditions based on the vehicle's current location and destination information.

[0962] "Means of autonomous driving" refers to a system that autonomously controls a vehicle based on calculated route information to operate it to its destination.

[0963] "Means of providing entertainment" refers to a system that acquires streaming data from the user's smartphone and plays it back through the vehicle's audio system.

[0964] "A means of monitoring the surrounding environment in real time and dynamically adjusting the route and speed" refers to a function that uses sensors such as LiDAR and cameras to detect the surrounding situation, and then generates AI to analyze that information and adjust the route and speed in real time.

[0965] "Means for receiving traffic signal information and emergency information and reflecting it in autonomous driving" refers to a system that uses signal timing and emergency information obtained in cooperation with the National Police Agency and traffic control systems to reflect it in autonomous driving control.

[0966] "Means of monitoring battery level" refers to a system that constantly monitors the vehicle's battery level and notifies the user when charging is necessary.

[0967] "Means of suggesting charging facilities" refers to a system that searches for the nearest charging station and guides the user when the battery level drops.

[0968] "Method for recalculating the route to the charging facility" refers to an algorithm that, when charging is needed, causes the server to recalculate the optimal route from the current location to the nearest charging facility.

[0969] "Automatic payment processing" refers to a function that automatically completes payment using pre-registered credit card information or a digital wallet upon arrival at a charging station.

[0970] This invention relates to a system that enables autonomous driving within a vehicle and integrates multiple functions, including entertainment provision, guidance and payment processing for charging facilities, and real-time driving adjustments based on traffic signals and emergency information. This system functions with generative artificial intelligence (generative AI model) at its core.

[0971] System Configuration

[0972] hardware

[0973] Terminal: Computer and sensor devices (cameras, LiDAR, radar, microphones, etc.) installed in the vehicle.

[0974] Server: A remote computer system that interacts with map databases and traffic information APIs.

[0975] User device: A mobile device such as a smartphone or tablet.

[0976] software

[0977] Generative AI models: speech recognition modules, route calculation algorithms, entertainment playback systems, payment processing systems.

[0978] Communication protocol: A standard internet protocol used for communication (e.g., HTTP / HTTPS).

[0979] Processing in the autonomous driving system

[0980] Speech recognition and analysis

[0981] The user boards the vehicle and gives a voice command for their destination. For example: "I want to go to Yokohama." The device's microphone captures the voice, and a voice recognition module analyzes it. The analysis result is generated as text and sent to the user for confirmation.

[0982] Root calculation

[0983] The terminal sends destination information to the server, which calculates the optimal route based on the latest map data and real-time traffic information. The calculation result is sent back to the terminal, and the terminal starts autonomous driving.

[0984] Autonomous driving

[0985] The terminal controls the vehicle based on the received route information. Various sensors (camera, LiDAR, radar) monitor the surrounding environment in real time and dynamically adjust the route and speed. The generated AI analyzes this information and sends appropriate operational instructions to the vehicle control system.

[0986] Entertainment provider

[0987] During operation, the user instructs the AI ​​to "play music." The terminal analyzes the voice command, retrieves streaming data from the user's device, and plays it through the in-car audio system.

[0988] Information and payment for charging facilities

[0989] The device monitors the battery level and suggests charging locations as needed. For example, it might notify the user, "You need to charge at the next rest stop." Once the user approves the suggestion, the device recalculates the route to the charging location, automatically processes payment upon arrival, and begins charging.

[0990] Integration with traffic control systems

[0991] The server connects in real time with the National Police Agency and traffic control systems to receive traffic signal timings and emergency information. Based on this information, the terminal updates the control plan for autonomous driving, ensuring safe and efficient operation.

[0992] Specific example

[0993] For example, if a user instructs the system to "go to Yokohama," the terminal sends this information to the server. The server calculates the route and sends the result back to the terminal. Once autonomous driving begins, if the user instructs the system to "play music" during the journey, the terminal retrieves music from the user's device and plays it. It suggests charging stations as needed, and resumes operation once charging is complete. Based on traffic information, necessary route changes are reflected in real time.

[0994] Examples of prompt messages include "I want to go to Yokohama," "Please play music," and "I need to charge my phone at the next rest stop."

[0995] In this way, integrated systems can significantly improve user convenience, security, and efficiency.

[0996] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0997] Step 1:

[0998] The user boards the vehicle and gives voice instructions for their destination. The input is voice, and the output is voice data. For example, the user might say, "I want to go to Yokohama."

[0999] Step 2:

[1000] The device captures audio data through its microphone. The input is the user's voice, and the output is digitized audio data. This audio data is sent to the speech recognition module of the generating AI.

[1001] Step 3:

[1002] The terminal's voice recognition module analyzes the voice data and converts it into text information. The input is digitized voice data, and the output is text-based instruction information. Specifically, it extracts the destination "Yokohama" as text.

[1003] Step 4:

[1004] The device then re-notifies the user of the analyzed text information. The input is transcribed instruction information, and the output is an audio notification. Specifically, it plays a confirmation message saying, "Your destination is Yokohama. Is this correct?"

[1005] Step 5:

[1006] The terminal sends text information to the server and requests the optimal route based on the latest map data and traffic information. The input is destination information (text), and the output is request data. Specifically, the data is transmitted using the Internet Protocol.

[1007] Step 6:

[1008] The server retrieves the latest data from the map database and traffic information API and calculates the optimal route. The input is the request data, and the output is the optimal route information. Specifically, it uses an algorithm to calculate the route.

[1009] Step 7:

[1010] The server returns the calculation results to the terminal. The input is the calculated route information, and the output is the returned data. Specifically, it sends data including the coordinates of each point along the route, the total distance, the time required, and any points to note.

[1011] Step 8:

[1012] The terminal initiates autonomous driving based on the received route information. The input is the optimal route information, and the output is vehicle control instructions. Specifically, it sends instructions to the vehicle's control system and starts autonomous driving.

[1013] Step 9:

[1014] The terminal monitors the surrounding environment in real time using sensors mounted on the vehicle (camera, LiDAR, radar), and the generated AI analyzes this information to dynamically adjust the route and speed. The input is sensor data, and the output is vehicle control instructions. Specific operations include obstacle detection and lane keeping.

[1015] Step 10:

[1016] During operation, the user gives a voice command saying, "Please play music." The input is voice, and the output is voice data. Specifically, the user says the command, "Please play music."

[1017] Step 11:

[1018] The terminal uses a streaming service API to retrieve streaming data from the user's device and play it back on the in-car audio system. The input is a voice command, and the output is audio data. Specifically, it retrieves streaming data and starts playback.

[1019] Step 12:

[1020] The device monitors the battery level and suggests charging locations if charging is needed. The input is battery data, and the output is notification data. Specifically, it sends a notification to the user stating, "Charging is needed at the next rest stop."

[1021] Step 13:

[1022] Once the user approves the charging suggestion, the device recalculates the route to the nearest charging station. The input is the user's approval information, and the output is the recalculated route information. Specifically, it sends a recalculation request to the server.

[1023] Step 14:

[1024] Upon arrival at a charging station, the device automatically processes payment and begins charging. The input is information about arrival at the charging station, and the output is payment confirmation and charging start notification. Specifically, it completes the payment using pre-registered payment information.

[1025] Step 15:

[1026] The server connects in real time with the National Police Agency and traffic control systems to receive traffic signal timings and emergency information. Inputs are traffic signal data and emergency information, while outputs are control instruction data. Specifically, it collects traffic information and sends appropriate instructions to terminals.

[1027] Step 16:

[1028] The terminal updates the autonomous driving control plan based on received traffic signal and emergency information. Inputs are traffic signal data and emergency information, while output is the updated control instructions. Specifically, it adjusts speed in accordance with signal timing and takes appropriate action based on emergency information.

[1029] Through this series of steps, the system of the present invention can improve user convenience, safety, and efficiency.

[1030] (Application Example 1)

[1031] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1032] Conventional autonomous vehicle systems have not adequately improved user convenience and safety. Specifically, there is a demand for seamless travel, such as users specifying destinations by voice and enjoying entertainment, as well as driving that reflects real-time traffic information, but systems that can achieve this are still limited. In addition, the procedures at charging stations are cumbersome, and automation of charging is desired. Therefore, there is a need for a system that enhances overall convenience and safety in autonomous driving.

[1033] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1034] In this invention, the server includes means for recognizing and analyzing voice instructions, means for communicating with the server to obtain the latest map data and traffic information and calculate the optimal route, and means for performing autonomous driving based on the calculated route information. This enables seamless destination setting, optimal route calculation, and autonomous driving control based on the user's voice instructions. Furthermore, by including functions such as entertainment playback means, battery level monitoring and charging facility guidance, automatic payment means, and cooperation with traffic control systems, user convenience and safety can be greatly improved. In addition, high-quality entertainment can be provided by utilizing voice instruction analysis using prompts and streaming playback functions.

[1035] "Means for recognizing and analyzing voice commands" refers to devices and software that understand voice commands entered by a user, analyze their content, and translate that into specific actions.

[1036] "Means of communicating with a server to obtain the latest map data and traffic information and calculate the optimal route" refers to equipment and software that communicates with an external server to obtain map data and traffic information that are updated in real time, and calculates the optimal driving route based on that data.

[1037] "Means for performing automated driving based on calculated route information" refers to devices and programs for automatically driving a vehicle based on acquired and calculated route information.

[1038] "Means for playing entertainment content" refers to devices and software used to play entertainment (music, videos, etc.) as instructed by the user.

[1039] "A means of monitoring battery levels, suggesting charging facilities, and recalculating routes to charging facilities as needed" refers to a system that constantly monitors the vehicle's battery level, suggests the nearest charging facility when necessary, and calculates a new route to that facility.

[1040] "Means of automatically processing payment upon arrival at a charging facility" refers to equipment and software that automatically completes payment and starts the charging process upon arrival at a charging facility, without requiring manual intervention from the user.

[1041] "Means of receiving traffic signal information and emergency information in cooperation with the National Police Agency and traffic control systems and reflecting it in autonomous driving" refers to a system that receives signal information and emergency information from the National Police Agency and traffic control systems in real time and adjusts the control of autonomous driving based on that information.

[1042] "Means of acquiring information from smart devices and utilizing it while driving" refers to a system that utilizes information acquired from the user's smartphone or other smart devices while driving, and reflects it in vehicle operation and entertainment provision.

[1043] "Means for analyzing and playing voice commands via prompts" refers to devices and software that analyze prompts given by a user via voice (e.g., a command to play music) and provide entertainment based on those commands.

[1044] "Means of acquiring and playing streaming data from a smartphone" refers to equipment and software that acquire data (such as music and videos) delivered in streaming format from a user's smartphone and play it back on an in-car system.

[1045] This invention relates to a vehicle system that integrates autonomous driving and generative AI. This system provides the optimal route based on user voice input and achieves autonomous driving using real-time updated map data and traffic information. Furthermore, it offers high levels of convenience and safety to the user through features such as charging station guidance and payment processing, entertainment provision, and coordination with police and traffic control systems.

[1046] Speech recognition and analysis

[1047] The user gets into the vehicle and specifies the destination to the generating AI by voice, saying, "Please take me to Yokohama." The voice recognition module installed in the vehicle receives this voice instruction, analyzes it, and extracts the destination "Yokohama." The generating AI then verbally notifies the user of the extracted information for confirmation. The voice recognition technology used here is the Google Speech Recognition API.

[1048] Route calculation and information retrieval

[1049] The terminal (vehicle) sends destination information to the server, which calculates the optimal route based on the latest map data and real-time traffic information. The calculation result is sent back to the terminal, and the terminal starts autonomous driving. A real-time map API is used in this process.

[1050] Autonomous driving

[1051] The terminal (vehicle) begins autonomous driving based on calculated route information. During operation, various sensors and cameras are installed, and the generating AI analyzes this information to dynamically correct the route. A high-precision GPS module is used in the hardware.

[1052] Entertainment provider

[1053] When a user commands "play music" while driving, the device analyzes the voice command, retrieves streaming data from the user's smartphone, and plays it through the car's audio system. This allows users to enjoy entertainment while driving.

[1054] Information and payment for charging facilities

[1055] The terminal monitors the battery level and suggests charging stations as needed. Once the user approves the suggestion, it recalculates the route to the nearest charging station. Upon arrival at the charging station, the terminal automatically processes the payment and begins charging. Once all processes are complete, the terminal notifies the user and resumes operation.

[1056] Integration with traffic control systems

[1057] The server connects in real time with the National Police Agency and traffic control systems to receive traffic signal timings and emergency information. The terminal uses this information to control autonomous driving and take appropriate action in response to emergency information.

[1058] Specific example

[1059] If a user is traveling from Shinjuku, Tokyo to Yokohama, the following prompt will be used:

[1060] "Launch SmartDrive Assistant"

[1061] "Please take me to Yokohama."

[1062] "Tell me the nearest charging station."

[1063] "Play the music."

[1064] In this way, the present invention can significantly improve user convenience, safety, and efficiency.

[1065] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1066] Step 1:

[1067] The user gets into the vehicle and enters a voice command specifying the destination. For example, they might say, "Please take me to Yokohama." A generative AI model receives this voice command and analyzes it with a speech recognition module. The input is the user's voice command, and the output is the analyzed text data. Through this process, the destination (in this case, "Yokohama") is extracted.

[1068] Step 2:

[1069] The terminal (vehicle) sends the analyzed destination information to the server. The input is destination text data, and the output is the result of the transmission to the server. The server uses real-time map data and traffic information to calculate the optimal route. This process generates the optimal route information to the destination.

[1070] Step 3:

[1071] The server returns the calculated optimal route information to the terminal (vehicle). The input is the route information calculated on the server, and the output is the result received by the terminal. The terminal receives this route information and starts autonomous driving. Through this process, the vehicle automatically travels towards its destination.

[1072] Step 4:

[1073] When a user gives a command such as "Play music" during operation, the generative AI model analyzes this voice command. The input is the voice command, and the output is the analyzed text data. This process recognizes the user's entertainment request.

[1074] Step 5:

[1075] The device retrieves streaming data from the user's smartphone based on the analyzed entertainment instructions. The input is the text data of the entertainment instructions, and the output is the retrieved streaming data. This process makes music and videos playable.

[1076] Step 6:

[1077] The device monitors the battery level and suggests charging facilities as needed. The input is battery level data, and the output is a charging suggestion notification to the user. When the user accepts the suggestion, the route to the nearest charging facility is recalculated. This process generates new route information to the charging facility.

[1078] Step 7:

[1079] Upon arrival at a charging station, the terminal automatically processes payment and begins charging. Inputs are data indicating arrival at the charging station and payment information, while outputs are the payment processing result and a charging start notification. This process allows users to complete charging without any effort on their part.

[1080] Step 8:

[1081] The server acquires information in real time from the National Police Agency and traffic control systems and transmits it to terminals. Inputs include traffic signal information and emergency information, while output is the results of the transmission to the terminals. The terminals use this information to control autonomous driving and respond to emergencies. This process ensures safe and efficient operation.

[1082] In this way, vehicle systems can significantly improve user convenience and safety through advanced autonomous driving and entertainment provision, automated charging and payment, and real-time integration of traffic information.

[1083] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1084] This invention relates to a vehicle system that integrates autonomous driving, generative artificial intelligence, and an emotion engine. This system provides the optimal route based on user voice input and achieves autonomous driving using the latest map data and traffic information. Furthermore, it offers high levels of convenience and safety to the user by providing charging station guidance and payment processing, entertainment, coordinating with police and traffic control systems, and recognizing the user's emotions.

[1085] Program processing and function description

[1086] Speech recognition and analysis

[1087] User: When boarding the vehicle, you can use voice commands to tell the generated artificial intelligence your destination. For example, you might say, "I want to go to Yokohama."

[1088] Terminal: The voice recognition module analyzes the user's voice and extracts destination information. The extracted information is then re-notified to the user via voice for confirmation.

[1089] Route calculation and information retrieval

[1090] Terminal: Sends destination information to the server. The server retrieves the latest map information from the map database and calculates the optimal route based on real-time traffic information. The calculation result is then sent back to the vehicle.

[1091] Autonomous driving

[1092] Terminal: Based on the received route information, it initiates autonomous driving. The generating AI starts the engine and begins operation. During operation, sensors and cameras constantly monitor the surrounding environment, and the generating AI dynamically adjusts the route.

[1093] Entertainment provider

[1094] User: Instructs the generating AI to "play music" while the vehicle is in operation.

[1095] Terminal: Analyzes voice commands, retrieves streaming data from the user's smartphone, and plays it. The in-car audio system activates and plays the desired music.

[1096] Information and payment for charging facilities

[1097] Device: Monitors battery level and suggests charging locations as needed. For example, it might notify the user, "You need to charge at the next rest stop." Once the user approves the suggestion, the device recalculates the route to the nearest charging location.

[1098] Terminal: Upon arrival at a charging station, the terminal automatically initiates the charging process. It processes payment in conjunction with the charging station's system and confirms that charging is started correctly. After charging is complete, it notifies the user and resumes operation.

[1099] Integration with traffic control systems

[1100] Server: Connects to the National Police Agency and traffic control systems to receive real-time information on traffic signal timings and emergency information.

[1101] Terminal: By reflecting the received traffic control information into the generating AI, it enables safe and efficient autonomous driving to continue.

[1102] Emotion recognition and response

[1103] The device features an emotion engine that analyzes the user's facial expressions and voice tone. The emotion engine recognizes the user's emotions and provides that information to the generating AI.

[1104] Terminal: Selects the appropriate driving mode based on the user's emotions. For example, if the user is feeling stressed, it will select a relaxing driving mode.

[1105] Terminal: Evaluates the user's stress level and fatigue, and suggests breaks as needed. Based on the evaluation results, it recalculates the route to rest points and suggests it to the user.

[1106] Device: Selects and suggests entertainment based on the user's mood. For example, if the user is in a good mood, it suggests rhythmic music. Conversely, if the user is sad, it suggests relaxing music.

[1107] Specific example

[1108] For example, if a user is traveling from Shinjuku, Tokyo to Yokohama, the following scenario will be executed.

[1109] The user gets into the vehicle and gives voice instructions to the AI ​​generator, saying, "Please take me to Yokohama."

[1110] The terminal analyzes the voice command and requests route calculation from the server.

[1111] The server calculates the optimal route, and the generating AI starts the automated driving process.

[1112] When a user requests "play music" during the ride, the AI ​​generates music data from the smartphone and plays it.

[1113] The device monitors the battery level and suggests charging facilities if charging is needed.

[1114] Once the user accepts the proposal and arrives at the charging station, the generating AI automatically processes the payment and starts charging.

[1115] The server provides traffic control information, and the generating AI adjusts the route in real time.

[1116] The device analyzes the user's facial expressions and voice, and if it determines that the user is tired, it suggests taking a break and guides them to an appropriate resting place.

[1117] Upon arrival in Yokohama, the AI ​​generates the data and terminates the operation, notifying the user.

[1118] In this way, the present invention can significantly improve user convenience, safety, and efficiency.

[1119] The following describes the processing flow.

[1120] Step 1:

[1121] When a user boards the vehicle, the terminal automatically activates its AI. A passenger sensor detects the user's presence, and a Bluetooth module scans the user's smartphone to attempt a connection. Once a Bluetooth connection is established, the user is notified.

[1122] Step 2:

[1123] The user gives voice instructions to the AI ​​generating the destination. For example, they might say, "I want to go to Yokohama." The device uses a voice recognition module to analyze the user's voice and extract the destination "Yokohama." The extracted information is then re-notified to the user via voice for confirmation.

[1124] Step 3:

[1125] The terminal sends destination information to the server. The server retrieves the latest map information from the map database, collects real-time traffic information, and calculates the optimal route. The calculation result is then sent back to the terminal.

[1126] Step 4:

[1127] The system initiates autonomous driving based on route information received by the terminal. The generating AI starts the engine and begins operation. During operation, various sensors and cameras constantly monitor the surrounding environment, and the generating AI dynamically adjusts the route.

[1128] Step 5:

[1129] The user gives a voice command, such as "I want to play music," while driving. The terminal analyzes the voice command and streams music data from the user's smartphone. The in-car audio system activates and plays the requested music.

[1130] Step 6:

[1131] The device monitors the battery level and, if it determines that charging is needed, suggests the most suitable charging location to the user. It notifies the user, "You need to charge at the next rest stop." Once the user accepts the suggestion, the device recalculates the route to the nearest charging location.

[1132] Step 7:

[1133] When the device arrives at the charging station, it automatically initiates the charging process. It processes payment in conjunction with the charging station's system and confirms that charging has started correctly. After charging is complete, it notifies the user and resumes operation.

[1134] Step 8:

[1135] The server connects to the National Police Agency and the traffic control system, receiving real-time information on traffic signal timings and emergency situations. The terminal then incorporates the received traffic control information into the generating AI, ensuring safe and efficient autonomous driving continues.

[1136] Step 9:

[1137] The device uses an emotion engine that analyzes the user's facial expressions and voice tone to recognize their emotions. If the user is stressed, the emotion engine provides that information to the generating AI, which then selects a relaxing driving mode.

[1138] Step 10:

[1139] The system assesses the user's stress level and fatigue, and suggests breaks as needed. Once the user approves the suggestion, the device recalculates the route to an appropriate rest stop and notifies the user.

[1140] Step 11:

[1141] The device selects entertainment based on the user's mood. For example, if the user is in a good mood, it suggests rhythmic music; conversely, if the user is sad, it suggests relaxing music.

[1142] Step 12:

[1143] Upon arrival at the destination, the terminal notifies the user that they have arrived at their destination. The generating AI stops autonomous driving and turns off the engine. When the user leaves the vehicle, the passenger sensor detects this and automatically shuts down the generating AI and other systems.

[1144] (Example 2)

[1145] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1146] Many autonomous driving systems currently on the market can recognize user voice commands, calculate routes, and drive autonomously, but they lack the ability to recognize user emotions and adjust driving modes accordingly. Furthermore, there is a need for a variety of functions to enhance user convenience, such as charging station suggestions and automatic payment, real-time traffic signal information, and entertainment features. In addition, the lack of technology to integrate and execute these functions makes improving the user experience a challenge.

[1147] The identification processing performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server has a generating artificial intelligence installed in the vehicle and includes means for recognizing and analyzing the user's voice instructions, means for communicating with the server to acquire the latest map data and traffic information and calculate the optimal route, means for performing automatic driving based on the calculated route information, means for analyzing the user's facial expressions and voice tone to recognize emotions and providing that information to the generating artificial intelligence, and means for selecting a driving mode based on the user's emotions. This makes it possible to adjust driving according to the user's emotions, significantly improving driving safety and comfort. Furthermore, by integrating and executing multiple functions such as suggesting charging facilities and automatic payment, reflecting real-time traffic signal information, and providing entertainment, user convenience and the quality of the experience can be enhanced.

[1148] "Generative artificial intelligence installed in a vehicle" refers to artificial intelligence installed in the vehicle's computer system that recognizes and analyzes user voice commands and performs autonomous driving and other vehicle operation management.

[1149] "User voice commands" refers to the act of a user giving instructions by voice regarding vehicle operation, setting a destination, etc.

[1150] "Means for recognizing and analyzing voice commands" refers to the collective term for hardware and software that capture, analyze, and understand user voice commands.

[1151] A "server" refers to a remote computer system that receives data transmitted from a vehicle, processes it, and sends the results back to the vehicle.

[1152] "Latest map data" refers to data that includes road and topographic information updated in real time using a geographic information system.

[1153] "Traffic information" refers to information about current road conditions, including dynamic data such as congestion, traffic accidents, and road construction.

[1154] "Means for calculating the optimal route" refers to algorithms and software used to calculate the most efficient and safe route to the user's destination.

[1155] "Means of autonomous driving" refers to an integrated system of hardware and software that enables autonomous operation of a vehicle.

[1156] "Means of recognizing emotions by analyzing a user's facial expressions and voice tone" refers to hardware and software that analyze the characteristics of a user's face and voice to determine their emotional state.

[1157] "Means for selecting a driving mode based on user emotions" refers to a control system that selects the optimal driving mode according to the recognized emotions of the user.

[1158] "Means of monitoring battery level" refers to sensors and software that continuously monitor the vehicle's battery status and provide necessary information.

[1159] "A means of suggesting charging facilities and recalculating the route to those facilities when charging is needed" refers to technology that, when the battery level drops, suggests the nearest charging facility to the user and calculates and suggests the optimal route to that facility.

[1160] "A means of automatically processing payment upon arrival at a charging facility" refers to a system that automatically completes payment when charging begins at a charging station.

[1161] "Means for recognizing music and video playback commands and acquiring streaming data from the user's smartphone for playback" refers to technology that recognizes a user's command to play music or videos, acquires the necessary data from the smartphone, and plays the content.

[1162] "Means of receiving traffic signal information and emergency information in cooperation with the National Police Agency and traffic control systems and reflecting it in autonomous driving" refers to technology that exchanges data in real time with the police and traffic control centers, receives information on changes in traffic signals and emergencies, and applies it to autonomous driving systems.

[1163] This invention relates to a vehicle system that integrates autonomous driving, generative artificial intelligence, and an emotion engine. This system provides the optimal route based on the user's voice input and achieves autonomous driving using the latest map data and traffic information. It also provides a high level of convenience and safety for the user by guiding them to charging stations and processing payments, providing entertainment, coordinating with police and traffic control systems, and recognizing user emotions.

[1164] Speech recognition and analysis

[1165] The user boards the vehicle and gives voice commands such as, "Please take me to Yokohama."

[1166] The terminal (in-vehicle system) captures audio using a microphone mounted on the vehicle and sends it to a speech recognition module (e.g., Google Cloud Speech-to-Text).

[1167] The device uses a speech recognition module to convert the voice data into text and extracts the destination information, "Yokohama." This information is then confirmed by the user via voice.

[1168] Route calculation and traffic information acquisition

[1169] The device sends the confirmed destination information to the server. It executes an API request to the server's URL and sends request data including the destination.

[1170] The server retrieves the latest map information and current traffic information from a map database (e.g., Google Maps API).

[1171] The server calculates the optimal route based on the information it has acquired, and this calculation uses real-time traffic data. The calculation result is returned to the terminal as an API response.

[1172] Autonomous driving

[1173] The terminal passes the route information it receives to a generating AI (e.g., Tesla's AutoPilot).

[1174] The terminal uses generated AI to start the engine and begin autonomous driving. During operation, it uses a 100-directional sensor and cameras to monitor the surrounding environment in real time.

[1175] The terminal uses generated AI to analyze data from sensors and cameras, and adjusts the route and driving actions as needed.

[1176] Entertainment provider

[1177] The user can give a voice command to "play music" while the vehicle is in operation.

[1178] The device analyzes the voice command and uses the API of a music streaming service (e.g., Spotify) to retrieve music data from the smartphone.

[1179] The device activates the in-car audio system and plays the acquired music data.

[1180] Information on charging facilities and payment processing

[1181] The device constantly monitors the battery level, and when the battery level drops below 20%, it notifies the user that "charging is needed at the next rest stop."

[1182] The user approves the notification.

[1183] The device sends another API request to the server to search for the nearest charging station.

[1184] The server provides information on the nearest charging station, recalculates the route, and sends it back to the terminal.

[1185] Upon arrival at the charging station, the system automatically initiates the charging process and handles payment in conjunction with the charging station's system. Once charging is complete, the user is notified by voice, and the vehicle resumes operation.

[1186] Integration with traffic control systems

[1187] The server connects in real time with the National Police Agency and traffic control systems to receive traffic signal timings and emergency information.

[1188] The server sends the received information to the terminal.

[1189] The terminal uses AI generation to dynamically reflect traffic control information, ensuring safe operation continues.

[1190] Emotion recognition and adaptive responses

[1191] The device is equipped with an emotion engine (e.g., Microsoft Azure Cognitive Services) that analyzes the user's facial expressions and voice tone.

[1192] The device analyzes the user's emotions from voice and camera data and recognizes that they are experiencing stress.

[1193] The device uses AI generation to select a relaxing driving mode based on the user's emotional information.

[1194] If the device detects that the user is stressed, it will suggest and play relaxing music. It will also suggest taking a break if it detects fatigue.

[1195] If the user agrees to the suggestion, the device calculates and guides them to the optimal resting place.

[1196] Specific examples and examples of prompt sentences for generative AI models

[1197] For example, if a user is traveling from Shinjuku, Tokyo to Yokohama:

[1198] The user gives a voice command saying, "Please take me to Yokohama."

[1199] The terminal analyzes the voice command and requests route calculation from the server.

[1200] The server calculates the optimal route and sends it back to the terminal.

[1201] The device initiates autonomous driving, monitoring the surroundings in real time while driving.

[1202] When the user requests that the device play music, the device will play music from the smartphone.

[1203] The device monitors the battery level and suggests charging facilities as needed. If the user agrees, it will guide them to the charging facility and automatically process payment and begin charging upon arrival.

[1204] The server transmits traffic control information to the terminal, which then adjusts its driving to ensure safety.

[1205] The device analyzes the user's emotions and, if necessary, provides guidance on relaxing driving modes or taking breaks.

[1206] Specific examples of prompt statements for generative AI models:

[1207] "Calculate the optimal route to our destination, Yokohama, and ensure safe and efficient driving."

[1208] "Please explain the procedure for analyzing user voice commands and providing entertainment."

[1209] "If the current battery level is below 20%, please explain how to suggest the nearest charging station and process the payment automatically."

[1210] "Please explain the process for recognizing user emotions and making suggestions to reduce stress."

[1211] The specific embodiments of the present invention have now been described. This embodiment significantly improves user safety and convenience.

[1212] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1213] Step 1: Receiving and analyzing voice input

[1214] Input: The user gives a voice command inside the vehicle saying, "Please take me to Yokohama."

[1215] Terminal: A microphone inside the vehicle captures the user's voice and sends the audio data to a speech recognition module (e.g., Google Cloud Speech-to-Text). The speech recognition module analyzes the audio data and extracts destination information such as "Yokohama".

[1216] Output: Destination information is returned to the terminal as text data.

[1217] Operation: The device confirms the voice analysis results with the user via voice. It asks the user, "Is Yokohama your destination?"

[1218] Step 2: Route calculation and acquisition of traffic information

[1219] Input: The terminal is given information about the destination, "Yokohama".

[1220] Terminal: Sends destination information to the server. The data, including the destination, is sent to the server as an API request.

[1221] Server: Retrieves the latest map information and real-time traffic information from a map database (e.g., Google Maps API). This information is used to calculate the optimal route.

[1222] Output: The calculation result is sent back to the terminal as route information.

[1223] Operation: The terminal receives route information calculated by the server and uses it in subsequent steps.

[1224] Step 3: Start autonomous driving

[1225] Input: Route information is provided to the terminal.

[1226] Terminal: Passes the received route information to the generating AI (e.g., Tesla's AutoPilot).

[1227] Terminal: The generating AI starts the engine and begins autonomous driving. It monitors the surrounding environment in real time using a 100-directional sensor and camera.

[1228] Output: The start state for autonomous driving mode is generated.

[1229] Operation: The generating AI analyzes data from sensors and cameras and adjusts the route and driving actions in real time.

[1230] Step 4: Providing Entertainment

[1231] Input: The user gives a voice command saying, "Please play music."

[1232] Terminal: Analyzes voice commands and retrieves music data from the user's smartphone using the API of a music streaming service (e.g., Spotify).

[1233] Terminal: Activates the in-car audio system and plays the acquired music data.

[1234] Output: Music playback begins.

[1235] Operation: The device streams music data based on voice commands and plays it on the audio system.

[1236] Step 5: Guidance to charging facilities and payment processing

[1237] Input: The device monitors the battery level, and when the battery level falls below 20%, it triggers a response.

[1238] Device: Notifies the user that "charging is required at the next rest stop."

[1239] User: Accept notification.

[1240] Terminal: Sends another API request to the server to search for the nearest charging station.

[1241] Server: Provides information on the nearest charging station, recalculates the route, and sends it back to the terminal.

[1242] Terminal: Upon arrival at the charging station, it automatically initiates the charging process and handles payment processing in conjunction with the charging station's system.

[1243] Output: Charging and payment processing are complete.

[1244] Operation: The device starts the charging process, notifies the user when it is complete, and resumes operation.

[1245] Step 6: Integration with traffic control systems

[1246] Input: The server receives real-time traffic signal information and emergency information from the National Police Agency and traffic control systems.

[1247] Server: Sends received information to the terminal.

[1248] Terminal: The generating AI dynamically reflects traffic control information, ensuring safe operation continues.

[1249] Output: The operational status is updated based on traffic control information.

[1250] Operation: The generating AI reflects traffic signal information in real time and adjusts driving accordingly.

[1251] Step 7: Emotion Recognition and Adaptive Response

[1252] Input: The user's facial expressions and voice tone are captured by the device.

[1253] Terminal: An emotion engine (e.g., Microsoft Azure Cognitive Services) analyzes the user's facial expressions and voice tone to recognize emotions.

[1254] Terminal: Provides emotional information to the AI ​​that generates it.

[1255] Generative AI: Selects a relaxing driving mode based on the user's emotions. Suggests and plays relaxing music as needed.

[1256] Output: The system initiates driving modes and provides entertainment tailored to the user's mood.

[1257] Operation: The generated AI selects a driving mode based on the user's emotional information and adjusts the driving to promote relaxation. It suggests breaks as needed and guides the user to the optimal rest stop.

[1258] (Application Example 2)

[1259] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1260] In modern autonomous vehicles, it is crucial to accurately recognize user voice commands and provide optimal routes using up-to-date map data and traffic information. However, in addition to this, it is also necessary to handle charging station guidance and payment processing, appropriate driving modes and entertainment based on user emotion recognition, and the reflection of real-time traffic control information. Conventional systems lack these comprehensive functions, resulting in challenges in ensuring sufficient user convenience and safety.

[1261] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a generating artificial intelligence installed in the vehicle and means for recognizing and analyzing the user's voice instructions, means for communicating with the server to acquire the latest map data and traffic information and calculate the optimal route, means for performing autonomous driving based on the calculated route information, means for having an emotion recognition engine to recognize emotions from the user's facial expressions and voice tone and provide driving modes and entertainment based on that, and means for guiding users to charging facilities and processing payments. This makes it possible to significantly improve user convenience and safety by providing the optimal route based on the user's voice instructions, recognizing and adapting to the situation during autonomous driving, guiding users to charging stations and processing payments, and providing entertainment that responds to emotions.

[1262] "Generative artificial intelligence" is an artificial intelligence that recognizes and analyzes user voice commands and generates the information necessary for decision-making.

[1263] "Map data" refers to data that represents geographical information such as roads, buildings, and topography in a digital format.

[1264] "Traffic information" refers to data that shows the current traffic conditions, including real-time information such as traffic congestion and accidents.

[1265] The "optimal route" is the most efficient path between a designated starting point and a destination, taking into account factors such as travel time, distance, and traffic conditions.

[1266] "Autonomous driving" is a technology that allows vehicles to move autonomously without driver intervention.

[1267] An "emotion recognition engine" is a technology that analyzes a user's facial expressions and voice tone to identify their emotional state.

[1268] "Entertainment provision" refers to the function of playing music, videos, and other entertainment content for users.

[1269] The "Charging Station Guide" is a function that displays the nearest suitable charging station to the user based on the vehicle's battery level.

[1270] "Payment processing" is the process by which the system pays for the services or goods provided on behalf of the user.

[1271] This invention relates to a system for autonomous driving vehicles that integrates generative artificial intelligence, an emotion recognition engine, real-time traffic information, and entertainment provision functions.

[1272] First, when a user gets into the vehicle, the AI ​​recognizes and analyzes the user's voice instructions. For example, if the user says, "I want to go to Yokohama," the voice is converted to text using the Google Cloud Speech-to-Text API, and "Yokohama" as the destination is extracted.

[1273] Next, this destination information is sent to a server, which uses the Google Maps API to calculate the optimal route based on the latest map data and real-time traffic information. The calculation results are sent back to the vehicle, and autonomous driving begins. Autonomous driving is performed using each vehicle manufacturer's autonomous driving API.

[1274] While driving, the generated AI uses an emotion recognition engine powered by the Affectiva SDK to analyze the user's facial expressions and voice tone, identifying the user's emotional state. For example, if the user is tired, the system will suggest a rest stop. This improves user safety and comfort.

[1275] Furthermore, when the battery level drops, the system automatically suggests a charging station. Once the user approves, it uses the Google Maps API to recalculate the optimal route to the charging station, and payment is automatically processed upon arrival.

[1276] In providing entertainment, when a user requests "play music," the AI ​​generates music and videos using the Spotify API or YouTube Data API and plays them through the in-car audio system.

[1277] Furthermore, the system collaborates with the National Police Agency and traffic control systems to acquire traffic signal and emergency information in real time, and adjusts autonomous driving based on this information. This enables safe operation that is in line with traffic conditions.

[1278] Finally, the following are concrete examples and examples of prompt messages.

[1279] Specific example:

[1280] For example, if a user is traveling from Shinjuku, Tokyo to Yokohama, the following scenario will be executed: The user gets into the vehicle and gives a voice command to the generating AI, "Take me to Yokohama, please." The voice is then converted to text using the Google Cloud Speech-to-Text API. Next, the optimal route is calculated using the Google Maps API, and autonomous driving begins. During the journey, if the user requests, "Please play music," the generating AI will play music using the Spotify API. If the battery level drops, the nearest charging station will be indicated, and payment will be processed automatically. In addition, the route is adjusted in real time based on traffic control information.

[1281] Example of a prompt:

[1282] When a user says, "Take me to Yokohama," the voice recognition module analyzes the voice, calculates the optimal route using the Google Maps API, and starts autonomous driving. If the user says, "Play music" during the ride, music will be played using the Spotify API. If the battery is low, the nearest charging station will be guided, and payment processing will be initiated. The emotion recognition engine analyzes the user's emotions and provides entertainment suggestions based on that.

[1283] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1284] Step 1:

[1285] The user boards the vehicle and gives voice commands to the generating artificial intelligence. For example, they might say, "I want to go to Yokohama." This voice input is acquired through the microphone inside the vehicle.

[1286] Input: Voice Instructions

[1287] Output: Audio data

[1288] Step 2:

[1289] The device uses the Google Cloud Speech-to-Text API to convert the user's voice commands into text. The destination is extracted based on the converted text data. This analysis result is recorded in a database.

[1290] Input: Audio data

[1291] Output: Text data (e.g., "I want to go to Yokohama")

[1292] Step 3:

[1293] The device sends the acquired text data to the server. The server uses the Google Maps API to obtain the latest map data and real-time traffic information based on the destination, and calculates the optimal route based on that data.

[1294] Input: Text data (destination information)

[1295] Output: Map data, traffic information, optimal route

[1296] Step 4:

[1297] The server sends the calculated optimal route information to a terminal inside the vehicle. Based on the received route information, the terminal issues instructions to the autonomous driving system and starts autonomous driving. During autonomous driving, sensor data (cameras, LiDAR, etc.) is constantly monitored, and the generating AI dynamically adjusts the route.

[1298] Input: Optimal route information

[1299] Output: Vehicle operation instructions, start of autonomous driving.

[1300] Step 5:

[1301] During operation, the terminal uses the Affectiva SDK to analyze the user's facial expressions and voice tone acquired from the in-vehicle camera and microphone. The emotion recognition engine recognizes the user's emotions, and based on that information, the system provides appropriate driving modes and entertainment.

[1302] Input: Facial expression data, voice tone

[1303] Output: Emotional state, driving mode instructions, entertainment suggestions

[1304] Step 6:

[1305] As an example of entertainment provision, when a user requests "play music," the device uses the Spotify API or YouTube Data API to retrieve music or videos and plays them through the in-car audio system.

[1306] Input: Voice command (entertainment request)

[1307] Output: Streaming data, entertainment playback

[1308] Step 7:

[1309] When the battery level drops, the device monitors the battery in real time and suggests the nearest charging station if necessary. Once the user approves the suggestion, the device uses the Google Maps API to calculate the optimal route to the charging facility and sends that information to the server.

[1310] Input: Battery level

[1311] Output: Suggested charging stations, optimal route information (to charging stations)

[1312] Step 8:

[1313] When the user arrives at the charging station, the terminal automatically processes the payment in conjunction with the charging station's system and begins charging. Once charging is complete, the terminal notifies the user and resumes operation.

[1314] Input: Arrival information for charging facilities

[1315] Output: Payment processing execution, charging start instruction, completion notification

[1316] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1317] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1318] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1319] [Fourth Embodiment]

[1320] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1321] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1322] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1323] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1324] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1325] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1326] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1327] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1328] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1329] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1330] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1331] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1332] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1333] This invention relates to a vehicle system that integrates autonomous driving and generative artificial intelligence. This system provides the optimal route based on user voice input and achieves autonomous driving using real-time updated map data and traffic information. Furthermore, it provides high levels of convenience and safety to the user through charging station guidance and payment processing, entertainment provision, and coordination with police and traffic control systems.

[1334] Program processing and function description

[1335] Speech recognition and analysis

[1336] User: When getting into the car, specify the destination to the generating AI by voice. For example, say, "I want to go to Yokohama."

[1337] Terminal: Generative artificial intelligence (generative AI) installed in the vehicle receives voice commands, and a voice recognition module analyzes them to extract the destination "Yokohama." The extracted information is then re-notified to the user by voice for confirmation.

[1338] Route calculation and information retrieval

[1339] Terminal (vehicle): Sends destination information to the server. For example, it requests a route to Yokohama.

[1340] Server: Calculates the optimal route based on the latest map data and real-time traffic information, and sends the result back to the vehicle.

[1341] Autonomous driving

[1342] Terminal (vehicle): Based on the received route information, it starts autonomous driving. During operation, it is equipped with various sensors and cameras, and the generating AI analyzes this information to dynamically correct the route.

[1343] Entertainment provider

[1344] User: Instructs the generating AI to "play music" while the vehicle is in operation.

[1345] Terminal (vehicle): Analyzes voice commands, retrieves streaming data from the user's smartphone, and plays it back on the in-vehicle audio system.

[1346] Information and payment for charging facilities

[1347] Terminal (vehicle): Monitors battery level and suggests charging facilities as needed. For example, it might notify you, "Charging is required at the next rest stop."

[1348] User: If you approve the proposal, the route to the nearest charging station will begin recalculating.

[1349] Terminal (vehicle): Upon arrival at a charging station, the system automatically processes payment and begins charging. Once all processing is complete, the user is notified and the vehicle resumes operation.

[1350] Integration with traffic control systems

[1351] Server: Connects in real time with the National Police Agency and traffic control systems to receive traffic signal timings and emergency information.

[1352] Terminal (vehicle): Controls autonomous driving based on received traffic signal information. Also, takes appropriate action in response to emergency information.

[1353] Specific example

[1354] For example, if a user is traveling from Shinjuku, Tokyo to Yokohama, the following scenarios are possible.

[1355] The user gets into the vehicle and gives voice instructions to the AI ​​generator, saying, "Take me to Yokohama, please."

[1356] The terminal (vehicle) analyzes the voice commands and requests route calculation from the server.

[1357] The server calculates the optimal route and sends it to the vehicle, and the generating AI starts autonomous driving.

[1358] When a user requests "play music" during the ride, the AI ​​generates music data from the smartphone and plays it.

[1359] The terminal (vehicle) monitors the battery level and suggests charging facilities if charging is needed.

[1360] Once the user accepts the proposal and arrives at the charging station, the generating AI automatically processes the payment and begins charging.

[1361] The server provides traffic control information, and the generating AI adjusts the route in real time.

[1362] Finally, upon arriving in Yokohama, the generating AI terminates the operation and notifies the user.

[1363] In this way, the present invention can significantly improve user convenience, safety, and efficiency.

[1364] The following describes the processing flow.

[1365] Step 1:

[1366] When a user boards the vehicle, the terminal automatically activates its AI. A passenger sensor detects the user's presence, and a Bluetooth module scans the user's smartphone to attempt a connection. Once a Bluetooth connection is established, the user is notified.

[1367] Step 2:

[1368] The user gives voice instructions to the generating AI about their destination. For example, they might say, "I want to go to Yokohama." The device uses a voice recognition module to analyze the user's voice and extract the destination "Yokohama." The generating AI then verbally notifies the user of the extracted information for confirmation.

[1369] Step 3:

[1370] The terminal sends destination information to the server. The server retrieves the latest map information from the map database, collects real-time traffic information, and calculates the optimal route. The calculation result is then sent back to the terminal.

[1371] Step 4:

[1372] The system initiates autonomous driving based on route information received by the terminal. The generating AI starts the engine and begins operation. During operation, sensors and cameras constantly monitor the surrounding environment, and the generating AI dynamically adjusts the route.

[1373] Step 5:

[1374] The user gives a command during operation, such as "Please play music." The terminal analyzes the voice command and streams music data from the user's smartphone. The audio system activates and plays the requested music.

[1375] Step 6:

[1376] The device monitors the battery level and, if it determines that charging is needed, suggests the most suitable charging location to the user. It notifies the user, "You need to charge at the next rest stop." Once the user accepts the suggestion, the device recalculates the route to the nearest charging location.

[1377] Step 7:

[1378] When the device arrives at the charging station, it automatically initiates the charging process. It processes payment in conjunction with the charging station's system and confirms that charging has started correctly. After charging is complete, it notifies the user and resumes operation.

[1379] Step 8:

[1380] The server connects to the National Police Agency and the traffic control system, receiving real-time information on traffic signal timings and emergency situations. The terminal then incorporates the received traffic control information into the generating AI, ensuring safe and efficient autonomous driving continues.

[1381] Step 9:

[1382] Upon arrival at the destination, the terminal notifies the user that they have arrived at their destination. The generating AI stops autonomous driving and turns off the engine. When the user leaves the vehicle, the passenger sensor detects this and automatically shuts down the generating AI and other systems.

[1383] (Example 1)

[1384] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1385] Modern automotive systems need to integrate many functions, including real-time autonomous driving adjustments based on traffic signals and emergency information, highly accurate route calculation, charging station guidance and automated payment processing, and consistent entertainment delivery. However, existing systems struggle to achieve these efficiently and safely. Furthermore, there is a need to ensure driving safety while maximizing user convenience.

[1386] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1387] In this invention, the server includes a generator artificial intelligence installed in the vehicle, means for recognizing and analyzing user voice commands, means for communicating with the server to obtain the latest map data and traffic information and calculate the optimal route, means for performing autonomous driving based on the calculated route information, means for obtaining and playing streaming data from the user's smartphone to provide entertainment during operation, means for monitoring the surrounding environment in real time and dynamically adjusting the route and speed, and means for coordinating with the National Police Agency and traffic control systems to receive traffic signal information and emergency information and reflect it in autonomous driving. This makes it possible to significantly improve user convenience, safety, and efficiency.

[1388] "Generative artificial intelligence" is a system that uses advanced algorithms, including natural language processing and speech recognition technologies, to analyze user instructions in real time and control autonomous driving or provide entertainment.

[1389] "Means for recognizing and analyzing user voice instructions" refers to a voice recognition module that analyzes voice acquired by microphones inside the vehicle and extracts the user's intended instructions.

[1390] A "server" refers to a computer system that receives requests from vehicles, calculates the optimal route based on map data and traffic information, and transmits that information to the vehicle.

[1391] "Up-to-date map data and traffic information" refers to road maps and traffic condition data obtained in real time from map service providers and traffic information services.

[1392] "Means for calculating the optimal route" refers to an algorithm in which a server calculates the most efficient route under specified conditions based on the vehicle's current location and destination information.

[1393] "Means of autonomous driving" refers to a system that autonomously controls a vehicle based on calculated route information to operate it to its destination.

[1394] "Means of providing entertainment" refers to a system that acquires streaming data from the user's smartphone and plays it back through the vehicle's audio system.

[1395] "A means of monitoring the surrounding environment in real time and dynamically adjusting the route and speed" refers to a function that uses sensors such as LiDAR and cameras to detect the surrounding situation, and then generates AI to analyze that information and adjust the route and speed in real time.

[1396] "Means for receiving traffic signal information and emergency information and reflecting it in autonomous driving" refers to a system that uses signal timing and emergency information obtained in cooperation with the National Police Agency and traffic control systems to reflect it in autonomous driving control.

[1397] "Means of monitoring battery level" refers to a system that constantly monitors the vehicle's battery level and notifies the user when charging is necessary.

[1398] "Means of suggesting charging facilities" refers to a system that searches for the nearest charging station and guides the user when the battery level drops.

[1399] "Method for recalculating the route to the charging facility" refers to an algorithm that, when charging is needed, causes the server to recalculate the optimal route from the current location to the nearest charging facility.

[1400] "Automatic payment processing" refers to a function that automatically completes payment using pre-registered credit card information or a digital wallet upon arrival at a charging station.

[1401] This invention relates to a system that enables autonomous driving within a vehicle and integrates multiple functions, including entertainment provision, guidance and payment processing for charging facilities, and real-time driving adjustments based on traffic signals and emergency information. This system functions with generative artificial intelligence (generative AI model) at its core.

[1402] System Configuration

[1403] hardware

[1404] Terminal: Computer and sensor devices (cameras, LiDAR, radar, microphones, etc.) installed in the vehicle.

[1405] Server: A remote computer system that interacts with map databases and traffic information APIs.

[1406] User device: A mobile device such as a smartphone or tablet.

[1407] software

[1408] Generative AI models: speech recognition modules, route calculation algorithms, entertainment playback systems, payment processing systems.

[1409] Communication protocol: A standard internet protocol used for communication (e.g., HTTP / HTTPS).

[1410] Processing in the autonomous driving system

[1411] Speech recognition and analysis

[1412] The user boards the vehicle and gives a voice command for their destination. For example: "I want to go to Yokohama." The device's microphone captures the voice, and a voice recognition module analyzes it. The analysis result is generated as text and sent to the user for confirmation.

[1413] Root calculation

[1414] The terminal sends destination information to the server, which calculates the optimal route based on the latest map data and real-time traffic information. The calculation result is sent back to the terminal, and the terminal starts autonomous driving.

[1415] Autonomous driving

[1416] The terminal controls the vehicle based on the received route information. Various sensors (camera, LiDAR, radar) monitor the surrounding environment in real time and dynamically adjust the route and speed. The generated AI analyzes this information and sends appropriate operational instructions to the vehicle control system.

[1417] Entertainment provider

[1418] During operation, the user instructs the AI ​​to "play music." The terminal analyzes the voice command, retrieves streaming data from the user's device, and plays it through the in-car audio system.

[1419] Information and payment for charging facilities

[1420] The device monitors the battery level and suggests charging locations as needed. For example, it might notify the user, "You need to charge at the next rest stop." Once the user approves the suggestion, the device recalculates the route to the charging location, automatically processes payment upon arrival, and begins charging.

[1421] Integration with traffic control systems

[1422] The server connects in real time with the National Police Agency and traffic control systems to receive traffic signal timings and emergency information. Based on this information, the terminal updates the control plan for autonomous driving, ensuring safe and efficient operation.

[1423] Specific example

[1424] For example, if a user instructs the system to "go to Yokohama," the terminal sends this information to the server. The server calculates the route and sends the result back to the terminal. Once autonomous driving begins, if the user instructs the system to "play music" during the journey, the terminal retrieves music from the user's device and plays it. It suggests charging stations as needed, and resumes operation once charging is complete. Based on traffic information, necessary route changes are reflected in real time.

[1425] Examples of prompt messages include "I want to go to Yokohama," "Please play music," and "I need to charge my phone at the next rest stop."

[1426] In this way, integrated systems can significantly improve user convenience, security, and efficiency.

[1427] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1428] Step 1:

[1429] The user boards the vehicle and gives voice instructions for their destination. The input is voice, and the output is voice data. For example, the user might say, "I want to go to Yokohama."

[1430] Step 2:

[1431] The device captures audio data through its microphone. The input is the user's voice, and the output is digitized audio data. This audio data is sent to the speech recognition module of the generating AI.

[1432] Step 3:

[1433] The terminal's voice recognition module analyzes the voice data and converts it into text information. The input is digitized voice data, and the output is text-based instruction information. Specifically, it extracts the destination "Yokohama" as text.

[1434] Step 4:

[1435] The device then re-notifies the user of the analyzed text information. The input is transcribed instruction information, and the output is an audio notification. Specifically, it plays a confirmation message saying, "Your destination is Yokohama. Is this correct?"

[1436] Step 5:

[1437] The terminal sends text information to the server and requests the optimal route based on the latest map data and traffic information. The input is destination information (text), and the output is request data. Specifically, the data is transmitted using the Internet Protocol.

[1438] Step 6:

[1439] The server retrieves the latest data from the map database and traffic information API and calculates the optimal route. The input is the request data, and the output is the optimal route information. Specifically, it uses an algorithm to calculate the route.

[1440] Step 7:

[1441] The server returns the calculation results to the terminal. The input is the calculated route information, and the output is the returned data. Specifically, it sends data including the coordinates of each point along the route, the total distance, the time required, and any points to note.

[1442] Step 8:

[1443] The terminal initiates autonomous driving based on the received route information. The input is the optimal route information, and the output is vehicle control instructions. Specifically, it sends instructions to the vehicle's control system and starts autonomous driving.

[1444] Step 9:

[1445] The terminal monitors the surrounding environment in real time using sensors mounted on the vehicle (camera, LiDAR, radar), and the generated AI analyzes this information to dynamically adjust the route and speed. The input is sensor data, and the output is vehicle control instructions. Specific operations include obstacle detection and lane keeping.

[1446] Step 10:

[1447] During operation, the user gives a voice command saying, "Please play music." The input is voice, and the output is voice data. Specifically, the user says the command, "Please play music."

[1448] Step 11:

[1449] The terminal uses a streaming service API to retrieve streaming data from the user's device and play it back on the in-car audio system. The input is a voice command, and the output is audio data. Specifically, it retrieves streaming data and starts playback.

[1450] Step 12:

[1451] The device monitors the battery level and suggests charging locations if charging is needed. The input is battery data, and the output is notification data. Specifically, it sends a notification to the user stating, "Charging is needed at the next rest stop."

[1452] Step 13:

[1453] Once the user approves the charging suggestion, the device recalculates the route to the nearest charging station. The input is the user's approval information, and the output is the recalculated route information. Specifically, it sends a recalculation request to the server.

[1454] Step 14:

[1455] Upon arrival at a charging station, the device automatically processes payment and begins charging. The input is information about arrival at the charging station, and the output is payment confirmation and charging start notification. Specifically, it completes the payment using pre-registered payment information.

[1456] Step 15:

[1457] The server connects in real time with the National Police Agency and traffic control systems to receive traffic signal timings and emergency information. Inputs are traffic signal data and emergency information, while outputs are control instruction data. Specifically, it collects traffic information and sends appropriate instructions to terminals.

[1458] Step 16:

[1459] The terminal updates the autonomous driving control plan based on received traffic signal and emergency information. Inputs are traffic signal data and emergency information, while output is the updated control instructions. Specifically, it adjusts speed in accordance with signal timing and takes appropriate action based on emergency information.

[1460] Through this series of steps, the system of the present invention can improve user convenience, safety, and efficiency.

[1461] (Application Example 1)

[1462] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1463] Conventional autonomous vehicle systems have not adequately improved user convenience and safety. Specifically, there is a demand for seamless travel, such as users specifying destinations by voice and enjoying entertainment, as well as driving that reflects real-time traffic information, but systems that can achieve this are still limited. In addition, the procedures at charging stations are cumbersome, and automation of charging is desired. Therefore, there is a need for a system that enhances overall convenience and safety in autonomous driving.

[1464] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1465] In this invention, the server includes means for recognizing and analyzing voice instructions, means for communicating with the server to obtain the latest map data and traffic information and calculate the optimal route, and means for performing autonomous driving based on the calculated route information. This enables seamless destination setting, optimal route calculation, and autonomous driving control based on the user's voice instructions. Furthermore, by including functions such as entertainment playback means, battery level monitoring and charging facility guidance, automatic payment means, and cooperation with traffic control systems, user convenience and safety can be greatly improved. In addition, high-quality entertainment can be provided by utilizing voice instruction analysis using prompts and streaming playback functions.

[1466] "Means for recognizing and analyzing voice commands" refers to devices and software that understand voice commands entered by a user, analyze their content, and translate that into specific actions.

[1467] "Means of communicating with a server to obtain the latest map data and traffic information and calculate the optimal route" refers to equipment and software that communicates with an external server to obtain map data and traffic information that are updated in real time, and calculates the optimal driving route based on that data.

[1468] "Means for performing automated driving based on calculated route information" refers to devices and programs for automatically driving a vehicle based on acquired and calculated route information.

[1469] "Means for playing entertainment content" refers to devices and software used to play entertainment (music, videos, etc.) as instructed by the user.

[1470] "A means of monitoring battery levels, suggesting charging facilities, and recalculating routes to charging facilities as needed" refers to a system that constantly monitors the vehicle's battery level, suggests the nearest charging facility when necessary, and calculates a new route to that facility.

[1471] "Means of automatically processing payment upon arrival at a charging facility" refers to equipment and software that automatically completes payment and starts the charging process upon arrival at a charging facility, without requiring manual intervention from the user.

[1472] "Means of receiving traffic signal information and emergency information in cooperation with the National Police Agency and traffic control systems and reflecting it in autonomous driving" refers to a system that receives signal information and emergency information from the National Police Agency and traffic control systems in real time and adjusts the control of autonomous driving based on that information.

[1473] "Means of acquiring information from smart devices and utilizing it while driving" refers to a system that utilizes information acquired from the user's smartphone or other smart devices while driving, and reflects it in vehicle operation and entertainment provision.

[1474] "Means for analyzing and playing voice commands via prompts" refers to devices and software that analyze prompts given by a user via voice (e.g., a command to play music) and provide entertainment based on those commands.

[1475] "Means of acquiring and playing streaming data from a smartphone" refers to equipment and software that acquire data (such as music and videos) delivered in streaming format from a user's smartphone and play it back on an in-car system.

[1476] This invention relates to a vehicle system that integrates autonomous driving and generative AI. This system provides the optimal route based on user voice input and achieves autonomous driving using real-time updated map data and traffic information. Furthermore, it offers high levels of convenience and safety to the user through features such as charging station guidance and payment processing, entertainment provision, and coordination with police and traffic control systems.

[1477] Speech recognition and analysis

[1478] The user gets into the vehicle and specifies the destination to the generating AI by voice, saying, "Please take me to Yokohama." The voice recognition module installed in the vehicle receives this voice instruction, analyzes it, and extracts the destination "Yokohama." The generating AI then verbally notifies the user of the extracted information for confirmation. The voice recognition technology used here is the Google Speech Recognition API.

[1479] Route calculation and information retrieval

[1480] The terminal (vehicle) sends destination information to the server, which calculates the optimal route based on the latest map data and real-time traffic information. The calculation result is sent back to the terminal, and the terminal starts autonomous driving. A real-time map API is used in this process.

[1481] Autonomous driving

[1482] The terminal (vehicle) begins autonomous driving based on calculated route information. During operation, various sensors and cameras are installed, and the generating AI analyzes this information to dynamically correct the route. A high-precision GPS module is used in the hardware.

[1483] Entertainment provider

[1484] When a user commands "play music" while driving, the device analyzes the voice command, retrieves streaming data from the user's smartphone, and plays it through the car's audio system. This allows users to enjoy entertainment while driving.

[1485] Information and payment for charging facilities

[1486] The terminal monitors the battery level and suggests charging stations as needed. Once the user approves the suggestion, it recalculates the route to the nearest charging station. Upon arrival at the charging station, the terminal automatically processes the payment and begins charging. Once all processes are complete, the terminal notifies the user and resumes operation.

[1487] Integration with traffic control systems

[1488] The server connects in real time with the National Police Agency and traffic control systems to receive traffic signal timings and emergency information. The terminal uses this information to control autonomous driving and take appropriate action in response to emergency information.

[1489] Specific example

[1490] If a user is traveling from Shinjuku, Tokyo to Yokohama, the following prompt will be used:

[1491] "Launch SmartDrive Assistant"

[1492] "Please take me to Yokohama."

[1493] "Tell me the nearest charging station."

[1494] "Play the music."

[1495] In this way, the present invention can significantly improve user convenience, safety, and efficiency.

[1496] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1497] Step 1:

[1498] The user gets into the vehicle and enters a voice command specifying the destination. For example, they might say, "Please take me to Yokohama." A generative AI model receives this voice command and analyzes it with a speech recognition module. The input is the user's voice command, and the output is the analyzed text data. Through this process, the destination (in this case, "Yokohama") is extracted.

[1499] Step 2:

[1500] The terminal (vehicle) sends the analyzed destination information to the server. The input is destination text data, and the output is the result of the transmission to the server. The server uses real-time map data and traffic information to calculate the optimal route. This process generates the optimal route information to the destination.

[1501] Step 3:

[1502] The server returns the calculated optimal route information to the terminal (vehicle). The input is the route information calculated on the server, and the output is the result received by the terminal. The terminal receives this route information and starts autonomous driving. Through this process, the vehicle automatically travels towards its destination.

[1503] Step 4:

[1504] When a user gives a command such as "Play music" during operation, the generative AI model analyzes this voice command. The input is the voice command, and the output is the analyzed text data. This process recognizes the user's entertainment request.

[1505] Step 5:

[1506] The device retrieves streaming data from the user's smartphone based on the analyzed entertainment instructions. The input is the text data of the entertainment instructions, and the output is the retrieved streaming data. This process makes music and videos playable.

[1507] Step 6:

[1508] The device monitors the battery level and suggests charging facilities as needed. The input is battery level data, and the output is a charging suggestion notification to the user. When the user accepts the suggestion, the route to the nearest charging facility is recalculated. This process generates new route information to the charging facility.

[1509] Step 7:

[1510] Upon arrival at a charging station, the terminal automatically processes payment and begins charging. Inputs are data indicating arrival at the charging station and payment information, while outputs are the payment processing result and a charging start notification. This process allows users to complete charging without any effort on their part.

[1511] Step 8:

[1512] The server acquires information in real time from the National Police Agency and traffic control systems and transmits it to terminals. Inputs include traffic signal information and emergency information, while output is the results of the transmission to the terminals. The terminals use this information to control autonomous driving and respond to emergencies. This process ensures safe and efficient operation.

[1513] In this way, vehicle systems can significantly improve user convenience and safety through advanced autonomous driving and entertainment provision, automated charging and payment, and real-time integration of traffic information.

[1514] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1515] This invention relates to a vehicle system that integrates autonomous driving, generative artificial intelligence, and an emotion engine. This system provides the optimal route based on user voice input and achieves autonomous driving using the latest map data and traffic information. Furthermore, it offers high levels of convenience and safety to the user by providing charging station guidance and payment processing, entertainment, coordinating with police and traffic control systems, and recognizing the user's emotions.

[1516] Program processing and function description

[1517] Speech recognition and analysis

[1518] User: When boarding the vehicle, you can use voice commands to tell the generated artificial intelligence your destination. For example, you might say, "I want to go to Yokohama."

[1519] Terminal: The voice recognition module analyzes the user's voice and extracts destination information. The extracted information is then re-notified to the user via voice for confirmation.

[1520] Route calculation and information retrieval

[1521] Terminal: Sends destination information to the server. The server retrieves the latest map information from the map database and calculates the optimal route based on real-time traffic information. The calculation result is then sent back to the vehicle.

[1522] Autonomous driving

[1523] Terminal: Based on the received route information, it initiates autonomous driving. The generating AI starts the engine and begins operation. During operation, sensors and cameras constantly monitor the surrounding environment, and the generating AI dynamically adjusts the route.

[1524] Entertainment provider

[1525] User: Instructs the generating AI to "play music" while the vehicle is in operation.

[1526] Terminal: Analyzes voice commands, retrieves streaming data from the user's smartphone, and plays it. The in-car audio system activates and plays the desired music.

[1527] Information and payment for charging facilities

[1528] Device: Monitors battery level and suggests charging locations as needed. For example, it might notify the user, "You need to charge at the next rest stop." Once the user approves the suggestion, the device recalculates the route to the nearest charging location.

[1529] Terminal: Upon arrival at a charging station, the terminal automatically initiates the charging process. It processes payment in conjunction with the charging station's system and confirms that charging is started correctly. After charging is complete, it notifies the user and resumes operation.

[1530] Integration with traffic control systems

[1531] Server: Connects to the National Police Agency and traffic control systems to receive real-time information on traffic signal timings and emergency information.

[1532] Terminal: By reflecting the received traffic control information into the generating AI, it enables safe and efficient autonomous driving to continue.

[1533] Emotion recognition and response

[1534] The device features an emotion engine that analyzes the user's facial expressions and voice tone. The emotion engine recognizes the user's emotions and provides that information to the generating AI.

[1535] Terminal: Selects the appropriate driving mode based on the user's emotions. For example, if the user is feeling stressed, it will select a relaxing driving mode.

[1536] Terminal: Evaluates the user's stress level and fatigue, and suggests breaks as needed. Based on the evaluation results, it recalculates the route to rest points and suggests it to the user.

[1537] Device: Selects and suggests entertainment based on the user's mood. For example, if the user is in a good mood, it suggests rhythmic music. Conversely, if the user is sad, it suggests relaxing music.

[1538] Specific example

[1539] For example, if a user is traveling from Shinjuku, Tokyo to Yokohama, the following scenario will be executed.

[1540] The user gets into the vehicle and gives voice instructions to the AI ​​generator, saying, "Please take me to Yokohama."

[1541] The terminal analyzes the voice command and requests route calculation from the server.

[1542] The server calculates the optimal route, and the generating AI starts the automated driving process.

[1543] When a user requests "play music" during the ride, the AI ​​generates music data from the smartphone and plays it.

[1544] The device monitors the battery level and suggests charging facilities if charging is needed.

[1545] Once the user accepts the proposal and arrives at the charging station, the generating AI automatically processes the payment and starts charging.

[1546] The server provides traffic control information, and the generating AI adjusts the route in real time.

[1547] The device analyzes the user's facial expressions and voice, and if it determines that the user is tired, it suggests taking a break and guides them to an appropriate resting place.

[1548] Upon arrival in Yokohama, the AI ​​generates the data and terminates the operation, notifying the user.

[1549] In this way, the present invention can significantly improve user convenience, safety, and efficiency.

[1550] The following describes the processing flow.

[1551] Step 1:

[1552] When a user boards the vehicle, the terminal automatically activates its AI. A passenger sensor detects the user's presence, and a Bluetooth module scans the user's smartphone to attempt a connection. Once a Bluetooth connection is established, the user is notified.

[1553] Step 2:

[1554] The user gives voice instructions to the AI ​​generating the destination. For example, they might say, "I want to go to Yokohama." The device uses a voice recognition module to analyze the user's voice and extract the destination "Yokohama." The extracted information is then re-notified to the user via voice for confirmation.

[1555] Step 3:

[1556] The terminal sends destination information to the server. The server retrieves the latest map information from the map database, collects real-time traffic information, and calculates the optimal route. The calculation result is then sent back to the terminal.

[1557] Step 4:

[1558] The system initiates autonomous driving based on route information received by the terminal. The generating AI starts the engine and begins operation. During operation, various sensors and cameras constantly monitor the surrounding environment, and the generating AI dynamically adjusts the route.

[1559] Step 5:

[1560] The user gives a voice command, such as "I want to play music," while driving. The terminal analyzes the voice command and streams music data from the user's smartphone. The in-car audio system activates and plays the requested music.

[1561] Step 6:

[1562] The device monitors the battery level and, if it determines that charging is needed, suggests the most suitable charging location to the user. It notifies the user, "You need to charge at the next rest stop." Once the user accepts the suggestion, the device recalculates the route to the nearest charging location.

[1563] Step 7:

[1564] When the device arrives at the charging station, it automatically initiates the charging process. It processes payment in conjunction with the charging station's system and confirms that charging has started correctly. After charging is complete, it notifies the user and resumes operation.

[1565] Step 8:

[1566] The server connects to the National Police Agency and the traffic control system, receiving real-time information on traffic signal timings and emergency situations. The terminal then incorporates the received traffic control information into the generating AI, ensuring safe and efficient autonomous driving continues.

[1567] Step 9:

[1568] The device uses an emotion engine that analyzes the user's facial expressions and voice tone to recognize their emotions. If the user is stressed, the emotion engine provides that information to the generating AI, which then selects a relaxing driving mode.

[1569] Step 10:

[1570] The system assesses the user's stress level and fatigue, and suggests breaks as needed. Once the user approves the suggestion, the device recalculates the route to an appropriate rest stop and notifies the user.

[1571] Step 11:

[1572] The device selects entertainment based on the user's mood. For example, if the user is in a good mood, it suggests rhythmic music; conversely, if the user is sad, it suggests relaxing music.

[1573] Step 12:

[1574] Upon arrival at the destination, the terminal notifies the user that they have arrived at their destination. The generating AI stops autonomous driving and turns off the engine. When the user leaves the vehicle, the passenger sensor detects this and automatically shuts down the generating AI and other systems.

[1575] (Example 2)

[1576] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1577] Many autonomous driving systems currently on the market can recognize user voice commands, calculate routes, and drive autonomously, but they lack the ability to recognize user emotions and adjust driving modes accordingly. Furthermore, there is a need for a variety of functions to enhance user convenience, such as charging station suggestions and automatic payment, real-time traffic signal information, and entertainment features. In addition, the lack of technology to integrate and execute these functions makes improving the user experience a challenge.

[1578] The identification processing performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server has a generating artificial intelligence installed in the vehicle and includes means for recognizing and analyzing the user's voice instructions, means for communicating with the server to acquire the latest map data and traffic information and calculate the optimal route, means for performing automatic driving based on the calculated route information, means for analyzing the user's facial expressions and voice tone to recognize emotions and providing that information to the generating artificial intelligence, and means for selecting a driving mode based on the user's emotions. This makes it possible to adjust driving according to the user's emotions, significantly improving driving safety and comfort. Furthermore, by integrating and executing multiple functions such as suggesting charging facilities and automatic payment, reflecting real-time traffic signal information, and providing entertainment, user convenience and the quality of the experience can be enhanced.

[1579] "Generative artificial intelligence installed in a vehicle" refers to artificial intelligence installed in the vehicle's computer system that recognizes and analyzes user voice commands and performs autonomous driving and other vehicle operation management.

[1580] "User voice commands" refers to the act of a user giving instructions by voice regarding vehicle operation, setting a destination, etc.

[1581] "Means for recognizing and analyzing voice commands" refers to the collective term for hardware and software that capture, analyze, and understand user voice commands.

[1582] A "server" refers to a remote computer system that receives data transmitted from a vehicle, processes it, and sends the results back to the vehicle.

[1583] "Latest map data" refers to data that includes road and topographic information updated in real time using a geographic information system.

[1584] "Traffic information" refers to information about current road conditions, including dynamic data such as congestion, traffic accidents, and road construction.

[1585] "Means for calculating the optimal route" refers to algorithms and software used to calculate the most efficient and safe route to the user's destination.

[1586] "Means of autonomous driving" refers to an integrated system of hardware and software that enables autonomous operation of a vehicle.

[1587] "Means of recognizing emotions by analyzing a user's facial expressions and voice tone" refers to hardware and software that analyze the characteristics of a user's face and voice to determine their emotional state.

[1588] "Means for selecting a driving mode based on user emotions" refers to a control system that selects the optimal driving mode according to the recognized emotions of the user.

[1589] "Means of monitoring battery level" refers to sensors and software that continuously monitor the vehicle's battery status and provide necessary information.

[1590] "A means of suggesting charging facilities and recalculating the route to those facilities when charging is needed" refers to technology that, when the battery level drops, suggests the nearest charging facility to the user and calculates and suggests the optimal route to that facility.

[1591] "A means of automatically processing payment upon arrival at a charging facility" refers to a system that automatically completes payment when charging begins at a charging station.

[1592] "Means for recognizing music and video playback commands and acquiring streaming data from the user's smartphone for playback" refers to technology that recognizes a user's command to play music or videos, acquires the necessary data from the smartphone, and plays the content.

[1593] "Means of receiving traffic signal information and emergency information in cooperation with the National Police Agency and traffic control systems and reflecting it in autonomous driving" refers to technology that exchanges data in real time with the police and traffic control centers, receives information on changes in traffic signals and emergencies, and applies it to autonomous driving systems.

[1594] This invention relates to a vehicle system that integrates autonomous driving, generative artificial intelligence, and an emotion engine. This system provides the optimal route based on the user's voice input and achieves autonomous driving using the latest map data and traffic information. It also provides a high level of convenience and safety for the user by guiding them to charging stations and processing payments, providing entertainment, coordinating with police and traffic control systems, and recognizing user emotions.

[1595] Speech recognition and analysis

[1596] The user boards the vehicle and gives voice commands such as, "Please take me to Yokohama."

[1597] The terminal (in-vehicle system) captures audio using a microphone mounted on the vehicle and sends it to a speech recognition module (e.g., Google Cloud Speech-to-Text).

[1598] The device uses a speech recognition module to convert the voice data into text and extracts the destination information, "Yokohama." This information is then confirmed by the user via voice.

[1599] Route calculation and traffic information acquisition

[1600] The device sends the confirmed destination information to the server. It executes an API request to the server's URL and sends request data including the destination.

[1601] The server retrieves the latest map information and current traffic information from a map database (e.g., Google Maps API).

[1602] The server calculates the optimal route based on the information it has acquired, and this calculation uses real-time traffic data. The calculation result is returned to the terminal as an API response.

[1603] Autonomous driving

[1604] The terminal passes the route information it receives to a generating AI (e.g., Tesla's AutoPilot).

[1605] The terminal uses generated AI to start the engine and begin autonomous driving. During operation, it uses a 100-directional sensor and cameras to monitor the surrounding environment in real time.

[1606] The terminal uses generated AI to analyze data from sensors and cameras, and adjusts the route and driving actions as needed.

[1607] Entertainment provider

[1608] The user can give a voice command to "play music" while the vehicle is in operation.

[1609] The device analyzes the voice command and uses the API of a music streaming service (e.g., Spotify) to retrieve music data from the smartphone.

[1610] The device activates the in-car audio system and plays the acquired music data.

[1611] Information on charging facilities and payment processing

[1612] The device constantly monitors the battery level, and when the battery level drops below 20%, it notifies the user that "charging is needed at the next rest stop."

[1613] The user approves the notification.

[1614] The device sends another API request to the server to search for the nearest charging station.

[1615] The server provides information on the nearest charging station, recalculates the route, and sends it back to the terminal.

[1616] Upon arrival at the charging station, the system automatically initiates the charging process and handles payment in conjunction with the charging station's system. Once charging is complete, the user is notified by voice, and the vehicle resumes operation.

[1617] Integration with traffic control systems

[1618] The server connects in real time with the National Police Agency and traffic control systems to receive traffic signal timings and emergency information.

[1619] The server sends the received information to the terminal.

[1620] The terminal uses AI generation to dynamically reflect traffic control information, ensuring safe operation continues.

[1621] Emotion recognition and adaptive responses

[1622] The device is equipped with an emotion engine (e.g., Microsoft Azure Cognitive Services) that analyzes the user's facial expressions and voice tone.

[1623] The device analyzes the user's emotions from voice and camera data and recognizes that they are experiencing stress.

[1624] The device uses AI generation to select a relaxing driving mode based on the user's emotional information.

[1625] If the device detects that the user is stressed, it will suggest and play relaxing music. It will also suggest taking a break if it detects fatigue.

[1626] If the user agrees to the suggestion, the device calculates and guides them to the optimal resting place.

[1627] Specific examples and examples of prompt sentences for generative AI models

[1628] For example, if a user is traveling from Shinjuku, Tokyo to Yokohama:

[1629] The user gives a voice command saying, "Please take me to Yokohama."

[1630] The terminal analyzes the voice command and requests route calculation from the server.

[1631] The server calculates the optimal route and sends it back to the terminal.

[1632] The device initiates autonomous driving, monitoring the surroundings in real time while driving.

[1633] When the user requests that the device play music, the device will play music from the smartphone.

[1634] The device monitors the battery level and suggests charging facilities as needed. If the user agrees, it will guide them to the charging facility and automatically process payment and begin charging upon arrival.

[1635] The server transmits traffic control information to the terminal, which then adjusts its driving to ensure safety.

[1636] The device analyzes the user's emotions and, if necessary, provides guidance on relaxing driving modes or taking breaks.

[1637] Specific examples of prompt statements for generative AI models:

[1638] "Calculate the optimal route to our destination, Yokohama, and ensure safe and efficient driving."

[1639] "Please explain the procedure for analyzing user voice commands and providing entertainment."

[1640] "If the current battery level is below 20%, please explain how to suggest the nearest charging station and process the payment automatically."

[1641] "Please explain the process for recognizing user emotions and making suggestions to reduce stress."

[1642] The specific embodiments of the present invention have now been described. This embodiment significantly improves user safety and convenience.

[1643] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1644] Step 1: Receiving and analyzing voice input

[1645] Input: The user gives a voice command inside the vehicle saying, "Please take me to Yokohama."

[1646] Terminal: A microphone inside the vehicle captures the user's voice and sends the audio data to a speech recognition module (e.g., Google Cloud Speech-to-Text). The speech recognition module analyzes the audio data and extracts destination information such as "Yokohama".

[1647] Output: Destination information is returned to the terminal as text data.

[1648] Operation: The device confirms the voice analysis results with the user via voice. It asks the user, "Is Yokohama your destination?"

[1649] Step 2: Route calculation and acquisition of traffic information

[1650] Input: The terminal is given information about the destination, "Yokohama".

[1651] Terminal: Sends destination information to the server. The data, including the destination, is sent to the server as an API request.

[1652] Server: Retrieves the latest map information and real-time traffic information from a map database (e.g., Google Maps API). This information is used to calculate the optimal route.

[1653] Output: The calculation result is sent back to the terminal as route information.

[1654] Operation: The terminal receives route information calculated by the server and uses it in subsequent steps.

[1655] Step 3: Start autonomous driving

[1656] Input: Route information is provided to the terminal.

[1657] Terminal: Passes the received route information to the generating AI (e.g., Tesla's AutoPilot).

[1658] Terminal: The generating AI starts the engine and begins autonomous driving. It monitors the surrounding environment in real time using a 100-directional sensor and camera.

[1659] Output: The start state for autonomous driving mode is generated.

[1660] Operation: The generating AI analyzes data from sensors and cameras and adjusts the route and driving actions in real time.

[1661] Step 4: Providing Entertainment

[1662] Input: The user gives a voice command saying, "Please play music."

[1663] Terminal: Analyzes voice commands and retrieves music data from the user's smartphone using the API of a music streaming service (e.g., Spotify).

[1664] Terminal: Activates the in-car audio system and plays the acquired music data.

[1665] Output: Music playback begins.

[1666] Operation: The device streams music data based on voice commands and plays it on the audio system.

[1667] Step 5: Guidance to charging facilities and payment processing

[1668] Input: The device monitors the battery level, and when the battery level falls below 20%, it triggers a response.

[1669] Device: Notifies the user that "charging is required at the next rest stop."

[1670] User: Accept notification.

[1671] Terminal: Sends another API request to the server to search for the nearest charging station.

[1672] Server: Provides information on the nearest charging station, recalculates the route, and sends it back to the terminal.

[1673] Terminal: Upon arrival at the charging station, it automatically initiates the charging process and handles payment processing in conjunction with the charging station's system.

[1674] Output: Charging and payment processing are complete.

[1675] Operation: The device starts the charging process, notifies the user when it is complete, and resumes operation.

[1676] Step 6: Integration with traffic control systems

[1677] Input: The server receives real-time traffic signal information and emergency information from the National Police Agency and traffic control systems.

[1678] Server: Sends received information to the terminal.

[1679] Terminal: The generating AI dynamically reflects traffic control information, ensuring safe operation continues.

[1680] Output: The operational status is updated based on traffic control information.

[1681] Operation: The generating AI reflects traffic signal information in real time and adjusts driving accordingly.

[1682] Step 7: Emotion Recognition and Adaptive Response

[1683] Input: The user's facial expressions and voice tone are captured by the device.

[1684] Terminal: An emotion engine (e.g., Microsoft Azure Cognitive Services) analyzes the user's facial expressions and voice tone to recognize emotions.

[1685] Terminal: Provides emotional information to the AI ​​that generates it.

[1686] Generative AI: Selects a relaxing driving mode based on the user's emotions. Suggests and plays relaxing music as needed.

[1687] Output: The system initiates driving modes and provides entertainment tailored to the user's mood.

[1688] Operation: The generated AI selects a driving mode based on the user's emotional information and adjusts the driving to promote relaxation. It suggests breaks as needed and guides the user to the optimal rest stop.

[1689] (Application Example 2)

[1690] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1691] In modern autonomous vehicles, it is crucial to accurately recognize user voice commands and provide optimal routes using up-to-date map data and traffic information. However, in addition to this, it is also necessary to handle charging station guidance and payment processing, appropriate driving modes and entertainment based on user emotion recognition, and the reflection of real-time traffic control information. Conventional systems lack these comprehensive functions, resulting in challenges in ensuring sufficient user convenience and safety.

[1692] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a generating artificial intelligence installed in the vehicle and means for recognizing and analyzing the user's voice instructions, means for communicating with the server to acquire the latest map data and traffic information and calculate the optimal route, means for performing autonomous driving based on the calculated route information, means for having an emotion recognition engine to recognize emotions from the user's facial expressions and voice tone and provide driving modes and entertainment based on that, and means for guiding users to charging facilities and processing payments. This makes it possible to significantly improve user convenience and safety by providing the optimal route based on the user's voice instructions, recognizing and adapting to the situation during autonomous driving, guiding users to charging stations and processing payments, and providing entertainment that responds to emotions.

[1693] "Generative artificial intelligence" is an artificial intelligence that recognizes and analyzes user voice commands and generates the information necessary for decision-making.

[1694] "Map data" refers to data that represents geographical information such as roads, buildings, and topography in a digital format.

[1695] "Traffic information" refers to data that shows the current traffic conditions, including real-time information such as traffic congestion and accidents.

[1696] The "optimal route" is the most efficient path between a designated starting point and a destination, taking into account factors such as travel time, distance, and traffic conditions.

[1697] "Autonomous driving" is a technology that allows vehicles to move autonomously without driver intervention.

[1698] An "emotion recognition engine" is a technology that analyzes a user's facial expressions and voice tone to identify their emotional state.

[1699] "Entertainment provision" refers to the function of playing music, videos, and other entertainment content for users.

[1700] The "Charging Station Guide" is a function that displays the nearest suitable charging station to the user based on the vehicle's b...

Claims

1. It has a generating artificial intelligence installed inside the vehicle, and means for recognizing and analyzing the user's voice instructions, A means of communicating with a server to obtain the latest map data and traffic information, and calculating the optimal route, A means of performing autonomous driving based on calculated route information, A system that includes this.

2. A means to monitor the battery level, suggest charging facilities, and recalculate the route to those facilities if charging is needed, A method of automatically processing payment upon arrival at the charging station, The system according to claim 1, further comprising:

3. A means for recognizing music and video playback commands, and acquiring and playing streaming data from the user's smartphone, In cooperation with the National Police Agency and the traffic control system, a means of receiving traffic signal information and emergency information and reflecting it in autonomous driving, The system according to claim 1, further comprising:

Images