System

Abstract

To provide a driving support system combining generative AI and a voice recognition system.SOLUTION: A system provides advice of a driving operation, detects fatigue or concentration reduction of a driver and alerts the driver or proposes rest. Further, a destination is set by verbally interacting with generative AI during driving and recommendation is made based on likes and tastes of the driver. Moreover, an entertainment content such as music, radio, audio book, etc., is provided. Thus, safety and comfort of the driver can be improved.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Current driving assistance systems have difficulty detecting driver fatigue and loss of concentration and providing appropriate advice and break suggestions. Furthermore, they lack functionality to improve driver comfort and satisfaction, such as setting destinations while driving, providing recommendations tailored to the driver's hobbies and preferences, and providing entertainment content. [Means for solving the problem]

[0005] We provide a driving assistance system that combines generative AI and a voice recognition system. This system provides driving operation advice, detects driver fatigue or loss of concentration, and issues alerts and suggests breaks. Furthermore, while driving, the system allows the driver to set a destination through verbal interaction with the generative AI, and makes recommendations tailored to the driver's hobbies and preferences. It also provides entertainment content such as music, radio, and audiobooks. This can improve driver safety and comfort. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment Example 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] One embodiment of the present invention is a driving assistance system that combines generative AI and a voice recognition system. This system provides advice to the driver regarding their driving operations. For example, when a driver attempts to change lanes, the AI ​​determines the approaching speed and distance of the vehicle behind and provides voice advice on whether the lane change is safe.

[0029] "Example 2"

[0030] Furthermore, this system detects driver fatigue or loss of concentration and issues an alert or suggests taking a break. Specifically, it analyzes the driver's line of sight, facial expressions, driving patterns, etc. to detect fatigue or loss of concentration. If detected, the AI ​​will recommend an appropriate rest spot and suggest taking a break via voice. "Example 3"

[0031] This system also allows the driver to set a destination through verbal interaction with the generative AI while driving. If the driver says, "I want to go to the nearest gas station," the AI ​​will search for the nearest gas station and begin navigation. It also makes recommendations based on the driver's tastes and preferences. For example, it can learn the driver's favorite music genres and artists and automatically play that music while driving.

[0032] "Example 4"

[0033] In addition, the system provides entertainment content such as music, radio, and audiobooks. If the driver says, "I want to listen to an audiobook," the AI ​​will resume the audiobook the driver was previously listening to or recommend a new audiobook, allowing the driver to enjoy entertainment while driving.

[0034] The processing flow of each embodiment will be described below.

[0035] "Example 1"

[0036] Step 1: When a driver attempts to change lanes, the generative AI determines the approaching speed and distance of the vehicle behind.

[0037] Step 2: Based on the AI's judgment, the driver is given audio advice on whether a safe lane change is possible.

[0038] "Example 2"

[0039] Step 1: Analyze the driver's gaze, facial expressions, driving patterns, etc. to detect fatigue and loss of concentration.

[0040] Step 2: If fatigue or a lack of concentration is detected, the AI ​​will recommend an appropriate rest spot. Step 3: The AI ​​will suggest a rest spot to the driver via voice.

[0041] "Example 3"

[0042] Step 1: When the driver says, "I want to go to the nearest gas station," the AI ​​searches for the nearest gas station.

[0043] Step 2: Start navigation with the searched gas station as your destination.

[0044] Step 3: Make recommendations based on the driver's tastes and preferences. For example, learn the driver's favorite music genres and artists and automatically play that music while driving.

[0045] "Example 4"

[0046] Step 1: When the driver says, "I want to listen to an audiobook," the AI ​​will resume the audiobook the driver was previously listening to or recommend a new audiobook. Step 2: The recommended audiobook will be provided to the driver via audio, allowing the driver to enjoy entertainment while driving.

[0047] Example 1

[0048] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0049] Conventional driver assistance systems do not provide sufficient real-time advice to drivers on their driving operations. They also lack the ability to detect driver fatigue or loss of concentration and provide appropriate alerts or suggest rest breaks. Furthermore, there is a need for a system that effectively utilizes the driver's voice input to provide driving operation advice.

[0050] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0051] In this invention, the server includes means for providing driving operation advice, means for detecting driver fatigue or loss of concentration and issuing an alert or suggesting a break, means for capturing voice input from the driver and converting the voice data into text data, means for analyzing the text data and understanding the driver's intentions, means for acquiring environmental data from vehicle sensors, means for generating advice using generative artificial intelligence based on the acquired environmental data, and means for converting the generated advice into voice and providing it to the driver. This makes it possible to support the driver in safe driving operations in real time.

[0052] "Generative AI" is an AI technology that has the ability to generate new information and advice based on data.

[0053] A "voice recognition system" is a technology that converts voice data into text data.

[0054] A "driving assistance system" is a system that provides advice and support to drivers regarding driving operations.

[0055] The "means for providing driving operation advice" is a function for generating and providing appropriate advice for the driver's driving operation.

[0056] "Means for detecting driver fatigue or loss of concentration and issuing alerts or suggesting breaks" refers to a function that monitors the driver's condition and issues a warning or suggests a break when fatigue or loss of concentration is detected.

[0057] The "means for capturing the driver's voice input and converting the voice data into text data" is a technology for collecting the driver's voice and converting it into text format.

[0058] The "means for analyzing text data and understanding the driver's intention" is a function for analyzing the converted text data and understanding the driver's intention.

[0059] "Means for acquiring environmental data from vehicle sensors" refers to technology for collecting surrounding environmental data using sensors mounted on a vehicle.

[0060] "Means for generating advice using generative artificial intelligence" is a function that generates appropriate advice using generative artificial intelligence based on collected data.

[0061] The "means for converting the generated advice into audio and providing it to the driver" is a technology for converting the generated advice into audio format and communicating it to the driver.

[0062] This invention is a driving assistance system that combines generative artificial intelligence and a voice recognition system, and provides real-time driving advice to the driver. Specific embodiments of this system will be described below.

[0063] Hardware and software used

[0064] Hardware: Microphones, speakers, rear camera, and radar sensors installed in the vehicle

[0065] Software: Generative artificial intelligence models (e.g., GPT-4 (registered trademark)), speech recognition systems (e.g., Google (registered trademark) Cloud Speech-to-Text), speech synthesis systems (e.g., Google Cloud Text-to-Speech)

[0066] System configuration

[0067] 1. Capture voice input

[0068] The device uses a microphone installed inside the vehicle to capture the driver's voice in real time.

[0069] Specifically, the driver says, "I want to change lanes."

[0070] 2. Speech Recognition and Text Conversion

[0071] The device sends the captured audio data to the Google Cloud Speech-to-Text API.

[0072] The server receives the voice data and converts it into text data.

[0073] Specifically, the speech "I want to change lanes" is converted into text "I want to change lanes."

[0074] 3. Intent Analysis

[0075] The server analyzes the converted text data and understands the driver's intentions.

[0076] Specifically, from the text "I want to change lanes," the system recognizes that the driver wants to change lanes.

[0077] 4. Acquisition of environmental data

[0078] The server collects data in real time from sensors installed in the vehicle (rear camera and radar).

[0079] Specifically, data on the approaching speed and distance of the vehicle behind is acquired.

[0080] 5. Generating Advice

[0081] The server generates advice using a generative artificial intelligence model (e.g., GPT-4) based on the acquired environmental data.

[0082] Specifically, the advice generated is "A vehicle behind is approaching. Changing lanes is dangerous."

[0083] 6. Audio output of advice

[0084] The device converts the generated advice into audio using the Google Cloud Text-to-Speech API.

[0085] The terminal provides the converted audio to the driver through the car's speakers.

[0086] Specifically, the driver hears a voice saying, "A vehicle behind is approaching. Changing lanes is dangerous."

[0087] Specific examples

[0088] As a specific example, consider the case where a driver attempts to change lanes.

[0089] The user says, "I want to change lanes."

[0090] The device captures this audio with its microphone and sends it to the Google Cloud Speech-to-Text API.

[0091] The server receives the voice data and converts it into text saying "I want to change lanes."

[0092] The server parses this text and recognizes that the driver wants to change lanes.

[0093] The server acquires data on the approaching speed and distance of the vehicle behind from the vehicle's rear camera and radar.

[0094] The server uses a generative artificial intelligence model to generate advice such as, "A vehicle behind is approaching. Changing lanes is dangerous."

[0095] The device converts this advice into audio using the Google Cloud Text-to-Speech API and provides it to the driver through the car's speakers.

[0096] Prompt Sentence Examples

[0097] "When a driver attempts to change lanes, the system determines whether the lane change is safe based on the approaching speed and distance of the vehicle behind, and generates appropriate advice."

[0098] In this way, the driver assistance system can support the driver in safe driving operations in real time.

[0099] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0100] Step 1:

[0101] Capturing voice input

[0102] The device uses a microphone installed inside the vehicle to capture the driver's voice in real time.

[0103] Input: Driver's voice (e.g., "I want to change lanes")

[0104] Output: Captured audio data

[0105] Specific operation: When the driver says, "I want to change lanes," the microphone picks up the voice and sends it to the terminal as voice data.

[0106] Step 2:

[0107] Speech recognition and text conversion

[0108] The terminal transmits the captured voice data to a voice recognition system.

[0109] The server converts the voice data into text data using a voice recognition system.

[0110] Input: Captured audio data

[0111] Output: Converted text data (e.g., "I want to change lanes")

[0112] How it works: The device sends voice data to the Google Cloud Speech-to-Text API, and the server converts the voice data into text, such as "I want to change lanes."

[0113] Step 3:

[0114] Intention Analysis

[0115] The server analyzes the converted text data and understands the driver's intentions.

[0116] Input: Converted text data (e.g., "I want to change lanes")

[0117] Output: Driver's intention (e.g., desire to change lanes)

[0118] Specific operation: The server analyzes the text data and recognizes the intention to "change lanes."

[0119] Step 4:

[0120] Obtaining environmental data

[0121] The server collects data in real time from sensors installed in the vehicle (rear camera and radar).

[0122] Input: Environmental data from sensors (e.g. approaching speed and distance of rear vehicles)

[0123] Output: Acquired environmental data

[0124] Specific operation: The server collects data on the approach speed and distance of the vehicle behind from the rear camera and radar.

[0125] Step 5:

[0126] Generating Advice

[0127] The server generates advice using a generative artificial intelligence model based on the acquired environmental data.

[0128] Input: Driver's intention, environmental data

[0129] Output: Generated advice (e.g. "A vehicle is approaching from behind. Changing lanes is dangerous.")

[0130] Specific operation: The server uses a generative artificial intelligence model to consider the approaching speed and distance of the vehicle behind and generates advice such as, "A vehicle behind is approaching. Changing lanes is dangerous."

[0131] Step 6:

[0132] Audio output of advice

[0133] The terminal converts the generated advice into speech using a speech synthesis system.

[0134] The terminal provides the converted audio to the driver through the car's speakers.

[0135] Input: Generated advice (in text format)

[0136] Output: Voice advice (e.g. "A vehicle behind you is approaching. Changing lanes is dangerous.")

[0137] Specific operation: The device converts the generated advice into audio using the Google Cloud Text-to-Speech API and provides it to the driver through the car's speakers.

[0138] (Application example 1)

[0139] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0140] Conventional driver assistance systems offer limited advice to drivers on their driving operations, and lack specific support for ensuring safety, particularly when changing lanes or turning left or right. They also lack the ability to detect driver fatigue or loss of concentration and provide alerts or suggestions for breaks at appropriate times. Furthermore, they lack the ability to recognize the driver's voice commands and provide the necessary information, resulting in low convenience while driving.

[0141] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0142] In this invention, the server is a driving assistance system that combines a generative AI and a voice recognition system, and includes: means for providing driving operation advice; means for detecting driver fatigue or loss of concentration and issuing an alert or suggesting a break; means for acquiring data in real time from the vehicle's cameras and sensors and analyzing it with the generative AI; means for determining the approaching speed and distance of a vehicle behind the driver when the driver attempts to change lanes or turn right or left, and providing voice advice on whether safe operation is possible; and means for recognizing the driver's voice commands and providing necessary information. This enables the driver to perform driving operations safely and efficiently, improving convenience and safety while driving.

[0143] "Generative AI" is an artificial intelligence technology that generates new information and predictions based on data.

[0144] A "voice recognition system" is a technology that analyzes voice and converts it into text data.

[0145] A "driving assistance system" is a system that assists the driver in driving operations and improves safety and convenience.

[0146] "Driving operation advice" refers to providing appropriate instructions and advice to the driver regarding operations.

[0147] "Detecting driver fatigue and loss of concentration" refers to monitoring the driver's condition and identifying fatigue and loss of concentration.

[0148] "Alerts and rest suggestions" refers to notifications that alert the driver or encourage them to take a break.

[0149] "Acquiring data in real time from the vehicle's cameras and sensors" means collecting information on the current situation in real time using cameras and sensors installed in the vehicle.

[0150] "Analysis by generative AI" means that generative AI processes the collected data and provides the necessary information and judgments.

[0151] "Judgment of approaching speed and distance of rear vehicles" means measuring the speed and distance of vehicles approaching from behind and evaluating safety.

[0152] "Voice advice" means providing voice instructions and advice to the driver based on the analysis results.

[0153] "Recognizing driver's voice commands" means analyzing the voice instructions given by the driver and understanding their content.

[0154] "Providing necessary information" means providing the driver with the information they require in an appropriate manner.

[0155] The system for carrying out this invention is a driving assistance system that combines generative AI and a voice recognition system. Specific embodiments of this system will be described below.

[0156] Hardware and software used

[0157] Hardware:

[0158] Camera mounted on the vehicle

[0159] Sensors installed in the vehicle

[0160] Microphone for voice recognition

[0161] Smartphone (for running applications)

[0162] software:

[0163] OpenCV (camera image acquisition)

[0164] TENSORFLOW (registered trademark) (Analysis of generative AI models)

[0165] SpeechRecognition

[0166] gTTS (speech synthesis)

[0167] System processing overview

[0168] 1. Data Acquisition:

[0169] The system obtains real-time data from cameras and sensors installed in the vehicle. The cameras capture images of the vehicle behind, and the sensors collect information such as the vehicle's speed and distance.

[0170] 2. Data Analysis:

[0171] The acquired data is fed into a generative AI model to analyze the approaching speed and distance of the vehicle behind. Using TensorFlow, this data is processed in real time to determine whether a safe maneuver is possible.

[0172] 3. Audio advice:

[0173] Based on the analysis results, the system provides audio advice to the driver. For example, if it is safe to change lanes, it will notify the driver by saying "It is safe to change lanes," and if it is dangerous, it will warn the driver by saying "It is dangerous to change lanes." The audio is generated using gTTS and output from the speaker.

[0174] 4. Voice command recognition:

[0175] The system recognizes the driver's voice commands and provides the necessary information. It uses SpeechRecognition to analyze the voice instructions given by the driver and understand their content. For example, if the driver instructs the system to "turn right at the next intersection," the system will respond with "turn right at the next intersection."

[0176] Specific examples

[0177] Lane change advice:

[0178] When a driver attempts to change lanes, the system analyzes the approaching speed and distance of the vehicle behind and provides voice advice such as "Changing lanes is safe" or "Changing lanes is dangerous."

[0179] Voice command recognition and response:

[0180] When the driver issues a voice command such as "Turn right at the next intersection," the system responds with "Turn right at the next intersection."

[0181] Prompt Sentence Examples

[0182] "Analyze the approaching speed and distance of the vehicle behind you to determine whether a safe lane change is possible."

[0183] "Recognize the driver's voice commands and provide the necessary information."

[0184] In this way, the driver can perform driving operations safely and efficiently, improving convenience and safety while driving.

[0185] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0186] Step 1:

[0187] The server receives real-time data from the cameras and sensors installed in the vehicle. The cameras capture images of the vehicle behind, and the sensors collect information such as the vehicle's speed and distance. The input is the camera images and sensor data, and the output is a set of these data.

[0188] Step 2:

[0189] The server inputs the acquired data into a generative AI model to analyze the approaching speed and distance of the vehicle behind. Specifically, it uses TensorFlow to analyze camera footage and calculate the position and speed of the vehicle behind. The input is camera footage and sensor data, and the output is information on the approaching speed and distance of the vehicle behind.

[0190] Step 3:

[0191] The server provides audio advice to the driver based on the analysis results. For example, if it is safe to change lanes, it will notify the driver by saying "It is safe to change lanes," and if it is dangerous, it will warn the driver by saying "It is dangerous to change lanes." Voice is generated using gTTS and output from the speaker. The input is information on the approaching speed and distance of the vehicle behind, and the output is audio advice.

[0192] Step 4:

[0193] The server recognizes the driver's voice commands and provides the necessary information. Using SpeechRecognition, the system analyzes the voice instructions given by the driver and understands their content. For example, if the driver instructs the system to "turn right at the next intersection," the system responds with "turn right at the next intersection." The input is the driver's voice command, and the output is a voice response.

[0194] Step 5:

[0195] The server detects driver fatigue or loss of concentration and issues an alert or suggests taking a break. It analyzes data from sensors and monitors the driver's condition. If fatigue or loss of concentration is detected, it notifies the driver by voice, saying "Please take a break." The input is sensor data, and the output is an alert or voice notification suggesting a break.

[0196] Step 6:

[0197] The server makes recommendations tailored to the driver's hobbies and preferences. It analyzes the driver's past behavioral data and voice commands to suggest appropriate entertainment content and destinations. The input is the driver's past behavioral data and voice commands, and the output is recommendation information.

[0198] Step 7:

[0199] The server provides entertainment content such as music, radio, audiobooks, etc. Based on the driver's voice command, it selects and plays the appropriate content. The input is the driver's voice command, and the output is the playback of the entertainment content.

[0200] Example 2

[0201] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0202] Conventional driver assistance systems lack the ability to detect driver fatigue and loss of concentration in real time and suggest rest at appropriate times. This has led to a demand for effective means to ensure driver safety. Furthermore, they are also inadequate in providing recommendations and entertainment content tailored to the driver's hobbies and preferences.

[0203] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0204] In this invention, the server includes means for collecting data on the driver's line of sight, facial expression, and driving operation, means for analyzing the collected data and detecting driver fatigue or loss of concentration, means for recommending appropriate rest spots when fatigue or loss of concentration is detected, and means for suggesting rest by voice, thereby ensuring the safety of the driver and making it possible to suggest rest at appropriate times.

[0205] "Means for collecting data on the driver's gaze, facial expressions, and driving operations" refers to devices that use cameras and sensors installed inside the vehicle to obtain data such as the driver's gaze movements, changes in facial expressions, steering operation, and frequency of brake use in real time.

[0206] "Means for analyzing collected data to detect driver fatigue or loss of concentration" refers to a system that uses image recognition software and machine learning models to analyze collected data and detect signs of driver fatigue or loss of concentration.

[0207] The "means of recommending appropriate rest spots when fatigue or a decrease in concentration is detected" is a system that uses a GPS module to obtain the driver's current location, uses a map information service to search for nearby rest spots, and recommends them to the driver.

[0208] The "means for suggesting a break by voice" is a device that uses voice synthesis software to suggest to the driver by voice that they take a break.

[0209] "Generative AI" is an artificial intelligence system that uses natural language processing and machine learning techniques to generate appropriate responses and suggestions based on user input.

[0210] A "driving assistance system" is a system that provides functions such as driving operation advice, fatigue detection, and rest suggestion to assist drivers in safe driving.

[0211] "A means of making recommendations tailored to the driver's hobbies and preferences" is a system that recommends appropriate destinations and entertainment content based on data on the driver's past behavior and preferences.

[0212] The "means for providing entertainment content" is a device that provides entertainment content such as music, radio, audiobooks, etc. to the driver.

[0213] MODE FOR CARRYING OUT THE INVENTION

[0214] The present invention provides a system for collecting and analyzing data on a driver's line of sight, facial expression, and driving operation in order to ensure the safety of the driver. A specific embodiment of this system will be described below.

[0215] Hardware and software used

[0216] Hardware: Camera, sensors, microphone, GPS module

[0217] Software: Image recognition software (e.g., OpenCV), machine learning models (e.g., TensorFlow), speech synthesis software (e.g., Google Text-to-Speech API)

[0218] Data collection

[0219] The device uses cameras and sensors installed inside the vehicle to collect real-time data on the driver's gaze, facial expressions, and driving behavior. The camera captures the driver's face, and the sensors record steering wheel movements and brake use frequency.

[0220] Data analysis

[0221] The device inputs the collected data into image recognition software (OpenCV) and a machine learning model (TensorFlow) to analyze gaze, facial expressions, and driving behavior patterns. OpenCV analyzes the driver's blink frequency and eye movements, while TensorFlow compares the driving behavior patterns with the learning model.

[0222] Detecting fatigue and loss of concentration

[0223] Based on the analysis results, the device detects driver fatigue and loss of concentration. Increased blinking frequency and frequent averted gaze are detected as signs of fatigue and loss of concentration.

[0224] Recommend rest spots

[0225] The server obtains the current location using a GPS module and searches for nearby rest spots. The server uses a map information service (e.g., Google Maps API) to identify the nearest service area or parking area from the current location.

[0226] Audio break suggestions

[0227] The device uses a generative AI model to recommend suitable rest spots and uses speech synthesis software (Google Text-to-Speech API) to audibly suggest rest stops. The device generates a voice message such as "We recommend you take a break at a nearby service area" and plays it through the speaker.

[0228] Specific examples

[0229] Example 1: A camera detects that a user is blinking more frequently after driving for a long period of time. A sensor detects that the user's steering is becoming unstable. Based on this data, the device detects fatigue, and the server recommends a nearby rest area. The device makes a voice suggestion saying, "We recommend that you take a break at a nearby rest area."

[0230] Example 2: While driving on the highway, a camera detects that a user is frequently looking away. A sensor detects that the user is using the brakes more frequently. Based on this data, the device detects a decrease in concentration, and the server recommends the next parking area. The device suggests audibly, "We recommend that you take a break at the next parking area."

[0231] Prompt Sentence Examples

[0232] "Please explain how you would detect fatigue and suggest a break if the driver's gaze is frequently averted."

[0233] "Please explain what data you collect and how you analyze it to detect driver fatigue from facial expressions."

[0234] To ensure driver safety, the system collects and analyzes data in real time and suggests appropriate breaks.

[0235] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0236] Step 1:

[0237] Data collection

[0238] The device uses cameras and sensors installed inside the vehicle to collect data on the driver's gaze, facial expressions, and driving operations in real time.

[0239] Input: Camera footage, sensor data

[0240] Data processing: Facial features are extracted from camera footage, and frequency of steering and braking operations is recorded from sensor data.

[0241] Output: Gaze data, facial expression data, driving operation data

[0242] How it works: A camera captures the driver's face, and sensors record steering movements and how often the brakes are used.

[0243] Step 2:

[0244] Data analysis

[0245] The device inputs the collected data into image recognition software (OpenCV) and machine learning models (TensorFlow) to analyze patterns of gaze, facial expressions, and driving operations.

[0246] Input: Gaze data, facial expression data, driving operation data

[0247] Data calculation: OpenCV analyzes eye movements and blink frequency, and TensorFlow matches driving behavior patterns with the learning model.

[0248] Output: Gaze analysis results, facial expression analysis results, driving operation analysis results

[0249] How it works: OpenCV analyzes the driver's blink frequency and eye movements, and TensorFlow compares the driving behavior patterns with the learning model.

[0250] Step 3:

[0251] Detecting fatigue and loss of concentration

[0252] Based on the analysis results, the device detects the driver's fatigue and loss of concentration.

[0253] Input: Gaze analysis results, facial expression analysis results, driving operation analysis results

[0254] Data calculations: Detect signs of fatigue or loss of concentration based on blinking frequency, eye movement, and driving behavior patterns.

[0255] Output: Fatigue and concentration loss detection results

[0256] Specific behavior: Increased blinking rate and frequent averted gaze are detected as signs of fatigue and decreased concentration.

[0257] Step 4:

[0258] Recommend rest spots

[0259] The server uses a GPS module to obtain the current location and search for nearby rest spots.

[0260] Input: Current location data, fatigue and concentration loss detection results

[0261] Data calculation: Uses map information services to search for the nearest rest spot from your current location.

[0262] Output: Recommended rest spots

[0263] What happens: The server uses a map service to identify the nearest service or parking area from the current location.

[0264] Step 5:

[0265] Audio break suggestions

[0266] The device uses a generative AI model to recommend suitable rest spots and uses speech synthesis software (Google Text-to-Speech API) to suggest resting audibly.

[0267] Input: Recommended rest spots, fatigue and concentration loss detection results

[0268] Data calculation: Generative AI models select suitable rest spots, and speech synthesis software generates voice messages.

[0269] Output: Voice message

[0270] What happens: The device generates a voice message such as "We recommend you take a break at a nearby service area" and plays it through the speaker.

[0271] (Application example 2)

[0272] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0273] Conventional driving assistance systems lack the ability to detect driver fatigue and loss of concentration in real time and suggest appropriate breaks, which means driver safety is not fully ensured. Furthermore, technology for comprehensively analyzing the driver's gaze, facial expressions, and driving operation patterns is underdeveloped, resulting in low accuracy in detecting fatigue and loss of concentration. Furthermore, if break suggestions after detection are inappropriate, the driver may not recover sufficiently from fatigue, potentially increasing the risk of an accident.

[0274] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0275] In this invention, the server includes a means for analyzing the driver's line of sight, facial expressions, and driving operation patterns to detect fatigue or loss of concentration, a means for recommending appropriate rest spots and suggesting a break by voice when fatigue or loss of concentration is detected, a driving assistance system that combines generative AI and a voice recognition system, a means for providing driving operation advice, and a means for detecting the driver's fatigue or loss of concentration and issuing an alert or suggesting a break. This makes it possible to detect the driver's fatigue or loss of concentration with high accuracy and suggest a break at the appropriate time.

[0276] "Means of detecting fatigue and loss of concentration by analyzing the driver's line of sight, facial expressions, and driving operation patterns" refers to technology that uses cameras and sensors to monitor the driver's line of sight, facial expressions, and steering wheel and pedal operation patterns in real time, and analyzes this data to detect the driver's fatigue and loss of concentration.

[0277] "Means of recommending appropriate rest spots and suggesting a break via voice when detected" refers to a technology in which, when the driver's fatigue or lack of concentration is detected, AI selects the nearest rest spot and suggests a break to the driver via a voice assistant.

[0278] A "driving assistance system that combines generative AI and a voice recognition system" is a system that uses generative AI to generate appropriate responses to the driver's questions and instructions, and provides driving assistance by interacting with the driver through a voice recognition system.

[0279] "Means for providing driving operation advice" refers to technology that monitors the driver's driving operations and provides advice on driving methods and operations in real time as necessary.

[0280] "Means for detecting driver fatigue or loss of concentration and issuing an alert or suggesting a break" refers to technology that issues an alert and suggests taking an appropriate break when it detects that the driver is fatigued or losing concentration.

[0281] In order to implement the present invention, the following hardware and software are used.

[0282] Hardware

[0283] Camera: Used to monitor the driver's gaze and facial expressions.

[0284] Sensors: Used to detect driving patterns.

[0285] Microphone and speaker: Used for voice recognition and voice assistant.

[0286] software

[0287] OpenCV: A library for image processing.

[0288] dlib: A library for face and landmark detection.

[0289] SciPy: A library for performing distance calculations.

[0290] gTTS: A library for speech generation.

[0291] mpg321: Software for audio playback.

[0292] Program processing

[0293] The server uses cameras and sensors to monitor the driver's gaze, facial expressions, and driving behavior patterns in real time. This data is analyzed using OpenCV and dlib to detect driver fatigue and loss of concentration. Specifically, the eye aspect ratio (EAR) is calculated, and fatigue is detected when it falls below a certain threshold.

[0294] If fatigue is detected, the server uses AI to identify the nearest rest spot, and the voice assistant uses gTTS to suggest a break to the driver. The audio is played using mpg321.

[0295] Specific examples

[0296] For example, if a driver has been driving for a long time, the camera will monitor the driver's eye movements and detect a decrease in the aspect ratio of the eyes. At this time, the server will make a voice prompt saying, "Fatigue has been detected. Please take a break."

[0297] Prompt Sentence Examples

[0298] Example prompts to input to a generative AI model:

[0299] I would like to develop a system that detects driver fatigue and loss of concentration and recommends appropriate rest spots. Please tell me how to analyze the driver's gaze, facial expressions, and driving operation patterns to detect fatigue and loss of concentration. Also, please tell me how to suggest a break by voice if fatigue or loss of concentration is detected.

[0300] In this way, a driving assistance system for improving driver safety can be realized.

[0301] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0302] Step 1:

[0303] The server captures the driver's face using a camera. The input is video data from the camera, and the output is an image of the driver's face. Specifically, the camera captures the driver's face in real time and sends the video data to the server.

[0304] Step 2:

[0305] The server uses dlib to detect the driver's face and identify facial landmarks. The input is the driver's face image, and the output is the coordinates of the facial landmarks. Specifically, it applies dlib's face detection algorithm to identify 68 facial landmarks.

[0306] Step 3:

[0307] The server calculates the eye aspect ratio (EAR) using OpenCV. The input is the face landmark coordinates, and the output is the eye aspect ratio. Specifically, the eye landmark coordinates are used to calculate the vertical and horizontal distances of the eyes, and then the EAR is calculated.

[0308] Step 4:

[0309] The server compares the calculated EAR with a threshold to detect fatigue. The input is the eye aspect ratio, and the output is a fatigue detection flag. Specifically, if the EAR is below the set threshold, fatigue is detected.

[0310] Step 5:

[0311] If fatigue is detected, the server uses a generative AI model to recommend the nearest rest spot. The input is a fatigue detection flag and current location information, and the output is information about rest spots. Specifically, the generative AI model selects the optimal rest spot based on the current location information.

[0312] Step 6:

[0313] The server uses gTTS to generate a voice message suggesting a break. The input is information about rest spots, and the output is an audio file. Specifically, the server generates a voice message saying, "Fatigue has been detected. Please take a break."

[0314] Step 7:

[0315] The server plays the generated audio using mpg321. The input is an audio file, and the output is an audio alert. Specifically, the server plays the generated audio file and suggests the driver to take a break.

[0316] In this way, it is possible to detect driver fatigue and loss of concentration with high accuracy and suggest rest breaks at appropriate times.

[0317] Example 3

[0318] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0319] Conventional driving assistance systems require the driver to manually set a destination, which reduces safety while driving. They also lack the ability to provide entertainment content based on the driver's preferences, which reduces driving comfort. Furthermore, drivers must manually control playback when enjoying audiobooks, which reduces convenience.

[0320] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0321] In this invention, the server includes means for acquiring voice input, means for performing voice recognition, means for analyzing voice data using a generative AI model and determining appropriate actions, means for starting navigation, means for providing entertainment content based on the driver's hobbies and preferences, and means for providing audiobooks. This allows the driver to set a destination and receive entertainment content by voice input without performing manual operations while driving.

[0322] "Means for obtaining voice input" refers to a device or function for capturing the driver's voice in real time and transmitting it to the system.

[0323] "Means for performing speech recognition" refers to software or algorithms for converting captured speech data into text data.

[0324] "Means for analyzing voice data using a generative AI model and determining appropriate actions" refers to a system that uses a generative AI model to analyze text data and generate appropriate instructions and actions based on the driver's intentions.

[0325] A "means for initiating navigation" is software or a device that calculates a route to a destination and provides audio and visual guidance to the driver.

[0326] "Means for providing entertainment content based on the driver's tastes and preferences" refers to a system that learns the driver's past behavior and preferences, and based on that, selects and provides appropriate entertainment content such as music, radio, and audiobooks.

[0327] The "means for providing audiobooks" is a system for resuming an audiobook that was previously being listened to or recommending and playing a new audiobook when a driver expresses a desire to listen to an audiobook.

[0328] This invention is a driving assistance system that acquires voice input and analyzes it using a generative AI model to provide appropriate navigation and entertainment content to the driver. Specific embodiments of this system are described below.

[0329] Hardware and software used

[0330] Hardware: On-board computer, microphone, speaker, GPS module

[0331] Software: generative AI models (e.g., GPT-4), speech recognition software (e.g., Google Speech-to-Text), navigation software (e.g., Google Maps API), music streaming services (e.g., Spotify API)

[0332] System Operation

[0333] 1. Acquiring voice input

[0334] Terminal: The on-board computer receives the driver's voice through a microphone. The microphone captures the driver's speech in real time and sends it to the on-board computer as voice data.

[0335] Example: When a driver says, "I want to go to the nearest gas station," a microphone captures the voice and sends it to the car's computer.

[0336] 2. Voice Recognition

[0337] Server: The voice recognition software receives the voice data sent from the vehicle's computer and converts it into text. The voice recognition software uses services such as Google Speech-to-Text to achieve highly accurate voice recognition.

[0338] Example: Speech data such as "I want to go to the nearest gas station" is converted into text data.

[0339] 3. Analysis using generative AI models

[0340] Server: A generative AI model (e.g., GPT-4) analyzes the text data and determines the appropriate action. The generative AI model understands the driver's intentions and generates prompts to search for the required information.

[0341] Example: The instruction generated is "Find the nearest gas station and start navigation."

[0342] 4. Start navigation

[0343] Server: Navigation software (e.g., Google Maps API) receives instructions from the generative AI model, searches for the nearest gas station, calculates the optimal route from the search results, and generates navigation data.

[0344] Terminal: The on-board computer receives navigation data and provides audio and visual directions to the driver.

[0345] Example: Routes to the nearest gas station are displayed on the in-car display and voice guidance begins.

[0346] 5. Entertainment

[0347] Server: The generative AI model learns the driver's preferences and selects appropriate entertainment content. It uses music streaming services (e.g., Spotify API) to search for songs by the driver's favorite music genres and artists.

[0348] Device: The onboard computer plays the selected music.

[0349] Example: The driver's favorite jazz music is played through the car speakers.

[0350] 6. Audiobook Offerings

[0351] Server: When the driver says, "I want to listen to an audiobook," the generative AI model resumes the audiobook that was previously being listened to or recommends a new audiobook. It uses an audiobook service (e.g., Audible API) to search for a suitable audiobook.

[0352] Device: The vehicle's onboard computer plays the selected audiobook.

[0353] Example: An audiobook the driver was previously listening to continues playing, or a new audiobook starts playing over the car speakers.

[0354] Examples of prompt statements

[0355] Prompt 1: "I want to go to the nearest gas station."

[0356] Prompt 2: "I want to listen to an audiobook."

[0357] Prompt 3: "Play some jazz music."

[0358] This system allows the driver to easily set a destination and enjoy entertainment by voice while driving. The flow of the specification process in the third embodiment will be described with reference to FIG.

[0359] Step 1:

[0360] Acquiring voice input

[0361] Terminal: The on-board computer picks up the driver's voice through a microphone.

[0362] Input: Driver utterances (e.g., "I want to go to the nearest gas station").

[0363] How it works: The microphone captures the driver's speech in real time and sends it to the onboard computer as audio data.

[0364] Output: Audio data.

[0365] Step 2:

[0366] Voice Recognition

[0367] Server: Speech recognition software receives the voice data sent from the vehicle's computer and converts it into text.

[0368] Input: Audio data.

[0369] What happens: Speech recognition software (e.g., Google Speech-to-Text) analyzes the audio data and generates corresponding text data.

[0370] Output: Text data (e.g. "I want to go to the nearest gas station").

[0371] Step 3:

[0372] Analysis using generative AI models

[0373] Server: A generative AI model (e.g., GPT-4) analyzes the text data and determines the appropriate action.

[0374] Input: Text data.

[0375] How it works: The generative AI model analyzes text data, understands the driver's intent, and generates prompts to search for the required information.

[0376] Output: Instructions (e.g., "Find the nearest gas station and start navigation").

[0377] Step 4:

[0378] Start navigation

[0379] Server: Navigation software (e.g., Google Maps API) receives instructions from the generative AI model and searches for the nearest gas station.

[0380] Input: Instruction data.

[0381] What it does: Navigation software finds the nearest gas station and calculates the optimal route.

[0382] Output: Navigation data (route information).

[0383] Step 5:

[0384] Providing navigation

[0385] Terminal: The on-board computer receives navigation data and provides audio and visual directions to the driver.

[0386] Input: Navigation data.

[0387] Specific operation: Route information is displayed on the in-car display and voice guidance is provided through the speaker.

[0388] Output: Navigation instructions to the driver.

[0389] Step 6:

[0390] Entertainment provision

[0391] Server: The generative AI model learns the driver's preferences and selects appropriate entertainment content.

[0392] Input: Driver's preferences data.

[0393] Specific behavior: Uses music streaming services (e.g., Spotify API) to search for songs by the driver's preferred music genres and artists.

[0394] Output: Entertainment content (music data).

[0395] Step 7:

[0396] Entertainment Regeneration

[0397] Device: The onboard computer plays the selected music.

[0398] Input: Entertainment content (music data).

[0399] Specific operation: Play selected music through the car speakers.

[0400] Output: Music playback.

[0401] Step 8:

[0402] Audiobook offerings

[0403] Server: If the driver says, "I want to listen to an audiobook," the generative AI model will resume the audiobook they were previously listening to or recommend a new audiobook.

[0404] Input: Driver utterance (e.g., "I want to listen to an audiobook").

[0405] What it does: Uses an audiobook service (e.g., Audible API) to search for a suitable audiobook.

[0406] Output: Audiobook data.

[0407] Step 9:

[0408] Playing an audiobook

[0409] Device: The vehicle's onboard computer plays the selected audiobook.

[0410] Input: Audiobook data.

[0411] Specific behavior: Plays a selected audiobook through the car's speakers.

[0412] Output: Audiobook playback.

[0413] (Application example 3)

[0414] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0415] Conventional driver assistance systems lack the ability to detect driver fatigue or loss of concentration, or to provide driving advice. They also have limited capabilities for providing entertainment content while driving, or for making recommendations based on the driver's hobbies and preferences. Furthermore, they lack the ability to search for the nearest facility based on the driver's verbal input, initiate navigation, or automatically play the driver's preferred music or audiobook. To address these issues, more advanced driver assistance systems are needed.

[0416] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0417] In this invention, the server is a driving assistance system that combines a generative AI and a voice recognition system, and includes: means for providing driving operation advice; means for detecting driver fatigue or loss of concentration and issuing alerts or suggesting rest; means for setting a destination through verbal interaction with the generative AI while driving; means for making recommendations tailored to the driver's hobbies and preferences; means for providing entertainment content such as music, radio, and audiobooks; means for searching for the nearest facility based on what the driver has said verbally and starting navigation; and means for automatically playing music or audiobooks based on the driver's hobbies and preferences. This allows the driver to not only reach their destination safely and comfortably while driving, but also enjoy entertainment.

[0418] "Generative AI" refers to artificial intelligence that generates new information and content based on user input.

[0419] A "voice recognition system" is a system that has the technology to convert voice into text.

[0420] A "driving assistance system" is a system that assists the driver in driving and improves safety and comfort.

[0421] "Driving operation advice" is a function that suggests appropriate driving methods and operations to the driver.

[0422] "Detection of fatigue and loss of concentration" refers to a function that detects the driver's state of fatigue or loss of concentration.

[0423] "Alerts and rest suggestions" refers to a function that issues a warning to the driver and suggests taking a break.

[0424] "Destination setting" is a function that allows the driver to set the location they want to go to in the system.

[0425] "Recommendations based on hobbies and preferences" is a function that recommends content and services based on the driver's preferences.

[0426] "Entertainment content" refers to entertainment content such as music, radio, and audiobooks.

[0427] "Search for nearest facility" is a function that searches for the nearest facility based on what the driver says verbally.

[0428] "Start navigation" refers to the function that provides directions to your destination.

[0429] "Automatic music and audiobook playback" refers to a function that automatically plays music and audiobooks based on the driver's preferences.

[0430] The system for carrying out this invention is a driving assistance system that combines generative AI and a voice recognition system. Specific embodiments of this system will be described below.

[0431] Hardware and software used

[0432] The server uses the following hardware and software:

[0433] Hardware: Microphone (for audio input), Speaker (for audio output)

[0434] Software: speech_recognition library (speech recognition), pyttsx3 library (speech synthesis), requests library (API requests)

[0435] Data processing and calculation

[0436] The server performs the following data processing and calculations.

[0437] 1. Speech-to-text conversion: Use a microphone to collect the driver's voice and convert it to text using the speech_recognition library.

[0438] 2. Keyword detection: Detect specific keywords from the converted text and select the appropriate action.

[0439] 3. API Request: Send an API request using the requests library to search for, for example, the nearest gas station based on keywords.

[0440] 4. Audio output: Search results and recommendations are output as audio using the pyttsx3 library.

[0441] Specific examples

[0442] When the driver says, "I want to go to the nearest gas station," the server performs the following process.

[0443] 1. Collect audio with a microphone and convert it to text using the speech_recognition library.

[0444] 2. Detect the keyword "gas station" from the text and send an API request to find the nearest gas station.

[0445] 3. Receive the search results and output the name of the nearest gas station using the pyttsx3 library.

[0446] Also, if the driver says, "I want to listen to an audiobook," the server performs the following process.

[0447] 1. Collect audio with a microphone and convert it to text using the speech_recognition library.

[0448] 2. Detect the keyword "audiobook" from the text and refer to the driver's taste and preference data to resume the audiobook that was previously being listened to or recommend a new audiobook.

[0449] 3. Output the recommendations as audio using the pyttsx3 library.

[0450] Prompt Sentence Examples

[0451] An example of a prompt for the generative AI model is as follows:

[0452] If the user says "I want to go to the nearest gas station," create a program that searches for the nearest gas station and starts navigation. Also, if the user says "I want to listen to an audiobook," add a function to resume the audiobook that was previously being listened to.

[0453] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0454] Step 1:

[0455] The server uses a microphone to collect the driver's voice. The input is the driver's voice and the output is audio data. Specifically, the microphone captures the driver's speech in real time.

[0456] Step 2:

[0457] The server converts the collected voice data into text using the speech_recognition library. The input is voice data and the output is text data. Specifically, the voice recognition engine analyzes the voice data and generates the corresponding text.

[0458] Step 3:

[0459] The server detects specific keywords from the converted text. The input is the text data, and the output is the detected keywords. Specifically, the text analysis algorithm searches for keywords in the text and extracts the relevant keywords.

[0460] Step 4:

[0461] The server sends an appropriate API request based on the detected keyword. For example, if the keyword "gas station" is detected, it sends an API request to search for the nearest gas station. The input is the keyword, and the output is the response data of the API request. Specifically, it uses the requests library to send a request to an external API and receive a response.

[0462] Step 5:

[0463] The server analyzes the response data of the API request and extracts the necessary information. The input is the response data, and the output is the extracted information. Specifically, the response data analysis algorithm extracts the necessary information (for example, the name and location of the nearest gas station).

[0464] Step 6:

[0465] The server outputs the extracted information as voice using the pyttsx3 library. The input is the extracted information, and the output is voice data. Specifically, the voice synthesis engine converts the text information into voice and conveys it to the driver through the speaker.

[0466] Step 7:

[0467] When a driver says, "I want to listen to an audiobook," the server refers to the driver's interest and preference data and either resumes the audiobook they were listening to previously or recommends a new audiobook. The input is the driver's statement and interest and preference data, and the output is information about the recommended audiobook. Specifically, the server retrieves the driver's past listening history from the database and selects an appropriate audiobook.

[0468] Step 8:

[0469] The server outputs the information about the recommended audiobook as audio using the pyttsx3 library. The input is the information about the recommended audiobook, and the output is audio data. Specifically, the speech synthesis engine converts the text information into audio and transmits it to the driver through the speaker.

[0470] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0471] "Example 1"

[0472] One embodiment of the present invention is a driving assistance system that incorporates an emotion engine. This system recognizes the driver's emotions from their facial expressions and tone of voice, and adjusts the content of the driving assistance accordingly. For example, if the system determines that the driver is feeling angry or anxious, it will play calming music or encourage the driver to take deep breaths to calm the driver.

[0473] "Example 2"

[0474] Another embodiment of the present invention is a system in which an emotion engine recognizes the driver's emotions and adjusts the entertainment content provided in accordance with the emotions. For example, if the system determines that the driver is feeling happy or excited, it automatically selects and plays upbeat music that the driver prefers. Conversely, if the system determines that the driver is depressed, it automatically selects and plays calming music that the driver prefers.

[0475] "Example 3"

[0476] Furthermore, in another embodiment of the present invention, there is a system in which the emotion engine recognizes the driver's emotions and adjusts the content of driving assistance according to those emotions. For example, if the system determines that the driver is feeling tired or stressed, it recommends appropriate rest spots and suggests that the driver take a break.

[0477] "Example 4"

[0478] In another embodiment of the present invention, an emotion engine recognizes the driver's emotions and adjusts driving advice accordingly. For example, if the system determines that the driver is feeling anxious or scared, it provides the driver with advice on safe driving.

[0479] The processing flow of each embodiment will be described below.

[0480] "Example 1"

[0481] Step 1: The emotion engine recognizes the driver's facial expressions and tone of voice.

[0482] Step 2: The emotion engine determines the driver's emotion.

[0483] Step 3: The driving assistance system adjusts the content of driving assistance based on the emotion engine's judgment.

[0484] Step 4: If the system determines that the driver is feeling angry or anxious, it will try to calm the driver by playing calming music or asking the driver to take deep breaths.

[0485] "Example 2"

[0486] Step 1: The emotion engine recognizes the driver's facial expressions and tone of voice.

[0487] Step 2: The emotion engine determines the driver's emotion.

[0488] Step 3: The driving assistance system adjusts the entertainment content provided based on the emotion engine's judgment.

[0489] Step 4: If the system determines that the driver is feeling happy or excited, it will automatically select and play upbeat music that the driver prefers. Conversely, if the system determines that the driver is feeling depressed, it will automatically select and play calming music that the driver prefers.

[0490] "Example 3"

[0491] Step 1: The emotion engine recognizes the driver's facial expressions and tone of voice.

[0492] Step 2: The emotion engine determines the driver's emotion.

[0493] Step 3: The driving assistance system adjusts the content of driving assistance based on the emotion engine's judgment.

[0494] Step 4: If the system determines that the driver is feeling fatigued or stressed, it will recommend appropriate rest spots and suggest that the driver take a break.

[0495] "Example 4"

[0496] Step 1: The emotion engine recognizes the driver's facial expressions and tone of voice.

[0497] Step 2: The emotion engine determines the driver's emotion.

[0498] Step 3: The driver assistance system adjusts the driving advice based on the emotion engine's judgment.

[0499] Step 4: If the system determines that the driver is anxious or fearful, it will

[0500] and provide advice on safe driving.

[0501] Example 1

[0502] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0503] Conventional driving assistance systems have limited capabilities to provide advice to drivers on their driving operations or to detect driver fatigue or loss of concentration, making it difficult to fully ensure driver safety. Furthermore, they do not adjust driving assistance according to the driver's emotional state, making it impossible to reduce the driver's stress and anxiety. Furthermore, they lack the ability to measure the approaching speed and distance of vehicles behind in real time and determine safe lane changes, making it impossible to improve driver safety.

[0504] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0505] In this invention, the server is a driving assistance system that combines generative AI and a voice recognition system, and includes a means for providing driving operation advice, a means for detecting driver fatigue or loss of concentration and issuing an alert or suggesting a break, a means for measuring the approaching speed and distance of vehicles behind and determining whether a safe lane change is possible, and a means for recognizing the driver's emotions from their facial expressions and tone of voice and adjusting the content of driving assistance according to those emotions. This makes it possible to increase driver safety, reduce driver stress and anxiety, and provide a more comfortable driving environment.

[0506] "Generative AI" is a system that uses artificial intelligence technology to analyze data and generate appropriate advice and information.

[0507] A "voice recognition system" is a technology that analyzes voice data and converts it into text data.

[0508] A "driving assistance system" is a system that assists the driver in driving operations and improves safety and comfort.

[0509] "Driving advice" means instructions or advice provided to a driver to help them drive safely and efficiently.

[0510] "Detection of driver fatigue and loss of concentration" is a technology that monitors the driver's physical and mental state and identifies fatigue and loss of concentration.

[0511] "Alerts and rest suggestions" are notifications or suggestions to alert the driver or encourage them to take a break.

[0512] "Measurement of approaching speed and distance of rear vehicles" is a technology for measuring the speed and distance of other vehicles present behind the vehicle.

[0513] "Safe lane change judgment" is a technology that analyzes whether lane changes can be made safely and provides appropriate instructions to the driver.

[0514] "Recognition of driver's facial expressions and tone of voice" is a technology that analyzes the driver's facial expressions and tone of voice to identify their emotional state.

[0515] "Adjusting the content of driving assistance" is a technology that changes the content of driving assistance provided depending on the emotional state of the driver.

[0516] This invention is a driving assistance system that combines generative AI and a voice recognition system, providing driving advice, detecting driver fatigue or loss of concentration and issuing alerts or suggesting rest. It also includes functions to measure the approaching speed and distance of vehicles behind to determine whether a safe lane change is possible, and recognizing the driver's emotions from facial expressions and tone of voice and adjusting the content of driving assistance accordingly.

[0517] Hardware and software used

[0518] 1. The device collects data using hardware such as cameras, LIDAR sensors, and highly sensitive microphones mounted on the vehicle, allowing it to measure the approaching speed and distance of vehicles behind, as well as the driver's facial expressions and tone of voice in real time.

[0519] 2. The server uses generative AI (e.g., OpenAI®'s GPT-4) and emotion recognition engine (e.g., Affectiva's Emotion Recognition API) to analyze the collected data.

[0520] 3. The device converts the generated advice and driving assistance content into speech using a voice recognition system (e.g., Google Cloud Speech-to-Text API) and provides it to the driver.

[0521] Explanation of program processing

[0522] 1. Data Collection

[0523] The device uses the vehicle's onboard camera and LIDAR sensor to measure the approaching speed and distance of the vehicle behind in real time. The camera captures the vehicle behind and processes the video data to determine the vehicle's position. The LIDAR sensor uses lasers to measure distance.

[0524] The device uses the vehicle's on-board camera and microphone to collect the driver's facial expressions and tone of voice. The camera captures the driver's face, and the microphone records the driver's voice.

[0525] 2. Data Transmission

[0526] The device sends the collected data to a server, including the speed, distance, and direction of the vehicle behind, as well as the driver's facial expression, tone of voice, and heart rate. The device sends the data to the server via the Internet, and the data is encrypted before transmission.

[0527] 3. Data Analysis

[0528] The server inputs the received data into the generative AI, which analyzes it to determine whether it is safe to change lanes. The generative AI performs the analysis using prompts.

[0529] The server inputs the received data into an emotion recognition engine to analyze the driver's emotions. The emotion recognition engine performs the analysis using the prompt sentence.

[0530] 4. Advice Generation

[0531] Based on the analysis results, the generative AI generates appropriate advice for the driver, such as "A vehicle behind you is approaching. Please refrain from changing lanes."

[0532] Based on the analysis results, the emotion recognition engine generates appropriate driving assistance according to the driver's emotions. For example, it generates advice such as "Take a deep breath" and instructs the driver to play calming music.

[0533] 5. Audio conversion and execution

[0534] The device converts the generated advice into voice using a voice recognition system and provides it to the driver. The voice recognition system converts the text data into voice data and provides it to the driver through the speaker. At the same time, the music playback system plays the selected music.

[0535] Examples of concrete examples and prompts

[0536] Specific examples of lane change advice

[0537] The terminal measures the approaching speed of the vehicle behind as 80 km / h and the distance as 50 m, and sends this to the server.

[0538] The server inputs a prompt sentence into the generative AI, and as a result of the analysis, it generates the advice, "A vehicle behind you is approaching. Please refrain from changing lanes."

[0539] The device converts this advice into audio and provides it to the driver.

[0540] Specific examples of emotion-aware advice

[0541] The device collects data on the driver's facial expressions to detect anger and sends it to a server.

[0542] The server inputs the prompt sentence into the emotion recognition engine, and generates the advice "Take a deep breath" as an analysis result.

[0543] The device converts this advice into audio and provides it to the driver, while playing soothing music.

[0544] Prompt Sentence Examples

[0545] 1. Lane change advice prompt

[0546] The vehicle behind is approaching at 80km / h and the distance is 50m. Advise the driver whether it is safe to change lanes.

[0547] 2. Emotion-aware advice prompts

[0548] The driver is angry. Please provide some advice to help the driver calm down.

[0549] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0550] Step 1: Data collection

[0551] The device uses the vehicle's on-board camera and LIDAR sensor to measure the approaching speed and distance of the vehicle behind in real time. The camera captures images of the vehicle behind and processes the image data to determine the vehicle's position. The LIDAR sensor uses a laser to measure distance. The input is the camera image and LIDAR data, and the output is data on the speed and distance of the vehicle behind.

[0552] Step 2: Collecting Emotional Data

[0553] The device collects the driver's facial expressions and tone of voice using the camera and microphone installed in the vehicle. The camera captures the driver's face, and the microphone records the driver's voice. The input is the camera image and audio data, and the output is the driver's facial expression data and tone of voice data.

[0554] Step 3: Send data

[0555] The terminal sends the collected data to a server. This data includes the speed, distance, and direction of the vehicle behind, as well as the driver's facial expression, tone of voice, and heart rate. The terminal sends the data to the server via the Internet, and the data is encrypted before transmission. The input is the various collected data, and the output is the data sent to the server.

[0556] Step 4: Data analysis

[0557] The server inputs the received data into the generative AI, which analyzes it to determine whether it is safe to change lanes. The generative AI performs the analysis using prompts. The input is data on the speed and distance of the vehicle behind, and the output is advice on the safety of changing lanes.

[0558] Step 5: Sentiment Analysis

[0559] The server inputs the received data into an emotion recognition engine to analyze the driver's emotions. The emotion recognition engine performs the analysis using prompt sentences. The input is the driver's facial expression data and tone of voice data, and the output is information about the driver's emotional state.

[0560] Step 6: Advice Generation

[0561] Based on the analysis results, generative AI generates appropriate advice for the driver. For example, it generates advice such as, "A vehicle behind is approaching. Please refrain from changing lanes." The input is data related to the safety of lane changes, and the output is text data of the advice.

[0562] Step 7: Emotion-based advice generation

[0563] Based on the analysis results, the emotion recognition engine generates appropriate driving assistance according to the driver's emotions. For example, it generates advice such as "Take a deep breath" and instructs the driver to play calming music. The input is data about the driver's emotional state, and the output is text data of advice according to the emotion.

[0564] Step 8: Convert audio and run

[0565] The terminal converts the generated advice into voice using a voice recognition system and provides it to the driver. The voice recognition system converts the text data into voice data and provides it to the driver through the speaker. At the same time, the music playback system plays the selected music. The input is the text data of the advice, and the output is the voice data and the played music.

[0566] (Application example 1)

[0567] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0568] Conventional driver assistance systems are limited to providing advice on driver operation and fatigue detection, and lack support based on the driver's emotional state. Furthermore, they do not provide specific advice to ensure safety when changing lanes or making turns. This can increase driver stress and anxiety, potentially reducing driving safety.

[0569] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0570] In this invention, the server is a driving assistance system that combines generative AI and a voice recognition system, and includes: means for providing driving operation advice; means for detecting driver fatigue or loss of concentration and issuing an alert or suggesting a break; means for recognizing the driver's emotions from their facial expressions and tone of voice and adjusting the content of driving assistance according to those emotions; means for judging the approaching speed and distance of vehicles behind the driver when the driver changes lanes or turns right or left and providing voice advice on safe timing; and means for playing calming music or encouraging the driver to take deep breaths when the driver feels angry or anxious. This provides appropriate assistance according to the driver's emotional state, making it possible to improve driving safety and comfort.

[0571] "Generative AI" is an artificial intelligence technology that generates new information and content based on data.

[0572] A "voice recognition system" is a technology that analyzes human speech and converts it into text data.

[0573] A "driving assistance system" is a technology that assists the driver in driving operations and improves safety and comfort.

[0574] "Driving operation advice" refers to providing appropriate instructions and advice to the driver regarding the driving operations they perform.

[0575] "Detection of fatigue and loss of concentration" means monitoring the driver's physical and mental state to detect fatigue and loss of concentration.

[0576] "Alerts and rest suggestions" refers to notifications that alert the driver or encourage them to take a break.

[0577] "Emotion recognition" refers to determining a driver's emotional state from their facial expressions and tone of voice.

[0578] "Adjusting the content of driving assistance" means changing the content of assistance provided in accordance with the emotional state of the driver.

[0579] "Lane change and right / left turn advice" refers to voice instructions to the driver on the safe timing to change lanes or turn right or left.

[0580] "Judgment of approach speed and distance" means measuring the speed and distance of the vehicle behind and assessing safety.

[0581] "Playing calming music" means playing music that has a relaxing effect to calm the driver's emotions.

[0582] "Deep breathing prompt" means instructing the driver to take a deep breath.

[0583] As an embodiment of the present invention, the Emotion-Aware Driving Assistant (EADA) system will be described as an example. The EADA system is a driving assistance system that combines generative AI and a voice recognition system, and provides advice to the driver regarding driving operations and assistance according to the driver's emotional state.

[0584] System Configuration

[0585] The EADA system uses the following hardware and software:

[0586] Hardware:

[0587] Camera: Used to capture the driver's facial expressions.

[0588] Microphone: Used to capture the driver's tone of voice.

[0589] Speaker: Used to play voice advice and music.

[0590] software:

[0591] OpenCV: Used to process the images from the camera and perform facial recognition.

[0592] TensorFlow / Keras: Used to run the emotion recognition model.

[0593] pyttsx3: Used as a speech synthesis engine.

[0594] Data processing and calculation

[0595] The server converts the image captured by the camera to grayscale and performs facial recognition. It then extracts the area of ​​the recognized face and inputs it into an emotion recognition model to predict the emotion. Based on the predicted emotion, it provides audio advice or plays music.

[0596] Specific examples

[0597] If a driver attempts to change lanes and a vehicle behind is approaching at high speed, the system will provide a voice warning, "A vehicle behind is approaching. Changing lanes is dangerous." If the driver appears angry, the system will provide a voice warning, "Take a deep breath. We will play calming music," and will then play calming music.

[0598] Prompt Sentence Examples

[0599] An example of a prompt to input to a generative AI model is as follows:

[0600] When a driver attempts to change lanes, please design a system that judges the approaching speed and distance of vehicles behind and provides voice advice on the safe timing. Also, please add a function that recognizes the driver's emotions from their facial expressions and tone of voice, and calms them down by playing calming music or encouraging them to take deep breaths if they seem angry or anxious.

[0601] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0602] Step 1:

[0603] The server acquires video data from the camera. The input is real-time video from the camera, and the output is the acquired video data. Specifically, the camera captures the driver's face and sends the video data to the server.

[0604] Step 2:

[0605] The server converts the acquired video data to grayscale and performs face recognition. The input is the video data acquired in step 1, and the output is the recognized face area. Specifically, the server converts the video data to grayscale using OpenCV and applies a face detection algorithm.

[0606] Step 3:

[0607] The server extracts the recognized face area and performs preprocessing to input it into the emotion recognition model. The input is the face area recognized in step 2, and the output is preprocessed face image data. Specifically, the server resizes and normalizes the face area.

[0608] Step 4:

[0609] The server inputs the preprocessed facial image data into the emotion recognition model to predict the emotion. The input is the facial image data preprocessed in step 3, and the output is the predicted emotion label. Specifically, the emotion recognition model is executed using TensorFlow / Keras to obtain the emotion label.

[0610] Step 5:

[0611] The server determines the appropriate voice advice or music playback based on the predicted emotion label. The input is the emotion label predicted in step 4, and the output is the voice advice or music playback instruction. As a specific operation, it executes logic to select an action according to the emotion label.

[0612] Step 6:

[0613] The server uses a speech synthesis engine to generate voice advice and plays it through the speaker. The input is the voice advice instruction determined in step 5, and the output is the voice advice for the driver. Specifically, speech synthesis is performed using pyttsx3, and the voice is output from the speaker.

[0614] Step 7:

[0615] When the driver changes lanes or makes a right or left turn, the server determines the approaching speed and distance of the vehicle behind and gives voice advice on the safest timing. The input is data from cameras and sensors, and the output is voice advice on the safest timing. Specifically, the server calculates the speed and distance of the vehicle behind and executes an algorithm to evaluate safety.

[0616] Step 8:

[0617] If the driver feels angry or anxious, the server plays calming music or encourages deep breathing. The input is the emotion label predicted in step 4, and the output is instructions to play music or take deep breaths. The specific operation is to select and play music with a relaxing effect according to the emotion label.

[0618] Example 2

[0619] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0620] Conventional driving assistance systems lacked the ability to detect driver fatigue or loss of concentration, and often failed to suggest rest at the appropriate time. They also lacked the ability to provide entertainment content that responded to the driver's emotions, making it difficult to reduce stress or improve comfort while driving. Another problem was the lack of smooth interaction with generative AI when providing driving advice or setting a destination.

[0621] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0622] In this invention, the server includes a means for analyzing the driver's gaze, facial expression, and driving operation patterns to detect fatigue or loss of concentration, a means for recommending appropriate rest spots and suggesting a break by voice when fatigue or loss of concentration is detected, and a means for recognizing the driver's emotions and adjusting the entertainment content provided in accordance with the emotions. This makes it possible to appropriately detect the driver's fatigue or loss of concentration and suggest a break in a timely manner. Furthermore, providing entertainment content in accordance with the driver's emotions can reduce stress while driving and improve comfort.

[0623] "Means of analyzing the driver's line of sight, facial expression, and driving operation patterns to detect fatigue and loss of concentration" refers to technology that uses cameras and sensors to collect data on the driver's line of sight, changes in facial expression, and operation of the steering wheel and pedals, and analyzes this data to detect fatigue and loss of concentration in the driver.

[0624] "Means for recommending appropriate rest spots and suggesting a break by voice when detected" refers to a technology that uses a geographic information system (GIS) to search for the nearest rest spot when fatigue or a decline in concentration in the driver is detected, and then uses voice synthesis technology to suggest a break to the driver.

[0625] "Means for recognizing the driver's emotions and adjusting the entertainment content provided in accordance with those emotions" refers to technology that uses cameras and microphones to analyze the driver's facial expressions and tone of voice, identify the driver's emotions, and select and provide entertainment content such as music, radio, and audiobooks in accordance with those emotions.

[0626] A "driving assistance system that combines generative AI and voice recognition" is a system that combines a generative AI model with voice recognition technology to provide the driver with advice on driving operations and perform destination setting and other operations through dialogue while driving.

[0627] "Means for providing driving operation advice" refers to a technology that uses a generative AI model to analyze the driver's driving operations and provide appropriate driving methods and areas for improvement via voice or display.

[0628] "Means for setting a destination through verbal interaction with generative AI while driving" refers to a technology that uses voice recognition technology to understand the driver's verbal instructions and uses a generative AI model to set a destination and provide route guidance.

[0629] "Means for making recommendations tailored to the driver's hobbies and preferences" refers to technology that analyzes the driver's past behavior and preferences and, based on that, recommends appropriate entertainment content, rest spots, etc.

[0630] "Means for providing entertainment content such as music, radio, and audiobooks" refers to technology that selects and plays entertainment content such as music, radio programs, and audiobooks according to the driver's emotions and preferences.

[0631] MODE FOR CARRYING OUT THE INVENTION

[0632] This invention is a driving assistance system that detects driver fatigue and loss of concentration, suggests appropriate rest, and adjusts entertainment content according to the driver's emotions. This system also has the function of providing driving operation advice by combining a generative AI model and a voice recognition system.

[0633] Hardware and software used

[0634] Hardware: Cameras, sensors, microphones, speakers

[0635] Software: Image recognition software, emotion recognition engines, speech synthesis software, entertainment content management systems, geographic information systems (GIS)

[0636] Data processing and calculation

[0637] 1. Data Collection

[0638] The device collects data on the driver's gaze, facial expressions, driving operations, and voice through cameras, sensors, and microphones installed inside the vehicle.

[0639] How it works: The camera captures the driver's face in real time, sensors capture steering and pedal operation data, and a microphone records the driver's voice.

[0640] 2. Data Analysis

[0641] The server receives the collected data and analyzes it using image recognition software and data analysis software.

[0642] Specific operation: The server analyzes image data sent from the camera to detect eye movements and changes in facial expressions. It also analyzes data from sensors to analyze driving patterns.

[0643] 3. Detection of fatigue and loss of concentration

[0644] Based on the analysis results, the server runs an algorithm to detect driver fatigue and loss of concentration.

[0645] Specific operation: The server evaluates blink frequency, gaze fixation time, steering reaction time, etc. to detect signs of fatigue or loss of concentration.

[0646] 4. Alerts and break suggestions

[0647] If the server detects fatigue or a decrease in concentration, the AI ​​will recommend an appropriate rest spot and notify the device.

[0648] The device uses voice synthesis software to suggest breaks to the driver.

[0649] Specific operation: The server uses a geographic information system (GIS) to search for the nearest rest spot and sends the information to the device. The device then notifies the user by voice, saying, "There is a rest spot nearby. We recommend that you take a break."

[0650] 5. Emotion recognition

[0651] The server uses an emotion recognition engine to recognize the driver's emotions based on data acquired from the camera and microphone.

[0652] Specific operation: The server analyzes the driver's facial expressions and tone of voice to identify emotions such as joy, excitement, or depression.

[0653] 6. Entertainment Content Adjustment

[0654] The server selects appropriate content using an entertainment content management system according to the driver's emotions and transmits it to the terminal.

[0655] The device plays the selected content.

[0656] Specific operation: If the server determines that the driver is feeling happy or excited, it selects upbeat music and sends it to the device, which then plays that music. Conversely, if the server determines that the driver is feeling depressed, it selects calming music and plays it.

[0657] Specific examples

[0658] Examples of fatigue detection:

[0659] The device detects the driver's blinking frequency through the camera, and if the driver blinks more frequently than normal, the server detects fatigue.

[0660] The server will recommend an appropriate rest spot, and the device will notify the user by voice, "There is a rest spot nearby. We recommend that you take a break."

[0661] Examples of emotion recognition:

[0662] The device collects the driver's facial expressions and tone of voice through a camera and microphone, and the server determines that the driver is feeling happy if they are smiling and speaking.

[0663] The server selects up-tempo music and the device plays that music.

[0664] Prompt Sentence Examples

[0665] Fatigue detection prompt:

[0666] Please explain the system that analyzes the driver's gaze, facial expressions, and driving operation patterns to detect fatigue or loss of concentration. If fatigue or loss of concentration is detected, please also explain the function that uses AI to recommend appropriate rest spots and suggests taking a break by voice.

[0667] Emotion recognition prompt:

[0668] Describe a system that recognizes the driver's emotions and adjusts the entertainment content provided accordingly, including automatically selecting and playing upbeat music if the driver is feeling happy or excited.

[0669] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0670] Step 1:

[0671] Data collection

[0672] Input: Real-time data from cameras, sensors, and microphones

[0673] Specific operation: The device uses a camera installed inside the vehicle to capture the driver's face, uses sensors to obtain data on steering and pedal operation, and uses a microphone to record the driver's voice.

[0674] Output: Gaze data, facial expression data, driving operation data, voice data

[0675] Step 2:

[0676] Data analysis

[0677] Input: Gaze data, facial expression data, driving operation data, voice data

[0678] Specific operation: The server uses image recognition software to analyze image data sent from the camera to detect eye movements and changes in facial expressions. Data analysis software is used to analyze driving operation data from the sensors and analyze driving patterns.

[0679] Output: Gaze analysis results, facial expression analysis results, driving operation analysis results

[0680] Step 3:

[0681] Detecting fatigue and loss of concentration

[0682] Input: Gaze analysis results, facial expression analysis results, driving operation analysis results

[0683] How it works: Based on the analysis results, the server runs an algorithm to detect driver fatigue and loss of concentration. It evaluates factors such as blink frequency, gaze fixation time, and steering reaction time to detect signs of fatigue or loss of concentration.

[0684] Output: Fatigue and concentration loss detection results

[0685] Step 4:

[0686] Alerts and break suggestions

[0687] Input: Fatigue and concentration loss detection results

[0688] Specific operation: When the server detects fatigue or a lack of concentration, it uses a geographic information system (GIS) to search for the nearest rest spot and sends the information to the device. The device then uses voice synthesis software to notify the driver by voice, "There is a rest spot nearby. We recommend that you take a break."

[0689] Output: Audio notification of break suggestion

[0690] Step 5:

[0691] emotion recognition

[0692] Input: Gaze data, facial expression data, voice data

[0693] How it works: The server uses an emotion recognition engine to recognize the driver's emotions based on data acquired from the camera and microphone. It analyzes the driver's facial expressions and tone of voice to identify emotions such as joy, excitement, and depression.

[0694] Output: Emotion recognition result

[0695] Step 6:

[0696] Entertainment content adjustment

[0697] Input: Emotion recognition results

[0698] Specific operation: The server selects appropriate content using an entertainment content management system according to the driver's emotions and sends it to the device. The device then plays the selected content. For example, if the server determines that the driver is feeling happy or excited, it selects upbeat music and plays that music. Conversely, if the server determines that the driver is feeling depressed, it selects and plays calming music.

[0699] Output: Entertainment content playback

[0700] (Application example 2)

[0701] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0702] Conventional driving assistance systems lacked the ability to detect driver fatigue or loss of concentration, and often failed to suggest rest at appropriate times. Furthermore, they did not provide entertainment content tailored to the driver's emotions, meaning that stress reduction and relaxation effects while driving could not be expected. This resulted in issues such as insufficient safety and comfort for the driver.

[0703] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0704] In this invention, the server includes a means for providing driving operation advice, a means for detecting the driver's fatigue or loss of concentration and issuing an alert or suggesting a break, a means for recognizing the driver's emotions and adjusting entertainment content according to those emotions, a means for analyzing the driver's line of sight and facial expression in real time using a smartphone camera, and a means for recommending appropriate rest spots and suggesting a break by voice. This makes it possible to appropriately detect the driver's fatigue or loss of concentration and suggest a break in a timely manner. Furthermore, providing entertainment content according to the driver's emotions can reduce stress while driving and increase the relaxation effect.

[0705] "Means for providing driving operation advice" refers to a function that provides the driver with real-time instructions on appropriate driving methods and operating procedures.

[0706] "Means to detect driver fatigue or loss of concentration and issue alerts or suggest breaks" refers to a function that analyzes the driver's gaze, facial expressions, and driving patterns, and issues a warning and suggests appropriate breaks when fatigue or loss of concentration is detected.

[0707] "Means for recognizing the driver's emotions and adjusting entertainment content according to those emotions" refers to a function that reads the driver's emotions from their facial expressions and behavior, and selects and plays entertainment content such as music, radio, and audiobooks according to those emotions.

[0708] "Means for analyzing the driver's gaze and facial expressions in real time using a smartphone camera" refers to a function that uses a smartphone camera to capture a picture of the driver's face and analyzes the image in real time to read the driver's gaze and facial expressions.

[0709] "Means to recommend appropriate rest spots and suggest resting spots via voice" is a function in which AI selects the most suitable resting spot based on the driver's current location and surrounding geographic information, and suggests that location via voice.

[0710] The system for implementing this invention has the functions of detecting driver fatigue and loss of concentration and suggesting appropriate breaks, as well as adjusting entertainment content according to the driver's emotions. Specific embodiments of this system are described below.

[0711] Hardware and Software Configuration

[0712] This system uses hardware such as a smartphone, server, camera, and audio output device, and software such as OpenCV (image processing library), dlib (face detection and landmark detection library), and playsound (audio playback library).

[0713] Data processing and calculation

[0714] 1. Real-time analysis of driver gaze and facial expressions:

[0715] The system captures the driver's face using a smartphone camera and analyzes the footage in real time, detecting facial landmarks and reading gaze and facial expressions using OpenCV and dlib.

[0716] 2. Detecting fatigue and loss of concentration:

[0717] Based on the detected facial landmark data, the system analyzes the eye opening and closing status and eye movement, thereby detecting driver fatigue and loss of concentration.

[0718] 3. Break Suggestion:

[0719] If fatigue or a lack of concentration is detected, the server will recommend the best rest spot based on the driver's current location and surrounding geographic information, and will use a voice output device to suggest a break to the driver.

[0720] 4. Emotion Recognition and Entertainment Content Adjustment:

[0721] The system recognizes the driver's emotions from their facial expressions and selects and plays entertainment content (music, radio, audiobooks, etc.) according to their emotions. For example, if the driver is feeling happy, it will play upbeat music, and if they are feeling depressed, it will play calming music.

[0722] Specific examples

[0723] For example, if the driver has been driving for a long time, the system will detect fatigue by detecting changes in the state of their eyes. In that case, the system will suggest by voice, "Please stop at a nearby rest spot." Also, if the driver is smiling, the system will play up-tempo music to help the driver relax.

[0724] Prompt Sentence Examples

[0725] Develop a system that detects driver fatigue and loss of concentration and suggests appropriate rest spots. Also, add a function to recognize the driver's emotions and adjust entertainment content accordingly. The hardware used is a smartphone camera, and the software uses OpenCV and dlib.

[0726] In this way, a system can be realized that can improve the safety and comfort of the driver.

[0727] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0728] Step 1:

[0729] The smartphone camera is used to capture the driver's face. The input is the camera image, and the output is a real-time facial image. The terminal acquires this facial image and sends it to the next step.

[0730] Step 2:

[0731] Using OpenCV and dlib, facial landmarks are detected from the face image. The input is the face image acquired in step 1, and the output is facial landmark data. The device analyzes this landmark data to read gaze and facial expressions.

[0732] Step 3:

[0733] Based on facial landmark data, the device analyzes the eye opening / closing status and gaze movement. The input is the landmark data obtained in step 2, and the output is the detection results of the driver's fatigue and loss of concentration. Based on these detection results, the device evaluates the driver's condition.

[0734] Step 4:

[0735] If fatigue or a loss of concentration is detected, the server recommends the optimal rest spot based on the driver's current location and surrounding geographic information. The input is the driver's current location and the detection result, and the output is information about the recommended rest spot. The server sends this information to an audio output device.

[0736] Step 5:

[0737] A voice output device is used to suggest a break to the driver. The input is the information about the rest spots obtained in step 4, and the output is a voice suggestion to take a break. The terminal plays this voice to encourage the driver to take a break.

[0738] Step 6:

[0739] Recognizes emotions from the driver's facial expressions. The input is the facial landmark data obtained in step 2, and the output is the driver's emotion recognition result. The device proceeds to the next step based on this emotion recognition result.

[0740] Step 7:

[0741] Entertainment content is selected and played according to the driver's emotions. The input is the emotion recognition result obtained in step 6, and the output is the selected entertainment content. The device plays this content and provides entertainment according to the driver's emotions.

[0742] Example 3

[0743] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0744] Conventional driving assistance systems have difficulty effectively utilizing the driver's voice input to set destinations or provide entertainment content. They also lack the ability to recognize the driver's emotions and provide appropriate driving assistance. This has led to a demand for effective means to reduce driver fatigue and stress and promote safe driving.

[0745] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0746] In this invention, the server includes means for capturing the driver's voice input and converting the voice to text using a generative AI model, means for analyzing the text data and understanding the user's intent, and means for searching for a destination and setting navigation. This allows the driver to easily set a destination through voice input, and furthermore, by recognizing the driver's emotions and providing driving assistance accordingly, it becomes possible to reduce the driver's fatigue and stress and promote safe driving.

[0747] "Generative AI" is an artificial intelligence technology that analyzes input data such as voice and text and generates responses and suggestions in natural language.

[0748] A "speech recognition system" is a technology that converts voice input into text data and is used to understand what a user is saying.

[0749] A "driving assistance system" is a system that provides technologies and functions to support the driver in safe driving.

[0750] "Driving operation advice" is a function that suggests safe and efficient driving methods to the driver.

[0751] "Detection of fatigue and loss of concentration" is a technology that monitors the driver's physical and mental state to detect fatigue and loss of concentration.

[0752] "Alerts and rest suggestions" is a function that notifies and suggests drivers to take a break when they feel fatigued or their concentration is declining.

[0753] "Voice input capture" is the process of recording the driver's speech using a device such as a microphone and sending it to the system.

[0754] A "generative AI model" is an artificial intelligence algorithm that analyzes input data and generates appropriate responses or suggestions.

[0755] "Text data analysis" is the process of understanding text data converted by a speech recognition system and grasping the user's intent.

[0756] "Destination search" is a function that identifies the optimal destination based on the user's request and provides navigation information.

[0757] "Navigation setup" is the process of calculating the optimal route to a destination and providing guidance to the driver.

[0758] "Entertainment content" refers to content that can be enjoyed while driving, such as music, radio, and audiobooks.

[0759] "Emotion recognition" is a technology that analyzes the driver's facial expressions and tone of voice to understand their emotional state.

[0760] "Providing driving assistance" is a function that provides appropriate advice and suggestions based on the driver's emotions and condition.

[0761] This invention is a driving assistance system that combines generative AI and a voice recognition system. It captures the driver's voice input, converts the voice into text using a generative AI model, and understands the user's intent to set destinations, provide entertainment content, and assist driving.

[0762] Hardware and software used

[0763] Hardware: On-board computer, microphone, speaker, GPS module

[0764] Software: generative AI models (e.g., GPT-4), emotion recognition engines, navigation software, music playback software

[0765] System configuration

[0766] 1. Capture voice input

[0767] What the user says is captured by the device's microphone and transmitted to the vehicle's onboard computer.

[0768] 2. Speech-to-text

[0769] The server converts the voice data into text using a generative AI model.

[0770] The server analyzes the text data and understands the user's intent.

[0771] 3. Search for a destination and set up navigation

[0772] The server obtains the current location using the GPS module.

[0773] The server consults a map database to find the nearest destination.

[0774] The server runs the navigation software and calculates the route to the destination.

[0775] The server sends the calculated route information to the terminal.

[0776] The device will begin navigation through the speaker.

[0777] 4. Provision of entertainment content

[0778] A user says, "I want to listen to an audiobook."

[0779] The device's microphone captures the audio and transmits it to the vehicle's onboard computer.

[0780] The server uses a generative AI model to convert speech to text and analyze intent.

[0781] The server will resume the previously listened to audiobook or recommend a new audiobook.

[0782] Your device will play the audiobook through the speaker.

[0783] 5. Emotion Recognition and Driver Assistance

[0784] The device's emotion recognition engine analyzes the driver's facial expressions and tone of voice to recognize their emotions.

[0785] The device transmits the recognized emotion data to the server.

[0786] The server analyzes the emotional data and determines the driver's state.

[0787] The server determines the appropriate driving assistance and sends it to the device.

[0788] The device provides the driver with advice on rest and safe driving through a speaker.

[0789] Specific examples

[0790] Example prompt: "I want to go to the nearest gas station."

[0791] The user speaks this prompt.

[0792] The device's microphone captures the audio and transmits it to the vehicle's onboard computer.

[0793] The server uses a generative AI model to convert speech to text and search for the nearest gas station.

[0794] The server runs the navigation software and calculates the route.

[0795] The device will begin navigation through the speaker.

[0796] Example prompt: "I want to listen to an audiobook."

[0797] The user speaks this prompt.

[0798] The device's microphone captures the audio and transmits it to the vehicle's onboard computer.

[0799] The server uses a generative AI model to convert the speech to text and select an audiobook.

[0800] Your device will play the audiobook through the speaker.

[0801] Example prompt: "I'm tired."

[0802] The user speaks this prompt.

[0803] The device's emotion recognition engine recognizes the driver's fatigue.

[0804] The server recommends suitable rest spots, and the device suggests taking a break.

[0805] In this way, the present invention analyzes the driver's voice input and provides appropriate navigation, entertainment, and driving assistance, thereby improving the safety and comfort of the driver. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0806] Step 1:

[0807] Capturing voice input

[0808] The user says, "I want to go to the nearest gas station."

[0809] The device's microphone captures the audio and transmits it to the vehicle's onboard computer.

[0810] Input: User's voice

[0811] Output: Audio data

[0812] Step 2:

[0813] Speech-to-text

[0814] The server converts the voice data into text using a generative AI model.

[0815] The server analyzes the text data and understands the user's intent.

[0816] Input: Audio data

[0817] Output: Text data

[0818] What happens: The server uses a speech recognition algorithm to convert the speech data into text, then analyzes the text to extract the intent: "I want to go to the nearest gas station."

[0819] Step 3:

[0820] Search for destinations

[0821] The server obtains the current location using the GPS module.

[0822] The server consults a map database to find the nearest gas station.

[0823] Input: Text data, current location information

[0824] Output: Location of nearest gas station

[0825] What it does: The server gets the current location from the GPS module and queries a map database to identify the nearest gas station.

[0826] Step 4:

[0827] Navigation Settings

[0828] The server runs the navigation software and calculates the route to the destination.

[0829] The server sends the calculated route information to the terminal.

[0830] Input: Location of nearest gas station

[0831] Output: Route information

[0832] What it does: The server uses navigation software to calculate the optimal route from your current location to the nearest gas station and sends that route information to your device.

[0833] Step 5:

[0834] Start navigation

[0835] The device will begin navigation through the speaker.

[0836] The device provides real-time route guidance.

[0837] Input: Route information

[0838] Output: Voice guidance

[0839] Specific operation: The device provides real-time voice guidance through the speaker, such as "Turn right at the next intersection."

[0840] Step 6:

[0841] Providing entertainment content

[0842] A user says, "I want to listen to an audiobook."

[0843] The device's microphone captures the audio and transmits it to the vehicle's onboard computer.

[0844] Input: User's voice

[0845] Output: Audio data

[0846] Specific operation: The device captures what the user says and sends it to the vehicle's computer.

[0847] Step 7:

[0848] Speech-to-text conversion and intent analysis

[0849] The server uses a generative AI model to convert speech to text and analyze intent.

[0850] Input: Audio data

[0851] Output: Text data, intent information

[0852] Specific operation: The server converts the audio data into text and analyzes the intent, "I want to listen to an audiobook."

[0853] Step 8:

[0854] Content Selection

[0855] The server will resume the previously listened to audiobook or recommend a new audiobook.

[0856] Input: Intent information, user viewing history

[0857] Output: Audiobook data

[0858] Specific operation: The server refers to the user's listening history and selects an appropriate audiobook.

[0859] Step 9:

[0860] Playing content

[0861] Your device will play the audiobook through the speaker.

[0862] Input: Audiobook data

[0863] Output: Audio content

[0864] Specific behavior: The device will play the selected audiobook through the speaker.

[0865] Step 10:

[0866] Emotion Recognition and Driver Assistance

[0867] The device's emotion recognition engine analyzes the driver's facial expressions and tone of voice to recognize their emotions.

[0868] The device transmits the recognized emotion data to the server.

[0869] Input: Driver's facial expression data, voice tone

[0870] Output: Emotion data

[0871] Specific operation: The device uses an emotion recognition engine to analyze the driver's emotions and sends the data to the server.

[0872] Step 11:

[0873] Driving assistance decisions

[0874] The server analyzes the emotional data and determines the driver's state.

[0875] The server determines the appropriate driving assistance and sends it to the device.

[0876] Input: Emotion data

[0877] Output: Driver assistance information

[0878] Specific operation: The server analyzes the emotion data and recommends rest spots if the driver is tired, or determines safe driving advice if the driver feels anxious.

[0879] Step 12:

[0880] Providing driving assistance

[0881] The device provides the driver with advice on rest and safe driving through a speaker.

[0882] Input: Driver assistance information

[0883] Output: Audio advice

[0884] Specific operation: The device will provide voice advice through the speaker, such as "Take a break at the next service area."

[0885] (Application example 3)

[0886] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0887] Conventional driving assistance systems have the ability to detect driver fatigue and loss of concentration, and to provide driving operation advice, but they have the problem of not being able to recognize the driver's emotions in real time and provide driving assistance or entertainment that responds to those emotions. Furthermore, they lack the convenience of allowing the driver to specify a destination verbally, and they lack the ability to provide entertainment based on the driver's hobbies and preferences. This has led to problems in which driver safety and comfort are not adequately ensured.

[0888] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0889] In this invention, the server is a driving assistance system that combines generative AI and a voice recognition system, and includes: means for providing driving operation advice; means for detecting driver fatigue or loss of concentration and issuing an alert or suggesting a break; means for recognizing the driver's emotions and adjusting the content of driving assistance according to those emotions; means for searching for the optimal route and starting navigation when the driver verbally specifies a destination; means for recommending and playing music, radio, audiobooks, etc. based on the driver's hobbies and preferences; and means for providing advice on safe driving when the driver feels anxious or scared. This makes it possible to provide appropriate driving assistance and entertainment according to the driver's emotions.

[0890] "Generative AI" is an artificial intelligence system that understands the driver's voice commands and generates appropriate responses and actions.

[0891] A "voice recognition system" is a system that converts the driver's voice into text and understands its content.

[0892] "Driving operation advice" is a function that suggests safe and efficient driving methods to the driver.

[0893] "Detection of fatigue and loss of concentration" is a function that monitors the driver's physical and mental state and detects fatigue or loss of concentration.

[0894] "Alerts and rest suggestions" is a function that notifies or suggests drivers to take a break when they feel fatigued or their concentration is declining.

[0895] "Emotion recognition" is a technology that determines a driver's emotions in real time from facial expressions, tone of voice, etc.

[0896] "Adjusting the content of driving assistance" is a function that changes the content of driving assistance provided depending on the emotional state of the driver.

[0897] "Specifying a destination" refers to the act of the driver verbally telling the system where they want to go.

[0898] "Optimal route search" is a function that calculates the most efficient route to a specified destination.

[0899] "Start navigation" is a function that guides the driver to the destination based on the optimal route.

[0900] "Recommendation based on hobbies and preferences" is a function that learns the driver's past behavior and preferences and suggests entertainment content based on that.

[0901] "Entertainment playback" is a function that provides content such as music, radio, and audiobooks to the driver.

[0902] "Safe driving advice" is a function that provides advice to encourage safe driving when the driver is feeling anxious or scared.

[0903] As an embodiment of the present invention, the following system can be constructed.

[0904] First, the server provides a driving assistance system that combines generative AI and a voice recognition system. Specifically, the server uses a voice recognition system and generative AI to recognize the driver's voice commands and generate appropriate responses and actions. The voice recognition system uses the SpeechRecognition library, and the generative AI uses a natural language processing model.

[0905] The server then uses sensors to detect driver fatigue or loss of concentration. These include cameras and biometric sensors such as heart rate monitors. Data from these sensors is analyzed in real time to assess the driver's condition. If fatigue or loss of concentration is detected, the server will issue an alert and suggest appropriate rest spots.

[0906] Furthermore, the server uses an emotion recognition engine to recognize the driver's emotions in real time. This emotion recognition engine uses the EmotionRecognizer library. Depending on the driver's emotions, the server adjusts the content of the driving assistance. For example, if the driver is feeling anxious or scared, the server will provide advice on safe driving.

[0907] In addition, when the driver verbally specifies the destination, the server searches for the optimal route and starts navigation. This function uses the NavigationSystem module. When the driver says, "I want to go to the nearest gas station," the server searches for the nearest gas station and starts navigation.

[0908] Furthermore, the server recommends and plays music, radio, audiobooks, etc. based on the driver's preferences. This function uses the EntertainmentSystem module. When the driver says, "I want to listen to an audiobook," the server will resume the audiobook that was previously being listened to or recommend a new audiobook.

[0909] For example, if a driver says, "I want to go to the nearest gas station," the system will search for the nearest gas station and begin navigation. If a driver says, "I want to listen to an audiobook," the system will resume the audiobook they were previously listening to. Furthermore, if the system determines that the driver is feeling tired or stressed, it will recommend appropriate rest spots.

[0910] An example of a prompt is as follows:

[0911] "If the driver says, 'I want to go to the nearest gas station,' the system will search for the nearest gas station and begin navigation. If the driver says, 'I want to listen to an audiobook,' the system will resume the audiobook they were previously listening to. Furthermore, if the system determines that the driver is feeling tired or stressed, it will recommend appropriate rest spots."

[0912] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0913] Step 1:

[0914] The server uses a voice recognition system to obtain voice commands from the driver.

[0915] Input: Driver's voice

[0916] Data processing: Convert speech to text using a speech recognition system (SpeechRecognition library).

[0917] Output: Textual voice commands

[0918] Step 2:

[0919] The server uses generative AI to analyze voice commands and generate appropriate responses and actions.

[0920] Input: Text-based voice command

[0921] Data processing: Generative AI (natural language processing models) are used to analyze commands and generate responses and actions.

[0922] Output: Response text or action instructions

[0923] Step 3:

[0924] The server collects data from biometric sensors to detect driver fatigue or loss of concentration.

[0925] Input: Data from biometric sensors (camera, heart rate monitor)

[0926] Data processing: Analyzes sensor data in real time and evaluates the driver's condition.

[0927] Output: Driver fatigue and lack of concentration

[0928] Step 4:

[0929] The server uses an emotion recognition engine to recognize the driver's emotions.

[0930] Input: Driver's facial expression and tone of voice

[0931] Data processing: Emotions are determined in real time using an emotion recognition engine (EmotionRecognizer library).

[0932] Output: Driver's emotional state

[0933] Step 5:

[0934] The server adjusts the content of the driving assistance depending on the emotional state of the driver.

[0935] Input: Driver's emotional state

[0936] Data processing: Changing the content of driving assistance based on emotional state.

[0937] Output: Adjusted driving assistance content

[0938] Step 6:

[0939] When the driver verbally specifies the destination, the server searches for the optimal route and starts navigation.

[0940] Input: Specify the destination in text format

[0941] Data processing: Calculate the optimal route using the NavigationSystem module.

[0942] Output: Instruction to start navigation

[0943] Step 7:

[0944] The server recommends and plays music, radio, audiobooks, etc. based on the driver's tastes and preferences.

[0945] Input: Driver's hobbies and preferences data

[0946] Data Processing: Recommend appropriate content using the EntertainmentSystem module.

[0947] Output: Recommended entertainment content

[0948] Step 8:

[0949] The server provides advice on safe driving if the driver feels anxious or scared.

[0950] Input: Driver's emotional state

[0951] Data processing: Generate safe driving advice based on emotional state.

[0952] Output: Safe driving advice text

[0953] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0954] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0955] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0956] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0957] [Second embodiment]

[0958] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0959] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0960] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0961] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0962] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0963] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0964] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0965] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0966] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0967] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0968] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0969] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0970] "Example 1"

[0971] One embodiment of the present invention is a driving assistance system that combines generative AI and a voice recognition system. This system provides advice to the driver regarding their driving operations. For example, when a driver attempts to change lanes, the AI ​​determines the approaching speed and distance of the vehicle behind and provides voice advice on whether the lane change is safe.

[0972] "Example 2"

[0973] Furthermore, this system detects driver fatigue or loss of concentration and issues an alert or suggests taking a break. Specifically, it analyzes the driver's line of sight, facial expressions, driving patterns, etc. to detect fatigue or loss of concentration. If detected, the AI ​​will recommend an appropriate rest spot and suggest taking a break via voice. "Example 3"

[0974] This system also allows the driver to set a destination through verbal interaction with the generative AI while driving. If the driver says, "I want to go to the nearest gas station," the AI ​​will search for the nearest gas station and begin navigation. It also makes recommendations based on the driver's tastes and preferences. For example, it can learn the driver's favorite music genres and artists and automatically play that music while driving.

[0975] "Example 4"

[0976] In addition, the system provides entertainment content such as music, radio, and audiobooks. If the driver says, "I want to listen to an audiobook," the AI ​​will resume the audiobook the driver was previously listening to or recommend a new audiobook, allowing the driver to enjoy entertainment while driving.

[0977] The processing flow of each embodiment will be described below.

[0978] "Example 1"

[0979] Step 1: When a driver attempts to change lanes, the generative AI determines the approaching speed and distance of the vehicle behind.

[0980] Step 2: Based on the AI's judgment, the driver is given audio advice on whether a safe lane change is possible.

[0981] "Example 2"

[0982] Step 1: Analyze the driver's gaze, facial expressions, driving patterns, etc. to detect fatigue and loss of concentration.

[0983] Step 2: If fatigue or a lack of concentration is detected, the AI ​​will recommend an appropriate rest spot. Step 3: The AI ​​will suggest a rest spot to the driver via voice.

[0984] "Example 3"

[0985] Step 1: When the driver says, "I want to go to the nearest gas station," the AI ​​searches for the nearest gas station.

[0986] Step 2: Start navigation with the searched gas station as your destination.

[0987] Step 3: Make recommendations based on the driver's tastes and preferences. For example, learn the driver's favorite music genres and artists and automatically play that music while driving.

[0988] "Example 4"

[0989] Step 1: When the driver says, "I want to listen to an audiobook," the AI ​​will resume the audiobook the driver was previously listening to or recommend a new audiobook. Step 2: The recommended audiobook will be provided to the driver via audio, allowing the driver to enjoy entertainment while driving.

[0990] Example 1

[0991] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0992] Conventional driver assistance systems do not provide sufficient real-time advice to drivers on their driving operations. They also lack the ability to detect driver fatigue or loss of concentration and provide appropriate alerts or suggest rest breaks. Furthermore, there is a need for a system that effectively utilizes the driver's voice input to provide driving operation advice.

[0993] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0994] In this invention, the server includes means for providing driving operation advice, means for detecting driver fatigue or loss of concentration and issuing an alert or suggesting a break, means for capturing voice input from the driver and converting the voice data into text data, means for analyzing the text data and understanding the driver's intentions, means for acquiring environmental data from vehicle sensors, means for generating advice using generative artificial intelligence based on the acquired environmental data, and means for converting the generated advice into voice and providing it to the driver. This makes it possible to support the driver in safe driving operations in real time.

[0995] "Generative AI" is an AI technology that has the ability to generate new information and advice based on data.

[0996] A "voice recognition system" is a technology that converts voice data into text data.

[0997] A "driving assistance system" is a system that provides advice and support to drivers regarding driving operations.

[0998] The "means for providing driving operation advice" is a function for generating and providing appropriate advice for the driver's driving operation.

[0999] "Means for detecting driver fatigue or loss of concentration and issuing alerts or suggesting breaks" refers to a function that monitors the driver's condition and issues a warning or suggests a break when fatigue or loss of concentration is detected.

[1000] The "means for capturing the driver's voice input and converting the voice data into text data" is a technology for collecting the driver's voice and converting it into text format.

[1001] The "means for analyzing text data and understanding the driver's intention" is a function for analyzing the converted text data and understanding the driver's intention.

[1002] "Means for acquiring environmental data from vehicle sensors" refers to technology for collecting surrounding environmental data using sensors mounted on a vehicle.

[1003] "Means for generating advice using generative artificial intelligence" is a function that generates appropriate advice using generative artificial intelligence based on collected data.

[1004] The "means for converting the generated advice into audio and providing it to the driver" is a technology for converting the generated advice into audio format and communicating it to the driver.

[1005] This invention is a driving assistance system that combines generative artificial intelligence and a voice recognition system, and provides real-time driving advice to the driver. Specific embodiments of this system will be described below.

[1006] Hardware and software used

[1007] Hardware: Microphones, speakers, rear camera, and radar sensors installed in the vehicle

[1008] Software: Generative AI models (e.g., GPT-4), speech recognition systems (e.g., Google Cloud Speech-to-Text), speech synthesis systems (e.g., Google Cloud Text-to-Speech)

[1009] System configuration

[1010] 1. Capture voice input

[1011] The device uses a microphone installed inside the vehicle to capture the driver's voice in real time.

[1012] Specifically, the driver says, "I want to change lanes."

[1013] 2. Speech Recognition and Text Conversion

[1014] The device sends the captured audio data to the Google Cloud Speech-to-Text API.

[1015] The server receives the voice data and converts it into text data.

[1016] Specifically, the speech "I want to change lanes" is converted into text "I want to change lanes."

[1017] 3. Intent Analysis

[1018] The server analyzes the converted text data and understands the driver's intentions.

[1019] Specifically, from the text "I want to change lanes," the system recognizes that the driver wants to change lanes.

[1020] 4. Acquisition of environmental data

[1021] The server collects data in real time from sensors installed in the vehicle (rear camera and radar).

[1022] Specifically, data on the approaching speed and distance of the vehicle behind is acquired.

[1023] 5. Generating Advice

[1024] The server generates advice using a generative artificial intelligence model (e.g., GPT-4) based on the acquired environmental data.

[1025] Specifically, the advice generated is "A vehicle behind is approaching. Changing lanes is dangerous."

[1026] 6. Audio output of advice

[1027] The device converts the generated advice into audio using the Google Cloud Text-to-Speech API.

[1028] The terminal provides the converted audio to the driver through the car's speakers.

[1029] Specifically, the driver hears a voice saying, "A vehicle behind is approaching. Changing lanes is dangerous."

[1030] Specific examples

[1031] As a specific example, consider the case where a driver attempts to change lanes.

[1032] The user says, "I want to change lanes."

[1033] The device captures this audio with its microphone and sends it to the Google Cloud Speech-to-Text API.

[1034] The server receives the voice data and converts it into text saying "I want to change lanes."

[1035] The server parses this text and recognizes that the driver wants to change lanes.

[1036] The server acquires data on the approaching speed and distance of the vehicle behind from the vehicle's rear camera and radar.

[1037] The server uses a generative artificial intelligence model to generate advice such as, "A vehicle behind is approaching. Changing lanes is dangerous."

[1038] The device converts this advice into audio using the Google Cloud Text-to-Speech API and provides it to the driver through the car's speakers.

[1039] Prompt Sentence Examples

[1040] "When a driver attempts to change lanes, the system determines whether the lane change is safe based on the approaching speed and distance of the vehicle behind, and generates appropriate advice."

[1041] In this way, the driver assistance system can support the driver in safe driving operations in real time.

[1042] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1043] Step 1:

[1044] Capturing voice input

[1045] The device uses a microphone installed inside the vehicle to capture the driver's voice in real time.

[1046] Input: Driver's voice (e.g., "I want to change lanes")

[1047] Output: Captured audio data

[1048] Specific operation: When the driver says, "I want to change lanes," the microphone picks up the voice and sends it to the terminal as voice data.

[1049] Step 2:

[1050] Speech recognition and text conversion

[1051] The terminal transmits the captured voice data to a voice recognition system.

[1052] The server converts the voice data into text data using a voice recognition system.

[1053] Input: Captured audio data

[1054] Output: Converted text data (e.g., "I want to change lanes")

[1055] How it works: The device sends voice data to the Google Cloud Speech-to-Text API, and the server converts the voice data into text, such as "I want to change lanes."

[1056] Step 3:

[1057] Intention Analysis

[1058] The server analyzes the converted text data and understands the driver's intentions.

[1059] Input: Converted text data (e.g., "I want to change lanes")

[1060] Output: Driver's intention (e.g., desire to change lanes)

[1061] Specific operation: The server analyzes the text data and recognizes the intention to "change lanes."

[1062] Step 4:

[1063] Obtaining environmental data

[1064] The server collects data in real time from sensors installed in the vehicle (rear camera and radar).

[1065] Input: Environmental data from sensors (e.g. approaching speed and distance of rear vehicles)

[1066] Output: Acquired environmental data

[1067] Specific operation: The server collects data on the approach speed and distance of the vehicle behind from the rear camera and radar.

[1068] Step 5:

[1069] Generating Advice

[1070] The server generates advice using a generative artificial intelligence model based on the acquired environmental data.

[1071] Input: Driver's intention, environmental data

[1072] Output: Generated advice (e.g. "A vehicle is approaching from behind. Changing lanes is dangerous.")

[1073] Specific operation: The server uses a generative artificial intelligence model to consider the approaching speed and distance of the vehicle behind and generates advice such as, "A vehicle behind is approaching. Changing lanes is dangerous."

[1074] Step 6:

[1075] Audio output of advice

[1076] The terminal converts the generated advice into speech using a speech synthesis system.

[1077] The terminal provides the converted audio to the driver through the car's speakers.

[1078] Input: Generated advice (in text format)

[1079] Output: Voice advice (e.g. "A vehicle behind you is approaching. Changing lanes is dangerous.")

[1080] Specific operation: The device converts the generated advice into audio using the Google Cloud Text-to-Speech API and provides it to the driver through the car's speakers.

[1081] (Application example 1)

[1082] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1083] Conventional driver assistance systems offer limited advice to drivers on their driving operations, and lack specific support for ensuring safety, particularly when changing lanes or turning left or right. They also lack the ability to detect driver fatigue or loss of concentration and provide alerts or suggestions for breaks at appropriate times. Furthermore, they lack the ability to recognize the driver's voice commands and provide the necessary information, resulting in low convenience while driving.

[1084] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1085] In this invention, the server is a driving assistance system that combines a generative AI and a voice recognition system, and includes: means for providing driving operation advice; means for detecting driver fatigue or loss of concentration and issuing an alert or suggesting a break; means for acquiring data in real time from the vehicle's cameras and sensors and analyzing it with the generative AI; means for determining the approaching speed and distance of a vehicle behind the driver when the driver attempts to change lanes or turn right or left, and providing voice advice on whether safe operation is possible; and means for recognizing the driver's voice commands and providing necessary information. This enables the driver to perform driving operations safely and efficiently, improving convenience and safety while driving.

[1086] "Generative AI" is an artificial intelligence technology that generates new information and predictions based on data.

[1087] A "voice recognition system" is a technology that analyzes voice and converts it into text data.

[1088] A "driving assistance system" is a system that assists the driver in driving operations and improves safety and convenience.

[1089] "Driving operation advice" refers to providing appropriate instructions and advice to the driver regarding operations.

[1090] "Detecting driver fatigue and loss of concentration" refers to monitoring the driver's condition and identifying fatigue and loss of concentration.

[1091] "Alerts and rest suggestions" refers to notifications that alert the driver or encourage them to take a break.

[1092] "Acquiring data in real time from the vehicle's cameras and sensors" means collecting information on the current situation in real time using cameras and sensors installed in the vehicle.

[1093] "Analysis by generative AI" means that generative AI processes the collected data and provides the necessary information and judgments.

[1094] "Judgment of approaching speed and distance of rear vehicles" means measuring the speed and distance of vehicles approaching from behind and evaluating safety.

[1095] "Voice advice" means providing voice instructions and advice to the driver based on the analysis results.

[1096] "Recognizing driver's voice commands" means analyzing the voice instructions given by the driver and understanding their content.

[1097] "Providing necessary information" means providing the driver with the information they require in an appropriate manner.

[1098] The system for carrying out this invention is a driving assistance system that combines generative AI and a voice recognition system. Specific embodiments of this system will be described below.

[1099] Hardware and software used

[1100] Hardware:

[1101] Camera mounted on the vehicle

[1102] Sensors installed in the vehicle

[1103] Microphone for voice recognition

[1104] Smartphone (for running applications)

[1105] software:

[1106] OpenCV (camera image acquisition)

[1107] TensorFlow (Analysis of generative AI models)

[1108] SpeechRecognition

[1109] gTTS (speech synthesis)

[1110] System processing overview

[1111] 1. Data Acquisition:

[1112] The system obtains real-time data from cameras and sensors installed in the vehicle. The cameras capture images of the vehicle behind, and the sensors collect information such as the vehicle's speed and distance.

[1113] 2. Data Analysis:

[1114] The acquired data is fed into a generative AI model to analyze the approaching speed and distance of the vehicle behind. Using TensorFlow, this data is processed in real time to determine whether a safe maneuver is possible.

[1115] 3. Audio advice:

[1116] Based on the analysis results, the system provides audio advice to the driver. For example, if it is safe to change lanes, it will notify the driver by saying "It is safe to change lanes," and if it is dangerous, it will warn the driver by saying "It is dangerous to change lanes." The audio is generated using gTTS and output from the speaker.

[1117] 4. Voice command recognition:

[1118] The system recognizes the driver's voice commands and provides the necessary information. It uses SpeechRecognition to analyze the voice instructions given by the driver and understand their content. For example, if the driver instructs the system to "turn right at the next intersection," the system will respond with "turn right at the next intersection."

[1119] Specific examples

[1120] Lane change advice:

[1121] When a driver attempts to change lanes, the system analyzes the approaching speed and distance of the vehicle behind and provides voice advice such as "Changing lanes is safe" or "Changing lanes is dangerous."

[1122] Voice command recognition and response:

[1123] When the driver issues a voice command such as "Turn right at the next intersection," the system responds with "Turn right at the next intersection."

[1124] Prompt Sentence Examples

[1125] "Analyze the approaching speed and distance of the vehicle behind you to determine whether a safe lane change is possible."

[1126] "Recognize the driver's voice commands and provide the necessary information."

[1127] In this way, the driver can perform driving operations safely and efficiently, improving convenience and safety while driving.

[1128] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1129] Step 1:

[1130] The server receives real-time data from the cameras and sensors installed in the vehicle. The cameras capture images of the vehicle behind, and the sensors collect information such as the vehicle's speed and distance. The input is the camera images and sensor data, and the output is a set of these data.

[1131] Step 2:

[1132] The server inputs the acquired data into a generative AI model to analyze the approaching speed and distance of the vehicle behind. Specifically, it uses TensorFlow to analyze camera footage and calculate the position and speed of the vehicle behind. The input is camera footage and sensor data, and the output is information on the approaching speed and distance of the vehicle behind.

[1133] Step 3:

[1134] The server provides audio advice to the driver based on the analysis results. For example, if it is safe to change lanes, it will notify the driver by saying "It is safe to change lanes," and if it is dangerous, it will warn the driver by saying "It is dangerous to change lanes." Voice is generated using gTTS and output from the speaker. The input is information on the approaching speed and distance of the vehicle behind, and the output is audio advice.

[1135] Step 4:

[1136] The server recognizes the driver's voice commands and provides the necessary information. Using SpeechRecognition, the system analyzes the voice instructions given by the driver and understands their content. For example, if the driver instructs the system to "turn right at the next intersection," the system responds with "turn right at the next intersection." The input is the driver's voice command, and the output is a voice response.

[1137] Step 5:

[1138] The server detects driver fatigue or loss of concentration and issues an alert or suggests taking a break. It analyzes data from sensors and monitors the driver's condition. If fatigue or loss of concentration is detected, it notifies the driver by voice, saying "Please take a break." The input is sensor data, and the output is an alert or voice notification suggesting a break.

[1139] Step 6:

[1140] The server makes recommendations tailored to the driver's hobbies and preferences. It analyzes the driver's past behavioral data and voice commands to suggest appropriate entertainment content and destinations. The input is the driver's past behavioral data and voice commands, and the output is recommendation information.

[1141] Step 7:

[1142] The server provides entertainment content such as music, radio, audiobooks, etc. Based on the driver's voice command, it selects and plays the appropriate content. The input is the driver's voice command, and the output is the playback of the entertainment content.

[1143] Example 2

[1144] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1145] Conventional driver assistance systems lack the ability to detect driver fatigue and concentration loss in real time and suggest breaks at appropriate times. Therefore, effective measures to ensure driver safety are needed. Furthermore, they lack the ability to provide recommendations and entertainment content tailored to the driver's hobbies and preferences.

[1146] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1147] In this invention, the server includes means for collecting data on the driver's line of sight, facial expression, and driving operation, means for analyzing the collected data and detecting driver fatigue or loss of concentration, means for recommending appropriate rest spots when fatigue or loss of concentration is detected, and means for suggesting rest by voice, thereby ensuring the safety of the driver and making it possible to suggest rest at appropriate times.

[1148] "Means for collecting data on the driver's gaze, facial expressions, and driving operations" refers to devices that use cameras and sensors installed inside the vehicle to obtain data such as the driver's gaze movements, changes in facial expressions, steering operation, and frequency of brake use in real time.

[1149] "Means for analyzing collected data to detect driver fatigue or loss of concentration" refers to a system that uses image recognition software and machine learning models to analyze collected data and detect signs of driver fatigue or loss of concentration.

[1150] The "means of recommending appropriate rest spots when fatigue or a decrease in concentration is detected" is a system that uses a GPS module to obtain the driver's current location, uses a map information service to search for nearby rest spots, and recommends them to the driver.

[1151] The "means for suggesting a break by voice" is a device that uses voice synthesis software to suggest to the driver by voice that they take a break.

[1152] "Generative AI" is an artificial intelligence system that uses natural language processing and machine learning techniques to generate appropriate responses and suggestions based on user input.

[1153] A "driving assistance system" is a system that provides functions such as driving operation advice, fatigue detection, and rest suggestion to assist drivers in safe driving.

[1154] "A means of making recommendations tailored to the driver's hobbies and preferences" is a system that recommends appropriate destinations and entertainment content based on data on the driver's past behavior and preferences.

[1155] The "means for providing entertainment content" is a device that provides entertainment content such as music, radio, audiobooks, etc. to the driver.

[1156] MODE FOR CARRYING OUT THE INVENTION

[1157] The present invention provides a system for collecting and analyzing data on a driver's line of sight, facial expression, and driving operation in order to ensure the safety of the driver. A specific embodiment of this system will be described below.

[1158] Hardware and software used

[1159] Hardware: Camera, sensors, microphone, GPS module

[1160] Software: Image recognition software (e.g., OpenCV), machine learning models (e.g., TensorFlow), speech synthesis software (e.g., Google Text-to-Speech API)

[1161] Data collection

[1162] The device uses cameras and sensors installed inside the vehicle to collect real-time data on the driver's gaze, facial expressions, and driving behavior. The camera captures the driver's face, and the sensors record steering wheel movements and brake use frequency.

[1163] Data analysis

[1164] The device inputs the collected data into image recognition software (OpenCV) and a machine learning model (TensorFlow) to analyze gaze, facial expressions, and driving behavior patterns. OpenCV analyzes the driver's blink frequency and eye movements, while TensorFlow compares the driving behavior patterns with the learning model.

[1165] Detecting fatigue and loss of concentration

[1166] Based on the analysis results, the device detects driver fatigue and loss of concentration. Increased blinking frequency and frequent averted gaze are detected as signs of fatigue and loss of concentration.

[1167] Recommend rest spots

[1168] The server obtains the current location using a GPS module and searches for nearby rest spots. The server uses a map information service (e.g., Google Maps API) to identify the nearest service area or parking area from the current location.

[1169] Audio break suggestions

[1170] The device uses a generative AI model to recommend suitable rest spots and uses speech synthesis software (Google Text-to-Speech API) to audibly suggest rest stops. The device generates a voice message such as "We recommend you take a break at a nearby service area" and plays it through the speaker.

[1171] Specific examples

[1172] Example 1: A camera detects that a user is blinking more frequently after driving for a long period of time. A sensor detects that the user's steering is becoming unstable. Based on this data, the device detects fatigue, and the server recommends a nearby rest area. The device makes a voice suggestion saying, "We recommend that you take a break at a nearby rest area."

[1173] Example 2: While driving on the highway, a camera detects that a user is frequently looking away. A sensor detects that the user is using the brakes more frequently. Based on this data, the device detects a decrease in concentration, and the server recommends the next parking area. The device suggests audibly, "We recommend that you take a break at the next parking area."

[1174] Prompt Sentence Examples

[1175] "Please explain how you would detect fatigue and suggest a break if the driver's gaze is frequently averted."

[1176] "Please explain what data you collect and how you analyze it to detect driver fatigue from facial expressions."

[1177] To ensure driver safety, the system collects and analyzes data in real time and suggests appropriate breaks.

[1178] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1179] Step 1:

[1180] Data collection

[1181] The device uses cameras and sensors installed inside the vehicle to collect data on the driver's gaze, facial expressions, and driving operations in real time.

[1182] Input: Camera footage, sensor data

[1183] Data processing: Facial features are extracted from camera footage, and frequency of steering and braking operations is recorded from sensor data.

[1184] Output: Gaze data, facial expression data, driving operation data

[1185] How it works: A camera captures the driver's face, and sensors record steering movements and how often the brakes are used.

[1186] Step 2:

[1187] Data analysis

[1188] The device inputs the collected data into image recognition software (OpenCV) and machine learning models (TensorFlow) to analyze patterns of gaze, facial expressions, and driving operations.

[1189] Input: Gaze data, facial expression data, driving operation data

[1190] Data calculation: OpenCV analyzes eye movements and blink frequency, and TensorFlow matches driving behavior patterns with the learning model.

[1191] Output: Gaze analysis results, facial expression analysis results, driving operation analysis results

[1192] How it works: OpenCV analyzes the driver's blink frequency and eye movements, and TensorFlow compares the driving behavior patterns with the learning model.

[1193] Step 3:

[1194] Detecting fatigue and loss of concentration

[1195] Based on the analysis results, the device detects the driver's fatigue and loss of concentration.

[1196] Input: Gaze analysis results, facial expression analysis results, driving operation analysis results

[1197] Data calculations: Detect signs of fatigue or loss of concentration based on blinking frequency, eye movement, and driving behavior patterns.

[1198] Output: Fatigue and concentration loss detection results

[1199] Specific behavior: Increased blinking rate and frequent averted gaze are detected as signs of fatigue and decreased concentration.

[1200] Step 4:

[1201] Recommend rest spots

[1202] The server uses a GPS module to obtain the current location and search for nearby rest spots.

[1203] Input: Current location data, fatigue and concentration loss detection results

[1204] Data calculation: Uses map information services to search for the nearest rest spot from your current location.

[1205] Output: Recommended rest spots

[1206] What happens: The server uses a map service to identify the nearest service or parking area from the current location.

[1207] Step 5:

[1208] Audio break suggestions

[1209] The device uses a generative AI model to recommend suitable rest spots and uses speech synthesis software (Google Text-to-Speech API) to suggest resting audibly.

[1210] Input: Recommended rest spots, fatigue and concentration loss detection results

[1211] Data calculation: Generative AI models select suitable rest spots, and speech synthesis software generates voice messages.

[1212] Output: Voice message

[1213] What happens: The device generates a voice message such as "We recommend you take a break at a nearby service area" and plays it through the speaker.

[1214] (Application example 2)

[1215] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1216] Conventional driving assistance systems lack the ability to detect driver fatigue and loss of concentration in real time and suggest appropriate breaks, which means driver safety is not fully ensured. Furthermore, technology for comprehensively analyzing the driver's gaze, facial expressions, and driving operation patterns is underdeveloped, resulting in low accuracy in detecting fatigue and loss of concentration. Furthermore, if break suggestions after detection are inappropriate, the driver may not recover sufficiently from fatigue, potentially increasing the risk of an accident.

[1217] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1218] In this invention, the server includes a means for analyzing the driver's line of sight, facial expressions, and driving operation patterns to detect fatigue or loss of concentration, a means for recommending appropriate rest spots and suggesting a break by voice when fatigue or loss of concentration is detected, a driving assistance system that combines generative AI and a voice recognition system, a means for providing driving operation advice, and a means for detecting the driver's fatigue or loss of concentration and issuing an alert or suggesting a break. This makes it possible to detect the driver's fatigue or loss of concentration with high accuracy and suggest a break at the appropriate time.

[1219] "Means of detecting fatigue and loss of concentration by analyzing the driver's line of sight, facial expressions, and driving operation patterns" refers to technology that uses cameras and sensors to monitor the driver's line of sight, facial expressions, and steering wheel and pedal operation patterns in real time, and analyzes this data to detect the driver's fatigue and loss of concentration.

[1220] "Means of recommending appropriate rest spots and suggesting a break via voice when detected" refers to a technology in which, when the driver's fatigue or lack of concentration is detected, AI selects the nearest rest spot and suggests a break to the driver via a voice assistant.

[1221] A "driving assistance system that combines generative AI and a voice recognition system" is a system that uses generative AI to generate appropriate responses to the driver's questions and instructions, and provides driving assistance by interacting with the driver through a voice recognition system.

[1222] "Means for providing driving operation advice" refers to technology that monitors the driver's driving operations and provides advice on driving methods and operations in real time as necessary.

[1223] "Means for detecting driver fatigue or loss of concentration and issuing an alert or suggesting a break" refers to technology that issues an alert and suggests taking an appropriate break when it detects that the driver is fatigued or losing concentration.

[1224] In order to implement the present invention, the following hardware and software are used.

[1225] Hardware

[1226] Camera: Used to monitor the driver's gaze and facial expressions.

[1227] Sensors: Used to detect driving patterns.

[1228] Microphone and speaker: Used for voice recognition and voice assistant.

[1229] software

[1230] OpenCV: A library for image processing.

[1231] dlib: A library for face and landmark detection.

[1232] SciPy: A library for performing distance calculations.

[1233] gTTS: A library for speech generation.

[1234] mpg321: Software for audio playback.

[1235] Program processing

[1236] The server uses cameras and sensors to monitor the driver's gaze, facial expressions, and driving behavior patterns in real time. This data is analyzed using OpenCV and dlib to detect driver fatigue and loss of concentration. Specifically, the eye aspect ratio (EAR) is calculated, and fatigue is detected when it falls below a certain threshold.

[1237] If fatigue is detected, the server uses AI to identify the nearest rest spot, and the voice assistant uses gTTS to suggest a break to the driver. The audio is played using mpg321.

[1238] Specific examples

[1239] For example, if a driver has been driving for a long time, the camera will monitor the driver's eye movements and detect a decrease in the aspect ratio of the eyes. At this time, the server will make a voice prompt saying, "Fatigue has been detected. Please take a break."

[1240] Prompt Sentence Examples

[1241] Example prompts to input to a generative AI model:

[1242] I would like to develop a system that detects driver fatigue and loss of concentration and recommends appropriate rest spots. Please tell me how to analyze the driver's gaze, facial expressions, and driving operation patterns to detect fatigue and loss of concentration. Also, please tell me how to suggest a break by voice if fatigue or loss of concentration is detected.

[1243] In this way, a driving assistance system for improving driver safety can be realized.

[1244] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1245] Step 1:

[1246] The server captures the driver's face using a camera. The input is video data from the camera, and the output is an image of the driver's face. Specifically, the camera captures the driver's face in real time and sends the video data to the server.

[1247] Step 2:

[1248] The server uses dlib to detect the driver's face and identify facial landmarks. The input is the driver's face image, and the output is the coordinates of the facial landmarks. Specifically, it applies dlib's face detection algorithm to identify 68 facial landmarks.

[1249] Step 3:

[1250] The server calculates the eye aspect ratio (EAR) using OpenCV. The input is the face landmark coordinates, and the output is the eye aspect ratio. Specifically, the eye landmark coordinates are used to calculate the vertical and horizontal distances of the eyes, and then the EAR is calculated.

[1251] Step 4:

[1252] The server compares the calculated EAR with a threshold to detect fatigue. The input is the eye aspect ratio, and the output is a fatigue detection flag. Specifically, if the EAR is below the set threshold, fatigue is detected.

[1253] Step 5:

[1254] If fatigue is detected, the server uses a generative AI model to recommend the nearest rest spot. The input is a fatigue detection flag and current location information, and the output is information about rest spots. Specifically, the generative AI model selects the optimal rest spot based on the current location information.

[1255] Step 6:

[1256] The server uses gTTS to generate a voice message suggesting a break. The input is information about rest spots, and the output is an audio file. Specifically, the server generates a voice message saying, "Fatigue has been detected. Please take a break."

[1257] Step 7:

[1258] The server plays the generated audio using mpg321. The input is an audio file, and the output is an audio alert. Specifically, the server plays the generated audio file and suggests the driver to take a break.

[1259] In this way, it is possible to detect driver fatigue and loss of concentration with high accuracy and suggest rest breaks at appropriate times.

[1260] Example 3

[1261] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1262] Conventional driving assistance systems require the driver to manually set a destination, which reduces safety while driving. They also lack the ability to provide entertainment content based on the driver's preferences, which reduces driving comfort. Furthermore, drivers must manually control playback when enjoying audiobooks, which reduces convenience.

[1263] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1264] In this invention, the server includes means for acquiring voice input, means for performing voice recognition, means for analyzing voice data using a generative AI model and determining appropriate actions, means for starting navigation, means for providing entertainment content based on the driver's hobbies and preferences, and means for providing audiobooks. This allows the driver to set a destination and receive entertainment content by voice input without performing manual operations while driving.

[1265] "Means for obtaining voice input" refers to a device or function for capturing the driver's voice in real time and transmitting it to the system.

[1266] "Means for performing speech recognition" refers to software or algorithms for converting captured speech data into text data.

[1267] "Means for analyzing voice data using a generative AI model and determining appropriate actions" refers to a system that uses a generative AI model to analyze text data and generate appropriate instructions and actions based on the driver's intentions.

[1268] A "means for initiating navigation" is software or a device that calculates a route to a destination and provides audio and visual guidance to the driver.

[1269] "Means for providing entertainment content based on the driver's tastes and preferences" refers to a system that learns the driver's past behavior and preferences, and based on that, selects and provides appropriate entertainment content such as music, radio, and audiobooks.

[1270] The "means for providing audiobooks" is a system for resuming an audiobook that was previously being listened to or recommending and playing a new audiobook when a driver expresses a desire to listen to an audiobook.

[1271] This invention is a driving assistance system that acquires voice input and analyzes it using a generative AI model to provide appropriate navigation and entertainment content to the driver. Specific embodiments of this system are described below.

[1272] Hardware and software used

[1273] Hardware: On-board computer, microphone, speaker, GPS module

[1274] Software: generative AI models (e.g., GPT-4), speech recognition software (e.g., Google Speech-to-Text), navigation software (e.g., Google Maps API), music streaming services (e.g., Spotify API)

[1275] System Operation

[1276] 1. Acquiring voice input

[1277] Terminal: The on-board computer receives the driver's voice through a microphone. The microphone captures the driver's speech in real time and sends it to the on-board computer as voice data.

[1278] Example: When a driver says, "I want to go to the nearest gas station," a microphone captures the voice and sends it to the car's computer.

[1279] 2. Voice Recognition

[1280] Server: The voice recognition software receives the voice data sent from the vehicle's computer and converts it into text. The voice recognition software uses services such as Google Speech-to-Text to achieve highly accurate voice recognition.

[1281] Example: Speech data such as "I want to go to the nearest gas station" is converted into text data.

[1282] 3. Analysis using generative AI models

[1283] Server: A generative AI model (e.g., GPT-4) analyzes the text data and determines the appropriate action. The generative AI model understands the driver's intentions and generates prompts to search for the required information.

[1284] Example: The instruction generated is "Find the nearest gas station and start navigation."

[1285] 4. Start navigation

[1286] Server: Navigation software (e.g., Google Maps API) receives instructions from the generative AI model, searches for the nearest gas station, calculates the optimal route from the search results, and generates navigation data.

[1287] Terminal: The on-board computer receives navigation data and provides audio and visual directions to the driver.

[1288] Example: Routes to the nearest gas station are displayed on the in-car display and voice guidance begins.

[1289] 5. Entertainment

[1290] Server: The generative AI model learns the driver's preferences and selects appropriate entertainment content. It uses music streaming services (e.g., Spotify API) to search for songs by the driver's favorite music genres and artists.

[1291] Device: The onboard computer plays the selected music.

[1292] Example: The driver's favorite jazz music is played through the car speakers.

[1293] 6. Audiobook Offerings

[1294] Server: When the driver says, "I want to listen to an audiobook," the generative AI model resumes the audiobook that was previously being listened to or recommends a new audiobook. It uses an audiobook service (e.g., Audible API) to search for a suitable audiobook.

[1295] Device: The vehicle's onboard computer plays the selected audiobook.

[1296] Example: An audiobook the driver was previously listening to continues playing, or a new audiobook starts playing over the car speakers.

[1297] Examples of prompt statements

[1298] Prompt 1: "I want to go to the nearest gas station."

[1299] Prompt 2: "I want to listen to an audiobook."

[1300] Prompt 3: "Play some jazz music."

[1301] This system allows the driver to easily set a destination and enjoy entertainment by voice while driving. The flow of the specification process in the third embodiment will be described with reference to FIG.

[1302] Step 1:

[1303] Acquiring voice input

[1304] Terminal: The on-board computer picks up the driver's voice through a microphone.

[1305] Input: Driver utterances (e.g., "I want to go to the nearest gas station").

[1306] How it works: The microphone captures the driver's speech in real time and sends it to the onboard computer as audio data.

[1307] Output: Audio data.

[1308] Step 2:

[1309] Voice Recognition

[1310] Server: Speech recognition software receives the voice data sent from the vehicle's computer and converts it into text.

[1311] Input: Audio data.

[1312] What happens: Speech recognition software (e.g., Google Speech-to-Text) analyzes the audio data and generates corresponding text data.

[1313] Output: Text data (e.g. "I want to go to the nearest gas station").

[1314] Step 3:

[1315] Analysis using generative AI models

[1316] Server: A generative AI model (e.g., GPT-4) analyzes the text data and determines the appropriate action.

[1317] Input: Text data.

[1318] How it works: The generative AI model analyzes text data, understands the driver's intent, and generates prompts to search for the required information.

[1319] Output: Instructions (e.g., "Find the nearest gas station and start navigation").

[1320] Step 4:

[1321] Start navigation

[1322] Server: Navigation software (e.g., Google Maps API) receives instructions from the generative AI model and searches for the nearest gas station.

[1323] Input: Instruction data.

[1324] What it does: Navigation software finds the nearest gas station and calculates the optimal route.

[1325] Output: Navigation data (route information).

[1326] Step 5:

[1327] Providing navigation

[1328] Terminal: The on-board computer receives navigation data and provides audio and visual directions to the driver.

[1329] Input: Navigation data.

[1330] Specific operation: Route information is displayed on the in-car display and voice guidance is provided through the speaker.

[1331] Output: Navigation instructions to the driver.

[1332] Step 6:

[1333] Entertainment provision

[1334] Server: The generative AI model learns the driver's preferences and selects appropriate entertainment content.

[1335] Input: Driver's preferences data.

[1336] Specific behavior: Uses music streaming services (e.g., Spotify API) to search for songs by the driver's preferred music genres and artists.

[1337] Output: Entertainment content (music data).

[1338] Step 7:

[1339] Entertainment Regeneration

[1340] Device: The onboard computer plays the selected music.

[1341] Input: Entertainment content (music data).

[1342] Specific operation: Play selected music through the car speakers.

[1343] Output: Music playback.

[1344] Step 8:

[1345] Audiobook offerings

[1346] Server: If the driver says, "I want to listen to an audiobook," the generative AI model will resume the audiobook they were previously listening to or recommend a new audiobook.

[1347] Input: Driver utterance (e.g., "I want to listen to an audiobook").

[1348] What it does: Uses an audiobook service (e.g., Audible API) to search for a suitable audiobook.

[1349] Output: Audiobook data.

[1350] Step 9:

[1351] Playing an audiobook

[1352] Device: The vehicle's onboard computer plays the selected audiobook.

[1353] Input: Audiobook data.

[1354] Specific behavior: Plays a selected audiobook through the car's speakers.

[1355] Output: Audiobook playback.

[1356] (Application example 3)

[1357] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1358] Conventional driver assistance systems lack the ability to detect driver fatigue or loss of concentration, or to provide driving advice. They also have limited capabilities for providing entertainment content while driving, or for making recommendations based on the driver's hobbies and preferences. Furthermore, they lack the ability to search for the nearest facility based on the driver's verbal input, initiate navigation, or automatically play the driver's preferred music or audiobook. To address these issues, more advanced driver assistance systems are needed.

[1359] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1360] In this invention, the server is a driving assistance system that combines a generative AI and a voice recognition system, and includes: means for providing driving operation advice; means for detecting driver fatigue or loss of concentration and issuing alerts or suggesting rest; means for setting a destination through verbal interaction with the generative AI while driving; means for making recommendations tailored to the driver's hobbies and preferences; means for providing entertainment content such as music, radio, and audiobooks; means for searching for the nearest facility based on what the driver has said verbally and starting navigation; and means for automatically playing music or audiobooks based on the driver's hobbies and preferences. This allows the driver to not only reach their destination safely and comfortably while driving, but also enjoy entertainment.

[1361] "Generative AI" refers to artificial intelligence that generates new information and content based on user input.

[1362] A "voice recognition system" is a system that has the technology to convert voice into text.

[1363] A "driving assistance system" is a system that assists the driver in driving and improves safety and comfort.

[1364] "Driving operation advice" is a function that suggests appropriate driving methods and operations to the driver.

[1365] "Detection of fatigue and loss of concentration" refers to a function that detects the driver's state of fatigue or loss of concentration.

[1366] "Alerts and rest suggestions" refers to a function that issues a warning to the driver and suggests taking a break.

[1367] "Destination setting" is a function that allows the driver to set the location they want to go to in the system.

[1368] "Recommendations based on hobbies and preferences" is a function that recommends content and services based on the driver's preferences.

[1369] "Entertainment content" refers to entertainment content such as music, radio, and audiobooks.

[1370] "Search for nearest facility" is a function that searches for the nearest facility based on what the driver says verbally.

[1371] "Start navigation" refers to the function that provides directions to your destination.

[1372] "Automatic music and audiobook playback" refers to a function that automatically plays music and audiobooks based on the driver's preferences.

[1373] The system for carrying out this invention is a driving assistance system that combines generative AI and a voice recognition system. Specific embodiments of this system will be described below.

[1374] Hardware and software used

[1375] The server uses the following hardware and software:

[1376] Hardware: Microphone (for audio input), Speaker (for audio output)

[1377] Software: speech_recognition library (speech recognition), pyttsx3 library (speech synthesis), requests library (API requests)

[1378] Data processing and calculation

[1379] The server performs the following data processing and calculations.

[1380] 1. Speech-to-text conversion: Use a microphone to collect the driver's voice and convert it to text using the speech_recognition library.

[1381] 2. Keyword detection: Detect specific keywords from the converted text and select the appropriate action.

[1382] 3. API Request: Send an API request using the requests library to search for, for example, the nearest gas station based on keywords.

[1383] 4. Audio output: Search results and recommendations are output as audio using the pyttsx3 library.

[1384] Specific examples

[1385] When the driver says, "I want to go to the nearest gas station," the server performs the following process.

[1386] 1. Collect audio with a microphone and convert it to text using the speech_recognition library.

[1387] 2. Detect the keyword "gas station" from the text and send an API request to find the nearest gas station.

[1388] 3. Receive the search results and output the name of the nearest gas station using the pyttsx3 library.

[1389] Also, if the driver says, "I want to listen to an audiobook," the server performs the following process.

[1390] 1. Collect audio with a microphone and convert it to text using the speech_recognition library.

[1391] 2. Detect the keyword "audiobook" from the text and refer to the driver's taste and preference data to resume the audiobook that was previously being listened to or recommend a new audiobook.

[1392] 3. Output the recommendations as audio using the pyttsx3 library.

[1393] Prompt Sentence Examples

[1394] An example of a prompt for the generative AI model is as follows:

[1395] If the user says "I want to go to the nearest gas station," create a program that searches for the nearest gas station and starts navigation. Also, if the user says "I want to listen to an audiobook," add a function to resume the audiobook that was previously being listened to.

[1396] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1397] Step 1:

[1398] The server uses a microphone to collect the driver's voice. The input is the driver's voice and the output is audio data. Specifically, the microphone captures the driver's speech in real time.

[1399] Step 2:

[1400] The server converts the collected voice data into text using the speech_recognition library. The input is voice data and the output is text data. Specifically, the voice recognition engine analyzes the voice data and generates the corresponding text.

[1401] Step 3:

[1402] The server detects specific keywords from the converted text. The input is the text data, and the output is the detected keywords. Specifically, the text analysis algorithm searches for keywords in the text and extracts the relevant keywords.

[1403] Step 4:

[1404] The server sends an appropriate API request based on the detected keyword. For example, if the keyword "gas station" is detected, it sends an API request to search for the nearest gas station. The input is the keyword, and the output is the response data of the API request. Specifically, it uses the requests library to send a request to an external API and receive a response.

[1405] Step 5:

[1406] The server analyzes the response data of the API request and extracts the necessary information. The input is the response data, and the output is the extracted information. Specifically, the response data analysis algorithm extracts the necessary information (for example, the name and location of the nearest gas station).

[1407] Step 6:

[1408] The server outputs the extracted information as voice using the pyttsx3 library. The input is the extracted information, and the output is voice data. Specifically, the voice synthesis engine converts the text information into voice and conveys it to the driver through the speaker.

[1409] Step 7:

[1410] When a driver says, "I want to listen to an audiobook," the server refers to the driver's interest and preference data and either resumes the audiobook they were listening to previously or recommends a new audiobook. The input is the driver's statement and interest and preference data, and the output is information about the recommended audiobook. Specifically, the server retrieves the driver's past listening history from the database and selects an appropriate audiobook.

[1411] Step 8:

[1412] The server outputs the information about the recommended audiobook as audio using the pyttsx3 library. The input is the information about the recommended audiobook, and the output is audio data. Specifically, the speech synthesis engine converts the text information into audio and transmits it to the driver through the speaker.

[1413] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1414] "Example 1"

[1415] One embodiment of the present invention is a driving assistance system that incorporates an emotion engine. This system recognizes the driver's emotions from their facial expressions and tone of voice, and adjusts the content of the driving assistance accordingly. For example, if the system determines that the driver is feeling angry or anxious, it will play calming music or encourage the driver to take deep breaths to calm the driver.

[1416] "Example 2"

[1417] Another embodiment of the present invention is a system in which an emotion engine recognizes the driver's emotions and adjusts the entertainment content provided in accordance with the emotions. For example, if the system determines that the driver is feeling happy or excited, it automatically selects and plays upbeat music that the driver prefers. Conversely, if the system determines that the driver is depressed, it automatically selects and plays calming music that the driver prefers.

[1418] "Example 3"

[1419] Furthermore, in another embodiment of the present invention, there is a system in which the emotion engine recognizes the driver's emotions and adjusts the content of driving assistance according to those emotions. For example, if the system determines that the driver is feeling tired or stressed, it recommends appropriate rest spots and suggests that the driver take a break.

[1420] "Example 4"

[1421] In another embodiment of the present invention, an emotion engine recognizes the driver's emotions and adjusts driving advice accordingly. For example, if the system determines that the driver is feeling anxious or scared, it provides the driver with advice on safe driving.

[1422] The processing flow of each embodiment will be described below.

[1423] "Example 1"

[1424] Step 1: The emotion engine recognizes the driver's facial expressions and tone of voice.

[1425] Step 2: The emotion engine determines the driver's emotion.

[1426] Step 3: The driving assistance system adjusts the content of driving assistance based on the emotion engine's judgment.

[1427] Step 4: If the system determines that the driver is feeling angry or anxious, it will try to calm the driver by playing calming music or asking the driver to take deep breaths.

[1428] "Example 2"

[1429] Step 1: The emotion engine recognizes the driver's facial expressions and tone of voice.

[1430] Step 2: The emotion engine determines the driver's emotion.

[1431] Step 3: The driving assistance system adjusts the entertainment content provided based on the emotion engine's judgment.

[1432] Step 4: If the system determines that the driver is feeling happy or excited, it will automatically select and play upbeat music that the driver prefers. Conversely, if the system determines that the driver is feeling depressed, it will automatically select and play calming music that the driver prefers.

[1433] "Example 3"

[1434] Step 1: The emotion engine recognizes the driver's facial expressions and tone of voice.

[1435] Step 2: The emotion engine determines the driver's emotion.

[1436] Step 3: The driving assistance system adjusts the content of driving assistance based on the emotion engine's judgment.

[1437] Step 4: If the system determines that the driver is feeling fatigued or stressed, it will recommend appropriate rest spots and suggest that the driver take a break.

[1438] "Example 4"

[1439] Step 1: The emotion engine recognizes the driver's facial expressions and tone of voice.

[1440] Step 2: The emotion engine determines the driver's emotion.

[1441] Step 3: The driver assistance system adjusts the driving advice based on the emotion engine's judgment.

[1442] Step 4: If the system determines that the driver is feeling anxious or scared, it will provide the driver with advice on safe driving.

[1443] Example 1

[1444] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1445] Conventional driving assistance systems have limited capabilities to provide advice to drivers on their driving operations or to detect driver fatigue or loss of concentration, making it difficult to fully ensure driver safety. Furthermore, they do not adjust driving assistance according to the driver's emotional state, making it impossible to reduce the driver's stress and anxiety. Furthermore, they lack the ability to measure the approaching speed and distance of vehicles behind in real time and determine safe lane changes, making it impossible to improve driver safety.

[1446] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1447] In this invention, the server is a driving assistance system that combines generative AI and a voice recognition system, and includes a means for providing driving operation advice, a means for detecting driver fatigue or loss of concentration and issuing an alert or suggesting a break, a means for measuring the approaching speed and distance of vehicles behind and determining whether a safe lane change is possible, and a means for recognizing the driver's emotions from their facial expressions and tone of voice and adjusting the content of driving assistance according to those emotions. This makes it possible to increase driver safety, reduce driver stress and anxiety, and provide a more comfortable driving environment.

[1448] "Generative AI" is a system that uses artificial intelligence technology to analyze data and generate appropriate advice and information.

[1449] A "voice recognition system" is a technology that analyzes voice data and converts it into text data.

[1450] A "driving assistance system" is a system that assists the driver in driving operations and improves safety and comfort.

[1451] "Driving advice" means instructions or advice provided to a driver to help them drive safely and efficiently.

[1452] "Detection of driver fatigue and loss of concentration" is a technology that monitors the driver's physical and mental state and identifies fatigue and loss of concentration.

[1453] "Alerts and rest suggestions" are notifications or suggestions to alert the driver or encourage them to take a break.

[1454] "Measurement of approaching speed and distance of rear vehicles" is a technology for measuring the speed and distance of other vehicles present behind the vehicle.

[1455] "Safe lane change judgment" is a technology that analyzes whether lane changes can be made safely and provides appropriate instructions to the driver.

[1456] "Recognition of driver's facial expressions and tone of voice" is a technology that analyzes the driver's facial expressions and tone of voice to identify their emotional state.

[1457] "Adjusting the content of driving assistance" is a technology that changes the content of driving assistance provided depending on the emotional state of the driver.

[1458] This invention is a driving assistance system that combines generative AI and a voice recognition system, providing driving advice, detecting driver fatigue or loss of concentration and issuing alerts or suggesting rest. It also includes functions to measure the approaching speed and distance of vehicles behind to determine whether a safe lane change is possible, and recognizing the driver's emotions from facial expressions and tone of voice and adjusting the content of driving assistance accordingly.

[1459] Hardware and software used

[1460] 1. The device collects data using hardware such as cameras, LIDAR sensors, and highly sensitive microphones mounted on the vehicle, allowing it to measure the approaching speed and distance of vehicles behind, as well as the driver's facial expressions and tone of voice in real time.

[1461] 2. The server uses generative AI (e.g., OpenAI's GPT-4) and emotion recognition engine (e.g., Affectiva's Emotion Recognition API) to analyze the collected data.

[1462] 3. The device converts the generated advice and driving assistance content into speech using a voice recognition system (e.g., Google Cloud Speech-to-Text API) and provides it to the driver.

[1463] Explanation of program processing

[1464] 1. Data Collection

[1465] The device uses the vehicle's onboard camera and LIDAR sensor to measure the approaching speed and distance of the vehicle behind in real time. The camera captures the vehicle behind and processes the video data to determine the vehicle's position. The LIDAR sensor uses lasers to measure distance.

[1466] The device uses the vehicle's on-board camera and microphone to collect the driver's facial expressions and tone of voice. The camera captures the driver's face, and the microphone records the driver's voice.

[1467] 2. Data Transmission

[1468] The device sends the collected data to a server, including the speed, distance, and direction of the vehicle behind, as well as the driver's facial expression, tone of voice, and heart rate. The device sends the data to the server via the Internet, and the data is encrypted before transmission.

[1469] 3. Data Analysis

[1470] The server inputs the received data into the generative AI, which analyzes it to determine whether it is safe to change lanes. The generative AI performs the analysis using prompts.

[1471] The server inputs the received data into an emotion recognition engine to analyze the driver's emotions. The emotion recognition engine performs the analysis using the prompt sentence.

[1472] 4. Advice Generation

[1473] Based on the analysis results, the generative AI generates appropriate advice for the driver, such as "A vehicle behind you is approaching. Please refrain from changing lanes."

[1474] Based on the analysis results, the emotion recognition engine generates appropriate driving assistance according to the driver's emotions. For example, it generates advice such as "Take a deep breath" and instructs the driver to play calming music.

[1475] 5. Audio conversion and execution

[1476] The device converts the generated advice into voice using a voice recognition system and provides it to the driver. The voice recognition system converts the text data into voice data and provides it to the driver through the speaker. At the same time, the music playback system plays the selected music.

[1477] Examples of concrete examples and prompts

[1478] Specific examples of lane change advice

[1479] The terminal measures the approaching speed of the vehicle behind as 80 km / h and the distance as 50 m, and sends this to the server.

[1480] The server inputs a prompt sentence into the generative AI, and as a result of the analysis, it generates the advice, "A vehicle behind you is approaching. Please refrain from changing lanes."

[1481] The device converts this advice into audio and provides it to the driver.

[1482] Specific examples of emotion-aware advice

[1483] The device collects data on the driver's facial expressions to detect anger and sends it to a server.

[1484] The server inputs the prompt sentence into the emotion recognition engine, and generates the advice "Take a deep breath" as an analysis result.

[1485] The device converts this advice into audio and provides it to the driver, while playing soothing music.

[1486] Prompt Sentence Examples

[1487] 1. Lane change advice prompt

[1488] The vehicle behind is approaching at 80km / h and the distance is 50m. Advise the driver whether it is safe to change lanes.

[1489] 2. Emotion-aware advice prompts

[1490] The driver is angry. Please provide some advice to help the driver calm down.

[1491] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1492] Step 1: Data collection

[1493] The device uses the vehicle's on-board camera and LIDAR sensor to measure the approaching speed and distance of the vehicle behind in real time. The camera captures images of the vehicle behind and processes the image data to determine the vehicle's position. The LIDAR sensor uses a laser to measure distance. The input is the camera image and LIDAR data, and the output is data on the speed and distance of the vehicle behind.

[1494] Step 2: Collecting Emotional Data

[1495] The device collects the driver's facial expressions and tone of voice using the camera and microphone installed in the vehicle. The camera captures the driver's face, and the microphone records the driver's voice. The input is the camera image and audio data, and the output is the driver's facial expression data and tone of voice data.

[1496] Step 3: Send data

[1497] The terminal sends the collected data to a server. This data includes the speed, distance, and direction of the vehicle behind, as well as the driver's facial expression, tone of voice, and heart rate. The terminal sends the data to the server via the Internet, and the data is encrypted before transmission. The input is the various collected data, and the output is the data sent to the server.

[1498] Step 4: Data analysis

[1499] The server inputs the received data into the generative AI, which analyzes it to determine whether it is safe to change lanes. The generative AI performs the analysis using prompts. The input is data on the speed and distance of the vehicle behind, and the output is advice on the safety of changing lanes.

[1500] Step 5: Sentiment Analysis

[1501] The server inputs the received data into an emotion recognition engine to analyze the driver's emotions. The emotion recognition engine performs the analysis using prompt sentences. The input is the driver's facial expression data and tone of voice data, and the output is information about the driver's emotional state.

[1502] Step 6: Advice Generation

[1503] Based on the analysis results, generative AI generates appropriate advice for the driver. For example, it generates advice such as, "A vehicle behind is approaching. Please refrain from changing lanes." The input is data related to the safety of lane changes, and the output is text data of the advice.

[1504] Step 7: Emotion-based advice generation

[1505] Based on the analysis results, the emotion recognition engine generates appropriate driving assistance according to the driver's emotions. For example, it generates advice such as "Take a deep breath" and instructs the driver to play calming music. The input is data about the driver's emotional state, and the output is text data of advice according to the emotion.

[1506] Step 8: Convert audio and run

[1507] The terminal converts the generated advice into voice using a voice recognition system and provides it to the driver. The voice recognition system converts the text data into voice data and provides it to the driver through the speaker. At the same time, the music playback system plays the selected music. The input is the text data of the advice, and the output is the voice data and the played music.

[1508] (Application example 1)

[1509] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1510] Conventional driver assistance systems are limited to providing advice on driver operation and fatigue detection, and lack support based on the driver's emotional state. Furthermore, they do not provide specific advice to ensure safety when changing lanes or making turns. This can increase driver stress and anxiety, potentially reducing driving safety.

[1511] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1512] In this invention, the server is a driving assistance system that combines generative AI and a voice recognition system, and includes: means for providing driving operation advice; means for detecting driver fatigue or loss of concentration and issuing an alert or suggesting a break; means for recognizing the driver's emotions from their facial expressions and tone of voice and adjusting the content of driving assistance according to those emotions; means for judging the approaching speed and distance of vehicles behind the driver when the driver changes lanes or turns right or left and providing voice advice on safe timing; and means for playing calming music or encouraging the driver to take deep breaths when the driver feels angry or anxious. This provides appropriate assistance according to the driver's emotional state, making it possible to improve driving safety and comfort.

[1513] "Generative AI" is an artificial intelligence technology that generates new information and content based on data.

[1514] A "voice recognition system" is a technology that analyzes human speech and converts it into text data.

[1515] A "driving assistance system" is a technology that assists the driver in driving operations and improves safety and comfort.

[1516] "Driving operation advice" refers to providing appropriate instructions and advice to the driver regarding the driving operations they perform.

[1517] "Detection of fatigue and loss of concentration" means monitoring the driver's physical and mental state to detect fatigue and loss of concentration.

[1518] "Alerts and rest suggestions" refers to notifications that alert the driver or encourage them to take a break.

[1519] "Emotion recognition" refers to determining a driver's emotional state from their facial expressions and tone of voice.

[1520] "Adjusting the content of driving assistance" means changing the content of assistance provided in accordance with the emotional state of the driver.

[1521] "Lane change and right / left turn advice" refers to voice instructions to the driver on the safe timing to change lanes or turn right or left.

[1522] "Judgment of approach speed and distance" means measuring the speed and distance of the vehicle behind and assessing safety.

[1523] "Playing calming music" means playing music that has a relaxing effect to calm the driver's emotions.

[1524] "Deep breathing prompt" means instructing the driver to take a deep breath.

[1525] As an embodiment of the present invention, the Emotion-Aware Driving Assistant (EADA) system will be described as an example. The EADA system is a driving assistance system that combines generative AI and a voice recognition system, and provides advice to the driver regarding driving operations and assistance according to the driver's emotional state.

[1526] System Configuration

[1527] The EADA system uses the following hardware and software:

[1528] Hardware:

[1529] Camera: Used to capture the driver's facial expressions.

[1530] Microphone: Used to capture the driver's tone of voice.

[1531] Speaker: Used to play voice advice and music.

[1532] software:

[1533] OpenCV: Used to process the images from the camera and perform facial recognition.

[1534] TensorFlow / Keras: Used to run the emotion recognition model.

[1535] pyttsx3: Used as a speech synthesis engine.

[1536] Data processing and calculation

[1537] The server converts the image captured by the camera to grayscale and performs facial recognition. It then extracts the area of ​​the recognized face and inputs it into an emotion recognition model to predict the emotion. Based on the predicted emotion, it provides audio advice or plays music.

[1538] Specific examples

[1539] If a driver attempts to change lanes and a vehicle behind is approaching at high speed, the system will provide a voice warning, "A vehicle behind is approaching. Changing lanes is dangerous." If the driver appears angry, the system will provide a voice warning, "Take a deep breath. We will play calming music," and will then play calming music.

[1540] Prompt Sentence Examples

[1541] An example of a prompt to input to a generative AI model is as follows:

[1542] When a driver attempts to change lanes, please design a system that judges the approaching speed and distance of vehicles behind and provides voice advice on the safe timing. Also, please add a function that recognizes the driver's emotions from their facial expressions and tone of voice, and calms them down by playing calming music or encouraging them to take deep breaths if they seem angry or anxious.

[1543] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1544] Step 1:

[1545] The server acquires video data from the camera. The input is real-time video from the camera, and the output is the acquired video data. Specifically, the camera captures the driver's face and sends the video data to the server.

[1546] Step 2:

[1547] The server converts the acquired video data to grayscale and performs face recognition. The input is the video data acquired in step 1, and the output is the recognized face area. Specifically, the server converts the video data to grayscale using OpenCV and applies a face detection algorithm.

[1548] Step 3:

[1549] The server extracts the recognized face area and performs preprocessing to input it into the emotion recognition model. The input is the face area recognized in step 2, and the output is preprocessed face image data. Specifically, the server resizes and normalizes the face area.

[1550] Step 4:

[1551] The server inputs the preprocessed facial image data into the emotion recognition model to predict the emotion. The input is the facial image data preprocessed in step 3, and the output is the predicted emotion label. Specifically, the emotion recognition model is executed using TensorFlow / Keras to obtain the emotion label.

[1552] Step 5:

[1553] The server determines the appropriate voice advice or music playback based on the predicted emotion label. The input is the emotion label predicted in step 4, and the output is the voice advice or music playback instruction. As a specific operation, it executes logic to select an action according to the emotion label.

[1554] Step 6:

[1555] The server uses a speech synthesis engine to generate voice advice and plays it through the speaker. The input is the voice advice instruction determined in step 5, and the output is the voice advice for the driver. Specifically, speech synthesis is performed using pyttsx3, and the voice is output from the speaker.

[1556] Step 7:

[1557] When the driver changes lanes or makes a right or left turn, the server determines the approaching speed and distance of the vehicle behind and gives voice advice on the safest timing. The input is data from cameras and sensors, and the output is voice advice on the safest timing. Specifically, the server calculates the speed and distance of the vehicle behind and executes an algorithm to evaluate safety.

[1558] Step 8:

[1559] If the driver feels angry or anxious, the server plays calming music or encourages deep breathing. The input is the emotion label predicted in step 4, and the output is instructions to play music or take deep breaths. The specific operation is to select and play music with a relaxing effect according to the emotion label.

[1560] Example 2

[1561] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1562] Conventional driving assistance systems lacked the ability to detect driver fatigue or loss of concentration, and often failed to suggest rest at the appropriate time. They also lacked the ability to provide entertainment content that responded to the driver's emotions, making it difficult to reduce stress or improve comfort while driving. Another problem was the lack of smooth interaction with generative AI when providing driving advice or setting a destination.

[1563] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1564] In this invention, the server includes a means for analyzing the driver's gaze, facial expression, and driving operation patterns to detect fatigue or loss of concentration, a means for recommending appropriate rest spots and suggesting a break by voice when fatigue or loss of concentration is detected, and a means for recognizing the driver's emotions and adjusting the entertainment content provided in accordance with the emotions. This makes it possible to appropriately detect the driver's fatigue or loss of concentration and suggest a break in a timely manner. Furthermore, providing entertainment content in accordance with the driver's emotions can reduce stress while driving and improve comfort.

[1565] "Means of analyzing the driver's line of sight, facial expression, and driving operation patterns to detect fatigue and loss of concentration" refers to technology that uses cameras and sensors to collect data on the driver's line of sight, changes in facial expression, and operation of the steering wheel and pedals, and analyzes this data to detect fatigue and loss of concentration in the driver.

[1566] "Means for recommending appropriate rest spots and suggesting a break by voice when detected" refers to a technology that uses a geographic information system (GIS) to search for the nearest rest spot when fatigue or a decline in concentration in the driver is detected, and then uses voice synthesis technology to suggest a break to the driver.

[1567] "Means for recognizing the driver's emotions and adjusting the entertainment content provided in accordance with those emotions" refers to technology that uses cameras and microphones to analyze the driver's facial expressions and tone of voice, identify the driver's emotions, and select and provide entertainment content such as music, radio, and audiobooks in accordance with those emotions.

[1568] A "driving assistance system that combines generative AI and voice recognition" is a system that combines a generative AI model with voice recognition technology to provide the driver with advice on driving operations and perform destination setting and other operations through dialogue while driving.

[1569] "Means for providing driving operation advice" refers to a technology that uses a generative AI model to analyze the driver's driving operations and provide appropriate driving methods and areas for improvement via voice or display.

[1570] "Means for setting a destination through verbal interaction with generative AI while driving" refers to a technology that uses voice recognition technology to understand the driver's verbal instructions and uses a generative AI model to set a destination and provide route guidance.

[1571] "Means for making recommendations tailored to the driver's hobbies and preferences" refers to technology that analyzes the driver's past behavior and preferences and, based on that, recommends appropriate entertainment content, rest spots, etc.

[1572] "Means for providing entertainment content such as music, radio, and audiobooks" refers to technology that selects and plays entertainment content such as music, radio programs, and audiobooks according to the driver's emotions and preferences.

[1573] MODE FOR CARRYING OUT THE INVENTION

[1574] This invention is a driving assistance system that detects driver fatigue and loss of concentration, suggests appropriate rest, and adjusts entertainment content according to the driver's emotions. This system also has the function of providing driving operation advice by combining a generative AI model and a voice recognition system.

[1575] Hardware and software used

[1576] Hardware: Cameras, sensors, microphones, speakers

[1577] Software: Image recognition software, emotion recognition engines, speech synthesis software, entertainment content management systems, geographic information systems (GIS)

[1578] Data processing and calculation

[1579] 1. Data Collection

[1580] The device collects data on the driver's gaze, facial expressions, driving operations, and voice through cameras, sensors, and microphones installed inside the vehicle.

[1581] How it works: The camera captures the driver's face in real time, sensors capture steering and pedal operation data, and a microphone records the driver's voice.

[1582] 2. Data Analysis

[1583] The server receives the collected data and analyzes it using image recognition software and data analysis software.

[1584] Specific operation: The server analyzes image data sent from the camera to detect eye movements and changes in facial expressions. It also analyzes data from sensors to analyze driving patterns.

[1585] 3. Detection of fatigue and loss of concentration

[1586] Based on the analysis results, the server runs an algorithm to detect driver fatigue and loss of concentration.

[1587] Specific operation: The server evaluates blink frequency, gaze fixation time, steering reaction time, etc. to detect signs of fatigue or loss of concentration.

[1588] 4. Alerts and break suggestions

[1589] If the server detects fatigue or a decrease in concentration, the AI ​​will recommend an appropriate rest spot and notify the device.

[1590] The device uses voice synthesis software to suggest breaks to the driver.

[1591] Specific operation: The server uses a geographic information system (GIS) to search for the nearest rest spot and sends the information to the device. The device then notifies the user by voice, saying, "There is a rest spot nearby. We recommend that you take a break."

[1592] 5. Emotion recognition

[1593] The server uses an emotion recognition engine to recognize the driver's emotions based on data acquired from the camera and microphone.

[1594] Specific operation: The server analyzes the driver's facial expressions and tone of voice to identify emotions such as joy, excitement, or depression.

[1595] 6. Entertainment Content Adjustment

[1596] The server selects appropriate content using an entertainment content management system according to the driver's emotions and transmits it to the terminal.

[1597] The device plays the selected content.

[1598] Specific operation: If the server determines that the driver is feeling happy or excited, it selects upbeat music and sends it to the device, which then plays that music. Conversely, if the server determines that the driver is feeling depressed, it selects calming music and plays it.

[1599] Specific examples

[1600] Examples of fatigue detection:

[1601] The device detects the driver's blinking frequency through the camera, and if the driver blinks more frequently than normal, the server detects fatigue.

[1602] The server will recommend an appropriate rest spot, and the device will notify the user by voice, "There is a rest spot nearby. We recommend that you take a break."

[1603] Examples of emotion recognition:

[1604] The device collects the driver's facial expressions and tone of voice through a camera and microphone, and the server determines that the driver is feeling happy if they are smiling and speaking.

[1605] The server selects up-tempo music and the device plays that music.

[1606] Prompt Sentence Examples

[1607] Fatigue detection prompt:

[1608] Please explain the system that analyzes the driver's gaze, facial expressions, and driving operation patterns to detect fatigue or loss of concentration. If fatigue or loss of concentration is detected, please also explain the function that uses AI to recommend appropriate rest spots and suggests taking a break by voice.

[1609] Emotion recognition prompt:

[1610] Describe a system that recognizes the driver's emotions and adjusts the entertainment content provided accordingly, including automatically selecting and playing upbeat music if the driver is feeling happy or excited.

[1611] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1612] Step 1:

[1613] Data collection

[1614] Input: Real-time data from cameras, sensors, and microphones

[1615] Specific operation: The device uses a camera installed inside the vehicle to capture the driver's face, uses sensors to obtain data on steering and pedal operation, and uses a microphone to record the driver's voice.

[1616] Output: Gaze data, facial expression data, driving operation data, voice data

[1617] Step 2:

[1618] Data analysis

[1619] Input: Gaze data, facial expression data, driving operation data, voice data

[1620] Specific operation: The server uses image recognition software to analyze image data sent from the camera to detect eye movements and changes in facial expressions. Data analysis software is used to analyze driving operation data from the sensors and analyze driving patterns.

[1621] Output: Gaze analysis results, facial expression analysis results, driving operation analysis results

[1622] Step 3:

[1623] Detecting fatigue and loss of concentration

[1624] Input: Gaze analysis results, facial expression analysis results, driving operation analysis results

[1625] How it works: Based on the analysis results, the server runs an algorithm to detect driver fatigue and loss of concentration. It evaluates factors such as blink frequency, gaze fixation time, and steering reaction time to detect signs of fatigue or loss of concentration.

[1626] Output: Fatigue and concentration loss detection results

[1627] Step 4:

[1628] Alerts and break suggestions

[1629] Input: Fatigue and concentration loss detection results

[1630] Specific operation: When the server detects fatigue or a lack of concentration, it uses a geographic information system (GIS) to search for the nearest rest spot and sends the information to the device. The device then uses voice synthesis software to notify the driver by voice, "There is a rest spot nearby. We recommend that you take a break."

[1631] Output: Audio notification of break suggestion

[1632] Step 5:

[1633] emotion recognition

[1634] Input: Gaze data, facial expression data, voice data

[1635] How it works: The server uses an emotion recognition engine to recognize the driver's emotions based on data acquired from the camera and microphone. It analyzes the driver's facial expressions and tone of voice to identify emotions such as joy, excitement, and depression.

[1636] Output: Emotion recognition result

[1637] Step 6:

[1638] Entertainment content adjustment

[1639] Input: Emotion recognition results

[1640] Specific operation: The server selects appropriate content using an entertainment content management system according to the driver's emotions and sends it to the device. The device then plays the selected content. For example, if the server determines that the driver is feeling happy or excited, it selects upbeat music and plays that music. Conversely, if the server determines that the driver is feeling depressed, it selects and plays calming music.

[1641] Output: Entertainment content playback

[1642] (Application example 2)

[1643] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1644] Conventional driving assistance systems lacked the ability to detect driver fatigue or loss of concentration, and often failed to suggest rest at appropriate times. Furthermore, they did not provide entertainment content tailored to the driver's emotions, meaning that stress reduction and relaxation effects while driving could not be expected. This resulted in issues such as insufficient safety and comfort for the driver.

[1645] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1646] In this invention, the server includes a means for providing driving operation advice, a means for detecting the driver's fatigue or loss of concentration and issuing an alert or suggesting a break, a means for recognizing the driver's emotions and adjusting entertainment content according to those emotions, a means for analyzing the driver's line of sight and facial expression in real time using a smartphone camera, and a means for recommending appropriate rest spots and suggesting a break by voice. This makes it possible to appropriately detect the driver's fatigue or loss of concentration and suggest a break in a timely manner. Furthermore, providing entertainment content according to the driver's emotions can reduce stress while driving and increase the relaxation effect.

[1647] "Means for providing driving operation advice" refers to a function that provides the driver with real-time instructions on appropriate driving methods and operating procedures.

[1648] "Means to detect driver fatigue or loss of concentration and issue alerts or suggest breaks" refers to a function that analyzes the driver's gaze, facial expressions, and driving patterns, and issues a warning and suggests appropriate breaks when fatigue or loss of concentration is detected.

[1649] "Means for recognizing the driver's emotions and adjusting entertainment content according to those emotions" refers to a function that reads the driver's emotions from their facial expressions and behavior, and selects and plays entertainment content such as music, radio, and audiobooks according to those emotions.

[1650] "Means for analyzing the driver's gaze and facial expressions in real time using a smartphone camera" refers to a function that uses a smartphone camera to capture a picture of the driver's face and analyzes the image in real time to read the driver's gaze and facial expressions.

[1651] "Means to recommend appropriate rest spots and suggest resting spots via voice" is a function in which AI selects the most suitable resting spot based on the driver's current location and surrounding geographic information, and suggests that location via voice.

[1652] The system for implementing this invention has the functions of detecting driver fatigue and loss of concentration and suggesting appropriate breaks, as well as adjusting entertainment content according to the driver's emotions. Specific embodiments of this system are described below.

[1653] Hardware and Software Configuration

[1654] This system uses hardware such as a smartphone, server, camera, and audio output device, and software such as OpenCV (image processing library), dlib (face detection and landmark detection library), and playsound (audio playback library).

[1655] Data processing and calculation

[1656] 1. Real-time analysis of driver gaze and facial expressions:

[1657] The system captures the driver's face using a smartphone camera and analyzes the footage in real time, detecting facial landmarks and reading gaze and facial expressions using OpenCV and dlib.

[1658] 2. Detecting fatigue and loss of concentration:

[1659] Based on the detected facial landmark data, the system analyzes the eye opening and closing status and eye movement, thereby detecting driver fatigue and loss of concentration.

[1660] 3. Break Suggestion:

[1661] If fatigue or a lack of concentration is detected, the server will recommend the best rest spot based on the driver's current location and surrounding geographic information, and will use a voice output device to suggest a break to the driver.

[1662] 4. Emotion Recognition and Entertainment Content Adjustment:

[1663] The system recognizes the driver's emotions from their facial expressions and selects and plays entertainment content (music, radio, audiobooks, etc.) according to their emotions. For example, if the driver is feeling happy, it will play upbeat music, and if they are feeling depressed, it will play calming music.

[1664] Specific examples

[1665] For example, if the driver has been driving for a long time, the system will detect fatigue by detecting changes in the state of their eyes. In that case, the system will suggest by voice, "Please stop at a nearby rest spot." Also, if the driver is smiling, the system will play up-tempo music to help the driver relax.

[1666] Prompt Sentence Examples

[1667] Develop a system that detects driver fatigue and loss of concentration and suggests appropriate rest spots. Also, add a function to recognize the driver's emotions and adjust entertainment content accordingly. The hardware used is a smartphone camera, and the software uses OpenCV and dlib.

[1668] In this way, a system can be realized that can improve the safety and comfort of the driver.

[1669] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1670] Step 1:

[1671] The smartphone camera is used to capture the driver's face. The input is the camera image, and the output is a real-time facial image. The terminal acquires this facial image and sends it to the next step.

[1672] Step 2:

[1673] Using OpenCV and dlib, facial landmarks are detected from the face image. The input is the face image acquired in step 1, and the output is facial landmark data. The device analyzes this landmark data to read gaze and facial expressions.

[1674] Step 3:

[1675] Based on facial landmark data, the device analyzes the eye opening / closing status and gaze movement. The input is the landmark data obtained in step 2, and the output is the detection results of the driver's fatigue and loss of concentration. Based on these detection results, the device evaluates the driver's condition.

[1676] Step 4:

[1677] If fatigue or a loss of concentration is detected, the server recommends the optimal rest spot based on the driver's current location and surrounding geographic information. The input is the driver's current location and the detection result, and the output is information about the recommended rest spot. The server sends this information to an audio output device.

[1678] Step 5:

[1679] A voice output device is used to suggest a break to the driver. The input is the information about the rest spots obtained in step 4, and the output is a voice suggestion to take a break. The terminal plays this voice to encourage the driver to take a break.

[1680] Step 6:

[1681] Recognizes emotions from the driver's facial expressions. The input is the facial landmark data obtained in step 2, and the output is the driver's emotion recognition result. The device proceeds to the next step based on this emotion recognition result.

[1682] Step 7:

[1683] Entertainment content is selected and played according to the driver's emotions. The input is the emotion recognition result obtained in step 6, and the output is the selected entertainment content. The device plays this content and provides entertainment according to the driver's emotions.

[1684] Example 3

[1685] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1686] Conventional driving assistance systems have difficulty effectively utilizing the driver's voice input to set destinations or provide entertainment content. They also lack the ability to recognize the driver's emotions and provide appropriate driving assistance. This has led to a demand for effective means to reduce driver fatigue and stress and promote safe driving.

[1687] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1688] In this invention, the server includes means for capturing the driver's voice input and converting the voice to text using a generative AI model, means for analyzing the text data and understanding the user's intent, and means for searching for a destination and setting navigation. This allows the driver to easily set a destination through voice input, and furthermore, by recognizing the driver's emotions and providing driving assistance accordingly, it becomes possible to reduce the driver's fatigue and stress and promote safe driving.

[1689] "Generative AI" is an artificial intelligence technology that analyzes input data such as voice and text and generates responses and suggestions in natural language.

[1690] A "speech recognition system" is a technology that converts voice input into text data and is used to understand what a user is saying.

[1691] A "driving assistance system" is a system that provides technologies and functions to support the driver in safe driving.

[1692] "Driving operation advice" is a function that suggests safe and efficient driving methods to the driver.

[1693] "Detection of fatigue and loss of concentration" is a technology that monitors the driver's physical and mental state to detect fatigue and loss of concentration.

[1694] "Alerts and rest suggestions" is a function that notifies and suggests drivers to take a break when they feel fatigued or their concentration is declining.

[1695] "Voice input capture" is the process of recording the driver's speech using a device such as a microphone and sending it to the system.

[1696] A "generative AI model" is an artificial intelligence algorithm that analyzes input data and generates appropriate responses or suggestions.

[1697] "Text data analysis" is the process of understanding text data converted by a speech recognition system and grasping the user's intent.

[1698] "Destination search" is a function that identifies the optimal destination based on the user's request and provides navigation information.

[1699] "Navigation setup" is the process of calculating the optimal route to a destination and providing guidance to the driver.

[1700] "Entertainment content" refers to content that can be enjoyed while driving, such as music, radio, and audiobooks.

[1701] "Emotion recognition" is a technology that analyzes the driver's facial expressions and tone of voice to understand their emotional state.

[1702] "Providing driving assistance" is a function that provides appropriate advice and suggestions based on the driver's emotions and condition.

[1703] This invention is a driving assistance system that combines generative AI and a voice recognition system. It captures the driver's voice input, converts the voice into text using a generative AI model, and understands the user's intent to set destinations, provide entertainment content, and assist driving.

[1704] Hardware and software used

[1705] Hardware: On-board computer, microphone, speaker, GPS module

[1706] Software: generative AI models (e.g., GPT-4), emotion recognition engines, navigation software, music playback software

[1707] System configuration

[1708] 1. Capture voice input

[1709] What the user says is captured by the device's microphone and transmitted to the vehicle's onboard computer.

[1710] 2. Speech-to-text

[1711] The server converts the voice data into text using a generative AI model.

[1712] The server analyzes the text data and understands the user's intent.

[1713] 3. Search for a destination and set up navigation

[1714] The server obtains the current location using the GPS module.

[1715] The server consults a map database to find the nearest destination.

[1716] The server runs the navigation software and calculates the route to the destination.

[1717] The server sends the calculated route information to the terminal.

[1718] The device will begin navigation through the speaker.

[1719] 4. Provision of entertainment content

[1720] A user says, "I want to listen to an audiobook."

[1721] The device's microphone captures the audio and transmits it to the vehicle's onboard computer.

[1722] The server uses a generative AI model to convert speech to text and analyze intent.

[1723] The server will resume the previously listened to audiobook or recommend a new audiobook.

[1724] Your device will play the audiobook through the speaker.

[1725] 5. Emotion Recognition and Driver Assistance

[1726] The device's emotion recognition engine analyzes the driver's facial expressions and tone of voice to recognize their emotions.

[1727] The device transmits the recognized emotion data to the server.

[1728] The server analyzes the emotional data and determines the driver's state.

[1729] The server determines the appropriate driving assistance and sends it to the device.

[1730] The device provides the driver with advice on rest and safe driving through a speaker.

[1731] Specific examples

[1732] Example prompt: "I want to go to the nearest gas station."

[1733] The user speaks this prompt.

[1734] The device's microphone captures the audio and transmits it to the vehicle's onboard computer.

[1735] The server uses a generative AI model to convert speech to text and search for the nearest gas station.

[1736] The server runs the navigation software and calculates the route.

[1737] The device will begin navigation through the speaker.

[1738] Example prompt: "I want to listen to an audiobook."

[1739] The user speaks this prompt.

[1740] The device's microphone captures the audio and transmits it to the vehicle's onboard computer.

[1741] The server uses a generative AI model to convert the speech to text and select an audiobook.

[1742] Your device will play the audiobook through the speaker.

[1743] Example prompt: "I'm tired."

[1744] The user speaks this prompt.

[1745] The device's emotion recognition engine recognizes the driver's fatigue.

[1746] The server recommends suitable rest spots, and the device suggests taking a break.

[1747] In this way, the present invention analyzes the driver's voice input and provides appropriate navigation, entertainment, and driving assistance, thereby improving the safety and comfort of the driver. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1748] Step 1:

[1749] Capturing voice input

[1750] The user says, "I want to go to the nearest gas station."

[1751] The device's microphone captures the audio and transmits it to the vehicle's onboard computer.

[1752] Input: User's voice

[1753] Output: Audio data

[1754] Step 2:

[1755] Speech-to-text

[1756] The server converts the voice data into text using a generative AI model.

[1757] The server analyzes the text data and understands the user's intent.

[1758] Input: Audio data

[1759] Output: Text data

[1760] What happens: The server uses a speech recognition algorithm to convert the speech data into text, then analyzes the text to extract the intent: "I want to go to the nearest gas station."

[1761] Step 3:

[1762] Search for destinations

[1763] The server obtains the current location using the GPS module.

[1764] The server consults a map database to find the nearest gas station.

[1765] Input: Text data, current location information

[1766] Output: Location of nearest gas station

[1767] What it does: The server gets the current location from the GPS module and queries a map database to identify the nearest gas station.

[1768] Step 4:

[1769] Navigation Settings

[1770] The server runs the navigation software and calculates the route to the destination.

[1771] The server sends the calculated route information to the terminal.

[1772] Input: Location of nearest gas station

[1773] Output: Route information

[1774] What it does: The server uses navigation software to calculate the optimal route from your current location to the nearest gas station and sends that route information to your device.

[1775] Step 5:

[1776] Start navigation

[1777] The device will begin navigation through the speaker.

[1778] The device provides real-time route guidance.

[1779] Input: Route information

[1780] Output: Voice guidance

[1781] Specific operation: The device provides real-time voice guidance through the speaker, such as "Turn right at the next intersection."

[1782] Step 6:

[1783] Providing entertainment content

[1784] A user says, "I want to listen to an audiobook."

[1785] The device's microphone captures the audio and transmits it to the vehicle's onboard computer.

[1786] Input: User's voice

[1787] Output: Audio data

[1788] Specific operation: The device captures what the user says and sends it to the vehicle's computer.

[1789] Step 7:

[1790] Speech-to-text conversion and intent analysis

[1791] The server uses a generative AI model to convert speech to text and analyze intent.

[1792] Input: Audio data

[1793] Output: Text data, intent information

[1794] Specific operation: The server converts the audio data into text and analyzes the intent, "I want to listen to an audiobook."

[1795] Step 8:

[1796] Content Selection

[1797] The server will resume the previously listened to audiobook or recommend a new audiobook.

[1798] Input: Intent information, user viewing history

[1799] Output: Audiobook data

[1800] Specific operation: The server refers to the user's listening history and selects an appropriate audiobook.

[1801] Step 9:

[1802] Playing content

[1803] Your device will play the audiobook through the speaker.

[1804] Input: Audiobook data

[1805] Output: Audio content

[1806] Specific behavior: The device will play the selected audiobook through the speaker.

[1807] Step 10:

[1808] Emotion Recognition and Driver Assistance

[1809] The device's emotion recognition engine analyzes the driver's facial expressions and tone of voice to recognize their emotions.

[1810] The device transmits the recognized emotion data to the server.

[1811] Input: Driver's facial expression data, voice tone

[1812] Output: Emotion data

[1813] Specific operation: The device uses an emotion recognition engine to analyze the driver's emotions and sends the data to the server.

[1814] Step 11:

[1815] Driving assistance decisions

[1816] The server analyzes the emotional data and determines the driver's state.

[1817] The server determines the appropriate driving assistance and sends it to the device.

[1818] Input: Emotion data

[1819] Output: Driver assistance information

[1820] Specific operation: The server analyzes the emotion data and recommends rest spots if the driver is tired, or determines safe driving advice if the driver feels anxious.

[1821] Step 12:

[1822] Providing driving assistance

[1823] The device provides the driver with advice on rest and safe driving through a speaker.

[1824] Input: Driver assistance information

[1825] Output: Audio advice

[1826] Specific operation: The device will provide voice advice through the speaker, such as "Take a break at the next service area."

[1827] (Application example 3)

[1828] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1829] Conventional driving assistance systems have the ability to detect driver fatigue and loss of concentration, and to provide driving operation advice, but they have the problem of not being able to recognize the driver's emotions in real time and provide driving assistance or entertainment that responds to those emotions. Furthermore, they lack the convenience of allowing the driver to specify a destination verbally, and they lack the ability to provide entertainment based on the driver's hobbies and preferences. This has led to problems in which driver safety and comfort are not adequately ensured.

[1830] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1831] In this invention, the server is a driving assistance system that combines generative AI and a voice recognition system, and includes: means for providing driving operation advice; means for detecting driver fatigue or loss of concentration and issuing an alert or suggesting a break; means for recognizing the driver's emotions and adjusting the content of driving assistance according to those emotions; means for searching for the optimal route and starting navigation when the driver verbally specifies a destination; means for recommending and playing music, radio, audiobooks, etc. based on the driver's hobbies and preferences; and means for providing advice on safe driving when the driver feels anxious or scared. This makes it possible to provide appropriate driving assistance and entertainment according to the driver's emotions.

[1832] "Generative AI" is an artificial intelligence system that understands the driver's voice commands and generates appropriate responses and actions.

[1833] A "voice recognition system" is a system that converts the driver's voice into text and understands its content.

[1834] "Driving operation advice" is a function that suggests safe and efficient driving methods to the driver.

[1835] "Detection of fatigue and loss of concentration" is a function that monitors the driver's physical and mental state and detects fatigue or loss of concentration.

[1836] "Alerts and rest suggestions" is a function that notifies or suggests drivers to take a break when they feel fatigued or their concentration is declining.

[1837] "Emotion recognition" is a technology that determines a driver's emotions in real time from facial expressions, tone of voice, etc.

[1838] "Adjusting the content of driving assistance" is a function that changes the content of driving assistance provided depending on the emotional state of the driver.

[1839] "Specifying a destination" refers to the act of the driver verbally telling the system where they want to go.

[1840] "Optimal route search" is a function that calculates the most efficient route to a specified destination.

[1841] "Start navigation" is a function that guides the driver to the destination based on the optimal route.

[1842] "Recommendation based on hobbies and preferences" is a function that learns the driver's past behavior and preferences and suggests entertainment content based on that.

[1843] "Entertainment playback" is a function that provides content such as music, radio, and audiobooks to the driver.

[1844] "Safe driving advice" is a function that provides advice to encourage safe driving when the driver is feeling anxious or scared.

[1845] As an embodiment of the present invention, the following system can be constructed.

[1846] First, the server provides a driving assistance system that combines generative AI and a voice recognition system. Specifically, the server uses a voice recognition system and generative AI to recognize the driver's voice commands and generate appropriate responses and actions. The voice recognition system uses the SpeechRecognition library, and the generative AI uses a natural language processing model.

[1847] The server then uses sensors to detect driver fatigue or loss of concentration. These include cameras and biometric sensors such as heart rate monitors. Data from these sensors is analyzed in real time to assess the driver's condition. If fatigue or loss of concentration is detected, the server will issue an alert and suggest appropriate rest spots.

[1848] Furthermore, the server uses an emotion recognition engine to recognize the driver's emotions in real time. This emotion recognition engine uses the EmotionRecognizer library. Depending on the driver's emotions, the server adjusts the content of the driving assistance. For example, if the driver is feeling anxious or scared, the server will provide advice on safe driving.

[1849] In addition, when the driver verbally specifies the destination, the server searches for the optimal route and starts navigation. This function uses the NavigationSystem module. When the driver says, "I want to go to the nearest gas station," the server searches for the nearest gas station and starts navigation.

[1850] Furthermore, the server recommends and plays music, radio, audiobooks, etc. based on the driver's preferences. This function uses the EntertainmentSystem module. When the driver says, "I want to listen to an audiobook," the server will resume the audiobook that was previously being listened to or recommend a new audiobook.

[1851] For example, if a driver says, "I want to go to the nearest gas station," the system will search for the nearest gas station and begin navigation. If a driver says, "I want to listen to an audiobook," the system will resume the audiobook they were previously listening to. Furthermore, if the system determines that the driver is feeling tired or stressed, it will recommend appropriate rest spots.

[1852] An example of a prompt is as follows:

[1853] "If the driver says, 'I want to go to the nearest gas station,' the system will search for the nearest gas station and begin navigation. If the driver says, 'I want to listen to an audiobook,' the system will resume the audiobook they were previously listening to. Furthermore, if the system determines that the driver is feeling tired or stressed, it will recommend appropriate rest spots."

[1854] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1855] Step 1:

[1856] The server uses a voice recognition system to obtain voice commands from the driver.

[1857] Input: Driver's voice

[1858] Data processing: Convert speech to text using a speech recognition system (SpeechRecognition library).

[1859] Output: Textual voice commands

[1860] Step 2:

[1861] The server uses generative AI to analyze voice commands and generate appropriate responses and actions.

[1862] Input: Text-based voice command

[1863] Data processing: Generative AI (natural language processing models) are used to analyze commands and generate responses and actions.

[1864] Output: Response text or action instructions

[1865] Step 3:

[1866] The server collects data from biometric sensors to detect driver fatigue or loss of concentration.

[1867] Input: Data from biometric sensors (camera, heart rate monitor)

[1868] Data processing: Analyzes sensor data in real time and evaluates the driver's condition.

[1869] Output: Driver fatigue and lack of concentration

[1870] Step 4:

[1871] The server uses an emotion recognition engine to recognize the driver's emotions.

[1872] Input: Driver's facial expression and tone of voice

[1873] Data processing: Emotions are determined in real time using an emotion recognition engine (EmotionRecognizer library).

[1874] Output: Driver's emotional state

[1875] Step 5:

[1876] The server adjusts the content of the driving assistance depending on the emotional state of the driver.

[1877] Input: Driver's emotional state

[1878] Data processing: Changing the content of driving assistance based on emotional state.

[1879] Output: Adjusted driving assistance content

[1880] Step 6:

[1881] When the driver verbally specifies the destination, the server searches for the optimal route and starts navigation.

[1882] Input: Specify the destination in text format

[1883] Data processing: Calculate the optimal route using the NavigationSystem module.

[1884] Output: Instruction to start navigation

[1885] Step 7:

[1886] The server recommends and plays music, radio, audiobooks, etc. based on the driver's tastes and preferences.

[1887] Input: Driver's hobbies and preferences data

[1888] Data Processing: Recommend appropriate content using the EntertainmentSystem module.

[1889] Output: Recommended entertainment content

[1890] Step 8:

[1891] The server provides advice on safe driving if the driver feels anxious or scared.

[1892] Input: Driver's emotional state

[1893] Data processing: Generate safe driving advice based on emotional state.

[1894] Output: Safe driving advice text

[1895] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1896] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1897] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1898] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1899] [Third embodiment]

[1900] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1901] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1902] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1903] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1904] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1905] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1906] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1907] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1908] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1909] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1910] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1911] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1912] "Example 1"

[1913] One embodiment of the present invention is a driving assistance system that combines generative AI and a voice recognition system. This system provides advice to the driver regarding their driving operations. For example, when a driver attempts to change lanes, the AI ​​determines the approaching speed and distance of the vehicle behind and provides voice advice on whether the lane change is safe.

[1914] "Example 2"

[1915] Furthermore, this system detects driver fatigue or loss of concentration and issues an alert or suggests taking a break. Specifically, it analyzes the driver's line of sight, facial expressions, driving patterns, etc. to detect fatigue or loss of concentration. If detected, the AI ​​will recommend an appropriate rest spot and suggest taking a break via voice. "Example 3"

[1916] This system also allows the driver to set a destination through verbal interaction with the generative AI while driving. If the driver says, "I want to go to the nearest gas station," the AI ​​will search for the nearest gas station and begin navigation. It also makes recommendations based on the driver's tastes and preferences. For example, it can learn the driver's favorite music genres and artists and automatically play that music while driving.

[1917] "Example 4"

[1918] In addition, the system provides entertainment content such as music, radio, and audiobooks. If the driver says, "I want to listen to an audiobook," the AI ​​will resume the audiobook the driver was previously listening to or recommend a new audiobook, allowing the driver to enjoy entertainment while driving.

[1919] The processing flow of each embodiment will be described below.

[1920] "Example 1"

[1921] Step 1: When a driver attempts to change lanes, the generative AI determines the approaching speed and distance of the vehicle behind.

[1922] Step 2: Based on the AI's judgment, the driver is given audio advice on whether a safe lane change is possible.

[1923] "Example 2"

[1924] Step 1: Analyze the driver's gaze, facial expressions, driving patterns, etc. to detect fatigue and loss of concentration.

[1925] Step 2: If fatigue or a lack of concentration is detected, the AI ​​will recommend an appropriate rest spot. Step 3: The AI ​​will suggest a rest spot to the driver via voice.

[1926] "Example 3"

[1927] Step 1: When the driver says, "I want to go to the nearest gas station," the AI ​​searches for the nearest gas station.

[1928] Step 2: Start navigation with the searched gas station as your destination.

[1929] Step 3: Make recommendations based on the driver's tastes and preferences. For example, learn the driver's favorite music genres and artists and automatically play that music while driving.

[1930] "Example 4"

[1931] Step 1: When the driver says, "I want to listen to an audiobook," the AI ​​will resume the audiobook the driver was previously listening to or recommend a new audiobook. Step 2: The recommended audiobook will be provided to the driver via audio, allowing the driver to enjoy entertainment while driving.

[1932] Example 1

[1933] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1934] Conventional driver assistance systems do not provide sufficient real-time advice to drivers on their driving operations. They also lack the ability to detect driver fatigue or loss of concentration and provide appropriate alerts or suggest rest breaks. Furthermore, there is a need for a system that effectively utilizes the driver's voice input to provide driving operation advice.

[1935] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1936] In this invention, the server includes means for providing driving operation advice, means for detecting driver fatigue or loss of concentration and issuing an alert or suggesting a break, means for capturing voice input from the driver and converting the voice data into text data, means for analyzing the text data and understanding the driver's intentions, means for acquiring environmental data from vehicle sensors, means for generating advice using generative artificial intelligence based on the acquired environmental data, and means for converting the generated advice into voice and providing it to the driver. This makes it possible to support the driver in safe driving operations in real time.

[1937] "Generative AI" is an AI technology that has the ability to generate new information and advice based on data.

[1938] A "voice recognition system" is a technology that converts voice data into text data.

[1939] A "driving assistance system" is a system that provides advice and support to drivers regarding driving operations.

[1940] The "means for providing driving operation advice" is a function for generating and providing appropriate advice for the driver's driving operation.

[1941] "Means for detecting driver fatigue or loss of concentration and issuing alerts or suggesting breaks" refers to a function that monitors the driver's condition and issues a warning or suggests a break when fatigue or loss of concentration is detected.

[1942] The "means for capturing the driver's voice input and converting the voice data into text data" is a technology for collecting the driver's voice and converting it into text format.

[1943] The "means for analyzing text data and understanding the driver's intention" is a function for analyzing the converted text data and understanding the driver's intention.

[1944] "Means for acquiring environmental data from vehicle sensors" refers to technology for collecting surrounding environmental data using sensors mounted on a vehicle.

[1945] "Means for generating advice using generative artificial intelligence" is a function that generates appropriate advice using generative artificial intelligence based on collected data.

[1946] The "means for converting the generated advice into audio and providing it to the driver" is a technology for converting the generated advice into audio format and communicating it to the driver.

[1947] This invention is a driving assistance system that combines generative artificial intelligence and a voice recognition system, and provides real-time driving advice to the driver. Specific embodiments of this system will be described below.

[1948] Hardware and software used

[1949] Hardware: Microphones, speakers, rear camera, and radar sensors installed in the vehicle

[1950] Software: Generative AI models (e.g., GPT-4), speech recognition systems (e.g., Google Cloud Speech-to-Text), speech synthesis systems (e.g., Google Cloud Text-to-Speech)

[1951] System configuration

[1952] 1. Capture voice input

[1953] The device uses a microphone installed inside the vehicle to capture the driver's voice in real time.

[1954] Specifically, the driver says, "I want to change lanes."

[1955] 2. Speech Recognition and Text Conversion

[1956] The device sends the captured audio data to the Google Cloud Speech-to-Text API.

[1957] The server receives the voice data and converts it into text data.

[1958] Specifically, the speech "I want to change lanes" is converted into text "I want to change lanes."

[1959] 3. Intent Analysis

[1960] The server analyzes the converted text data and understands the driver's intentions.

[1961] Specifically, from the text "I want to change lanes," the system recognizes that the driver wants to change lanes.

[1962] 4. Acquisition of environmental data

[1963] The server collects data in real time from sensors installed in the vehicle (rear camera and radar).

[1964] Specifically, data on the approaching speed and distance of the vehicle behind is acquired.

[1965] 5. Generating Advice

[1966] The server generates advice using a generative artificial intelligence model (e.g., GPT-4) based on the acquired environmental data.

[1967] Specifically, the advice generated is "A vehicle behind is approaching. Changing lanes is dangerous."

[1968] 6. Audio output of advice

[1969] The device converts the generated advice into audio using the Google Cloud Text-to-Speech API.

[1970] The terminal provides the converted audio to the driver through the car's speakers.

[1971] Specifically, the driver hears a voice saying, "A vehicle behind is approaching. Changing lanes is dangerous."

[1972] Specific examples

[1973] As a specific example, consider the case where a driver attempts to change lanes.

[1974] The user says, "I want to change lanes."

[1975] The device captures this audio with its microphone and sends it to the Google Cloud Speech-to-Text API.

[1976] The server receives the voice data and converts it into text saying "I want to change lanes."

[1977] The server parses this text and recognizes that the driver wants to change lanes.

[1978] The server acquires data on the approaching speed and distance of the vehicle behind from the vehicle's rear camera and radar.

[1979] The server uses a generative artificial intelligence model to generate advice such as, "A vehicle behind is approaching. Changing lanes is dangerous."

[1980] The device converts this advice into audio using the Google Cloud Text-to-Speech API and provides it to the driver through the car's speakers.

[1981] Prompt Sentence Examples

[1982] "When a driver attempts to change lanes, the system determines whether the lane change is safe based on the approaching speed and distance of the vehicle behind, and generates appropriate advice."

[1983] In this way, the driver assistance system can support the driver in safe driving operations in real time.

[1984] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1985] Step 1:

[1986] Capturing voice input

[1987] The device uses a microphone installed inside the vehicle to capture the driver's voice in real time.

[1988] Input: Driver's voice (e.g., "I want to change lanes")

[1989] Output: Captured audio data

[1990] Specific operation: When the driver says, "I want to change lanes," the microphone picks up the voice and sends it to the terminal as voice data.

[1991] Step 2:

[1992] Speech recognition and text conversion

[1993] The terminal transmits the captured voice data to a voice recognition system.

[1994] The server converts the voice data into text data using a voice recognition system.

[1995] Input: Captured audio data

[1996] Output: Converted text data (e.g., "I want to change lanes")

[1997] How it works: The device sends voice data to the Google Cloud Speech-to-Text API, and the server converts the voice data into text, such as "I want to change lanes."

[1998] Step 3:

[1999] Intention Analysis

[2000] The server analyzes the converted text data and understands the driver's intentions.

[2001] Input: Converted text data (e.g., "I want to change lanes")

[2002] Output: Driver's intention (e.g., desire to change lanes)

[2003] Specific operation: The server analyzes the text data and recognizes the intention to "change lanes."

[2004] Step 4:

[2005] Obtaining environmental data

[2006] The server collects data in real time from sensors installed in the vehicle (rear camera and radar).

[2007] Input: Environmental data from sensors (e.g. approaching speed and distance of rear vehicles)

[2008] Output: Acquired environmental data

[2009] Specific operation: The server collects data on the approach speed and distance of the vehicle behind from the rear camera and radar.

[2010] Step 5:

[2011] Generating Advice

[2012] The server generates advice using a generative artificial intelligence model based on the acquired environmental data.

[2013] Input: Driver's intention, environmental data

[2014] Output: Generated advice (e.g. "A vehicle is approaching from behind. Changing lanes is dangerous.")

[2015] Specific operation: The server uses a generative artificial intelligence model to consider the approaching speed and distance of the vehicle behind and generates advice such as, "A vehicle behind is approaching. Changing lanes is dangerous."

[2016] Step 6:

[2017] Audio output of advice

[2018] The terminal converts the generated advice into speech using a speech synthesis system.

[2019] The terminal provides the converted audio to the driver through the car's speakers.

[2020] Input: Generated advice (in text format)

[2021] Output: Voice advice (e.g. "A vehicle behind you is approaching. Changing lanes is dangerous.")

[2022] Specific operation: The device converts the generated advice into audio using the Google Cloud Text-to-Speech API and provides it to the driver through the car's speakers.

[2023] (Application example 1)

[2024] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2025] Conventional driver assistance systems offer limited advice to drivers on their driving operations, and lack specific support for ensuring safety, particularly when changing lanes or turning left or right. They also lack the ability to detect driver fatigue or loss of concentration and provide alerts or suggestions for breaks at appropriate times. Furthermore, they lack the ability to recognize the driver's voice commands and provide the necessary information, resulting in low convenience while driving.

[2026] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2027] In this invention, the server is a driving assistance system that combines a generative AI and a voice recognition system, and includes: means for providing driving operation advice; means for detecting driver fatigue or loss of concentration and issuing an alert or suggesting a break; means for acquiring data in real time from the vehicle's cameras and sensors and analyzing it with the generative AI; means for determining the approaching speed and distance of a vehicle behind the driver when the driver attempts to change lanes or turn right or left, and providing voice advice on whether safe operation is possible; and means for recognizing the driver's voice commands and providing necessary information. This enables the driver to perform driving operations safely and efficiently, improving convenience and safety while driving.

[2028] "Generative AI" is an artificial intelligence technology that generates new information and predictions based on data.

[2029] A "voice recognition system" is a technology that analyzes voice and converts it into text data.

[2030] A "driving assistance system" is a system that assists the driver in driving operations and improves safety and convenience.

[2031] "Driving operation advice" refers to providing appropriate instructions and advice to the driver regarding operations.

[2032] "Detecting driver fatigue and loss of concentration" refers to monitoring the driver's condition and identifying fatigue and loss of concentration.

[2033] "Alerts and rest suggestions" refers to notifications that alert the driver or encourage them to take a break.

[2034] "Acquiring data in real time from the vehicle's cameras and sensors" means collecting information on the current situation in real time using cameras and sensors installed in the vehicle.

[2035] "Analysis by generative AI" means that generative AI processes the collected data and provides the necessary information and judgments.

[2036] "Judgment of approaching speed and distance of rear vehicles" means measuring the speed and distance of vehicles approaching from behind and evaluating safety.

[2037] "Voice advice" means providing voice instructions and advice to the driver based on the analysis results.

[2038] "Recognizing driver's voice commands" means analyzing the voice instructions given by the driver and understanding their content.

[2039] "Providing necessary information" means providing the driver with the information they require in an appropriate manner.

[2040] The system for carrying out this invention is a driving assistance system that combines generative AI and a voice recognition system. Specific embodiments of this system will be described below.

[2041] Hardware and software used

[2042] Hardware:

[2043] Camera mounted on the vehicle

[2044] Sensors installed in the vehicle

[2045] Microphone for voice recognition

[2046] Smartphone (for running applications)

[2047] software:

[2048] OpenCV (camera image acquisition)

[2049] TensorFlow (Analysis of generative AI models)

[2050] SpeechRecognition

[2051] gTTS (speech synthesis)

[2052] System processing overview

[2053] 1. Data Acquisition:

[2054] The system obtains real-time data from cameras and sensors installed in the vehicle. The cameras capture images of the vehicle behind, and the sensors collect information such as the vehicle's speed and distance.

[2055] 2. Data Analysis:

[2056] The acquired data is fed into a generative AI model to analyze the approaching speed and distance of the vehicle behind. Using TensorFlow, this data is processed in real time to determine whether a safe maneuver is possible.

[2057] 3. Audio advice:

[2058] Based on the analysis results, the system provides audio advice to the driver. For example, if it is safe to change lanes, it will notify the driver by saying "It is safe to change lanes," and if it is dangerous, it will warn the driver by saying "It is dangerous to change lanes." The audio is generated using gTTS and output from the speaker.

[2059] 4. Voice command recognition:

[2060] The system recognizes the driver's voice commands and provides the necessary information. It uses SpeechRecognition to analyze the voice instructions given by the driver and understand their content. For example, if the driver instructs the system to "turn right at the next intersection," the system will respond with "turn right at the next intersection."

[2061] Specific examples

[2062] Lane change advice:

[2063] When a driver attempts to change lanes, the system analyzes the approaching speed and distance of the vehicle behind and provides voice advice such as "Changing lanes is safe" or "Changing lanes is dangerous."

[2064] Voice command recognition and response:

[2065] When the driver issues a voice command such as "Turn right at the next intersection," the system responds with "Turn right at the next intersection."

[2066] Prompt Sentence Examples

[2067] "Analyze the approaching speed and distance of the vehicle behind you to determine whether a safe lane change is possible."

[2068] "Recognize the driver's voice commands and provide the necessary information."

[2069] In this way, the driver can perform driving operations safely and efficiently, improving convenience and safety while driving.

[2070] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2071] Step 1:

[2072] The server receives real-time data from the cameras and sensors installed in the vehicle. The cameras capture images of the vehicle behind, and the sensors collect information such as the vehicle's speed and distance. The input is the camera images and sensor data, and the output is a set of these data.

[2073] Step 2:

[2074] The server inputs the acquired data into a generative AI model to analyze the approaching speed and distance of the vehicle behind. Specifically, it uses TensorFlow to analyze camera footage and calculate the position and speed of the vehicle behind. The input is camera footage and sensor data, and the output is information on the approaching speed and distance of the vehicle behind.

[2075] Step 3:

[2076] The server provides audio advice to the driver based on the analysis results. For example, if it is safe to change lanes, it will notify the driver by saying "It is safe to change lanes," and if it is dangerous, it will warn the driver by saying "It is dangerous to change lanes." Voice is generated using gTTS and output from the speaker. The input is information on the approaching speed and distance of the vehicle behind, and the output is audio advice.

[2077] Step 4:

[2078] The server recognizes the driver's voice commands and provides the necessary information. Using SpeechRecognition, the system analyzes the voice instructions given by the driver and understands their content. For example, if the driver instructs the system to "turn right at the next intersection," the system responds with "turn right at the next intersection." The input is the driver's voice command, and the output is a voice response.

[2079] Step 5:

[2080] The server detects driver fatigue or loss of concentration and issues an alert or suggests taking a break. It analyzes data from sensors and monitors the driver's condition. If fatigue or loss of concentration is detected, it notifies the driver by voice, saying "Please take a break." The input is sensor data, and the output is an alert or voice notification suggesting a break.

[2081] Step 6:

[2082] The server makes recommendations tailored to the driver's hobbies and preferences. It analyzes the driver's past behavioral data and voice commands to suggest appropriate entertainment content and destinations. The input is the driver's past behavioral data and voice commands, and the output is recommendation information.

[2083] Step 7:

[2084] The server provides entertainment content such as music, radio, audiobooks, etc. Based on the driver's voice command, it selects and plays the appropriate content. The input is the driver's voice command, and the output is the playback of the entertainment content.

[2085] Example 2

[2086] Next, a description will be given of Example 2 of Form Example 2. In the following description, ...

Claims

1. A driving assistance system that combines generative AI and a voice recognition system, and provides driving operation advice; A means of detecting driver fatigue or loss of concentration and issuing alerts or suggesting rest breaks; A means for recognizing an emotion of the driver from a facial expression or a tone of voice of the driver, and adjusting the content of the driving assistance in accordance with the recognized emotion; A means for judging an approaching speed and distance of a vehicle behind the driver when the driver is about to change lanes or turn right or left, and giving a voice advice on a safe timing; A means for playing calming music or encouraging deep breathing when the driver feels angry or anxious; A system including:

2. The system according to claim 1, further comprising: a means for setting a destination through verbal interaction between the driver and the generative AI while driving; and a means for making recommendations according to the driver's hobbies and preferences.

3. 3. The system of claim 1 or 2, further comprising means for providing entertainment content.

Images