system

Abstract

Provide a system. [Solution] means for registering and storing driver identification information; means for collecting data in real time from sensors in the vehicle to monitor the vehicle's condition and driving status; A means of analyzing the collected data and providing appropriate information and advice to drivers; A means for interacting with the driver and providing chat and entertainment based on the driver's hobbies and preferences; A way to assign individual topics to each family member, A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional in-vehicle systems are limited to simple functions such as navigation and music playback for the driver, and are unable to sufficiently improve driver safety and comfort. Furthermore, they lack the means to comprehensively analyze real-time data on the driver's condition and the vehicle and provide appropriate feedback, resulting in a lack of interactive communication with the driver and reducing the risk of accidents while driving. Furthermore, when a family is traveling together, it is difficult to bring up appropriate topics of conversation and create a lively atmosphere in the car. These issues need to be resolved. [Means for solving the problem]

[0005] The present invention includes a means for registering and saving the driver's identification information, a means for collecting data in real time from vehicle sensors and monitoring the vehicle's status and driving situation, and a means for analyzing the collected data and providing the driver with appropriate information and advice. It also includes a means for providing casual conversation and entertainment based on the driver's hobbies and preferences through dialogue with the driver, and a means for allocating individual topics to each family member. This improves driver safety, enables comfortable communication while driving, and provides an in-vehicle environment that the whole family can enjoy.

[0006] ---

[0007] "Driver identification information" refers to information such as the name, age, past driving history, hobbies, and preferences that identify the driver.

[0008] "Registration" is the process by which a driver's identification information is entered and stored in the system.

[0009] "Vehicle sensors" are sensors for measuring vehicle speed, engine condition, fuel level, interior temperature, and the like.

[0010] "Collecting data in real time" means constantly collecting data about the current driving situation and vehicle condition.

[0011] "Monitoring" refers to the process of watching over collected data and detecting abnormalities or changes.

[0012] "Analysis" is the process of analyzing collected data and generating useful information and advice based on that data.

[0013] "Providing appropriate information and advice" means presenting information and advice to the driver based on the analyzed data to assist them in driving safely and comfortably.

[0014] "Dialogue" means that the driver and the system communicate through voice.

[0015] "Hobbies and preferences" are personal preferences such as the driver's favorite genre of music or topics of interest.

[0016] "Providing conversation and entertainment" means engaging in conversation on topics that match the driver's hobbies and preferences, and providing appropriate music and information.

[0017] "Allocating individual topics to each family member" means providing topics that are appropriate for each member in the car, depending on their interests, age, and situation. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10]1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] ---

[0040] The present invention relates to a system that collects data specific to a driver by registering and storing the driver's identification information, and provides safe driving and a comfortable driving environment based on that data.

[0041] System Configuration

[0042] The system of the present invention includes a vehicle terminal used by the user, a cloud server, and an interface for interaction with the driver. The terminal is installed in the vehicle and is responsible for the actual data collection and monitoring. The server analyzes and stores the data for long-term storage. The users are the driver and passengers, and each receives appropriate dialogue and feedback.

[0043] Program and processing flow

[0044] When a user first connects to the system, the server collects basic information about the user, including their name, age, hobbies, preferences, and driving history. Once the user begins driving, the device collects real-time data from the vehicle's sensors and sends it to the server. The server uses this data to analyze the vehicle's condition and driving situation, and provides appropriate information and advice under specific conditions.

[0045] As a specific example, if an engine abnormality is detected, the device immediately sends the abnormality data to the server. The server analyzes the received data and diagnoses the nature of the abnormality. The driver is then notified via the device with an audio alert saying, "An engine abnormality has occurred. Please stop the vehicle in a safe place and request repairs." By responding immediately in this way, serious vehicle breakdowns can be prevented.

[0046] In addition, if the server detects fatigue in a user who has been driving for a long time, it will provide advice such as, "Long-term driving can cause fatigue, so we recommend that you take a break." This can reduce the risk of accidents caused by fatigued driving.

[0047] In addition, sensing devices inside the vehicle can be used to conduct voice interactions with the user and passengers. For example, if a user asks, "Where is the next rest stop?", the device sends this information to a server, which returns data about the nearest rest stop. The server then generates an appropriate response and relays it to the driver via the device.

[0048] When the whole family is in the car, the device can use the microphone and speaker at each seat to enable individual conversations. For example, it can prompt a child in the back seat with a question like, "What book have you read recently?", while asking a parent in the passenger seat, "Are you thinking about planning a family trip?" In this way, everyone can have appropriate topics to talk about, creating a lively atmosphere in the car.

[0049] ---

[0050] The above is an embodiment of the present invention. However, the present invention is not limited to this embodiment and may be modified as appropriate. For example, the type of data collected, the analysis method, and the content of the advice provided can be changed according to the system specifications and the needs of the user.

[0051] The processing flow will be explained below.

[0052] ---

[0053] Step 1:

[0054] The user enters the vehicle and logs in to the in-vehicle terminal. The terminal detects this and sends the user's login information to the server. The server uses this information to authenticate the user's profile and returns the necessary data to the terminal.

[0055] Step 2:

[0056] The device collects real-time data from the vehicle's sensors, including vehicle speed, engine status, fuel level, and interior temperature, and periodically transmits this data to a server.

[0057] Step 3:

[0058] The server receives and analyzes real-time data sent from the device, including engine anomaly detection, fuel level checks, and driver fatigue assessment.

[0059] Step 4:

[0060] Based on the analysis results from the server, the device will provide the driver with appropriate information and advice. For example, if an engine abnormality is detected, the device will issue a voice alert saying, "An engine abnormality has occurred. Please stop in a safe place and request repairs."

[0061] Step 5:

[0062] While driving, the device uses a microphone to receive voice commands from the driver. For example, if the driver asks, "Where is the next rest stop?", the device sends the command to the server. The server analyzes and returns information about the nearest rest stop.

[0063] Step 6:

[0064] Based on the rest stop information returned from the server, the device notifies the driver, "The next rest stop is 5km away." In this way, answers to the driver's questions are provided in real time.

[0065] Step 7:

[0066] The server monitors the driver's fatigue level and generates advice to encourage rest if the driver has been driving for a long time. The device then communicates this to the driver. For example, it may say, "Long-term driving can cause fatigue, so we recommend that you take a break."

[0067] Step 8:

[0068] When the whole family is in the car, the device uses a microphone and speaker installed in each seat to individually target each person with a topic, such as asking a child in the back seat, "What book have you read recently?", or speaking to a parent in the front seat, "Are you thinking about your next family trip?"

[0069] ---

[0070] The above are the processing steps for implementing the present invention. By building a system in accordance with these steps, it is possible to significantly improve the safety and comfort of the driver and passengers.

[0071] Example 1

[0072] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0073] In recent years, there has been a demand for improved safety and comfort in vehicle driving. However, conventional systems are unable to fully utilize individual driver information, making it difficult to provide appropriate advice and warnings in real time. They are also inadequate at immediately detecting and responding to driver fatigue or vehicle abnormalities. Furthermore, their ability to provide information and entertainment tailored to individual needs through two-way dialogue with the driver is limited. There is a need to resolve these issues and improve driver safety and comfort.

[0074] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0075] In this invention, the server includes a means for registering and storing the driver's identification information, a means for collecting data in real time from the vehicle's sensors and monitoring the vehicle's condition and driving situation, and a means for analyzing the collected data and providing the driver with appropriate information and advice. This makes it possible to provide appropriate advice based on the driver's individual information in real time. It also makes it possible to reduce the risk of accidents by detecting the driver's fatigue level and encouraging them to take a break, and to avoid serious breakdowns by immediately notifying the driver of vehicle abnormalities. Furthermore, it is possible to improve the comfort of the driving environment by interacting with the driver and providing information and entertainment based on the driver's hobbies and preferences.

[0076] 1. "Driver identification information" refers to data such as name, age, hobbies and preferences, and driving history used to identify and manage individual drivers.

[0077] 2. "Means for registering and storing" means a technology or device for inputting driver identification information and storing that data in a storage device.

[0078] 3. "Sensor" means a device used to detect and collect data about the vehicle's condition and driving situation. Examples include engine sensors, speed sensors, and cameras.

[0079] 4. "Means for collecting and monitoring data in real time" refers to technology that continuously and instantly collects data obtained from sensors and monitors the vehicle's condition and driving situation.

[0080] 5. "Means for analyzing and providing data" refers to the technology that processes collected data using an analysis program and, as a result, provides useful information and advice to the driver.

[0081] 6. "Means of dialogue" means technology that enables two-way communication with the driver via voice and text.

[0082] 7. "Means for providing conversation and entertainment based on the driver's interests and preferences" refers to technology that provides dialogue and entertainment based on the driver's individual information, according to the driver's interests and preferences.

[0083] 8. "Means for individual topic allocation" refers to technology that provides different conversation content to different passengers in the vehicle, enabling communication that is tailored to their individual interests and situations.

[0084] 9. "Means for issuing an alert when an engine abnormality is detected" refers to technology and devices that monitor the engine condition and issue a warning to the driver when an abnormality is detected.

[0085] 10. "Means for detecting fatigue and providing advice to encourage rest" refers to technology that determines the driver's level of fatigue from continuous driving data and provides information to encourage rest at an appropriate time.

[0086] 11. "Means for analyzing questions based on voice input and providing answers" refers to technology that converts questions entered by the driver through voice into text, analyzes the content of that text, and generates and provides the most appropriate answer.

[0087] The present invention relates to a system that collects data specific to a driver by registering and storing the driver's identification information, and provides safe driving and a comfortable driving environment based on that data.

[0088] System Configuration

[0089] The system of the present invention includes a terminal in the vehicle used by the user, a cloud server, and an interface for interaction with the driver. The terminal is installed in the vehicle and is responsible for the actual data collection and monitoring, while the server analyzes and stores the data for long-term storage.

[0090] Program and process description

[0091] When a user first connects to the system, the server collects basic information about the user, including their name, age, hobbies, preferences, and driving history. Once the user begins driving, the device collects real-time data from the vehicle's sensors and sends it to the server. The server uses this data to analyze the vehicle's condition and driving situation, and provides appropriate information and advice under specific conditions.

[0092] Specifically, the terminal uses the following hardware and software:

[0093] In-vehicle devices: for data collection and monitoring

[0094] Cloud server: for data analysis and long-term storage

[0095] Sensors (e.g. engine sensors, speed sensors, cameras): for real-time data collection and interaction

[0096] The server processes and calculates the collected data as follows:

[0097] 1. Receives data sent from each vehicle sensor and monitors it in real time.

[0098] 2. Use a data analysis program (e.g., Python script) to detect engine anomalies and fatigued driving.

[0099] 3. After detecting an anomaly, appropriate feedback and advice is generated and notified to the user in real time.

[0100] Specific examples

[0101] Examples of anomaly detection:

[0102] Situation: The server detects an abnormality in the engine sensor.

[0103] Processing: The abnormal data is analyzed, and an audio alert is sent to the user via the device stating, "An abnormality has occurred in the engine. Please stop the vehicle in a safe place and request repairs."

[0104] Prompt sentence to input to the generative AI model:

[0105] "An engine anomaly has been detected. Please generate a script to diagnose the anomaly and provide an audio alert to the user."

[0106] Examples of fatigue detection:

[0107] Situation: The server analyzes long-term driving data and estimates fatigue.

[0108] Action: Provide the user with the advice, "Driving for long periods of time can cause fatigue, so we recommend you take a break."

[0109] Prompt sentence to input to the generative AI model:

[0110] "If fatigue due to long driving is detected, generate a script that provides advice to encourage the user to take an appropriate break."

[0111] Examples of voice interactions:

[0112] Situation: A user asks the device, "Where is the next rest stop?"

[0113] Processing: The device sends this question to the server, which responds with information about the nearest rest area. The device then provides this information to the user.

[0114] Prompt sentence to input to the generative AI model:

[0115] "When a user asks, 'Where is the next rest stop?', generate a script that looks up information about the nearest rest stop and responds."

[0116] Examples of interactions with the whole family:

[0117] Situation: There are multiple passengers in the vehicle.

[0118] Processing: The device uses data from each seat's microphone and speaker to facilitate personalized conversations, asking the child in the back seat, "What book have you read recently?" and the parent in the front seat, "Are you thinking about planning a family trip?"

[0119] Prompt sentence to input to the generative AI model:

[0120] "Generate scripts that provide appropriate conversation topics for each passenger in the car. For example, ask the child in the back seat, 'What book have you read recently?' and the parent in the front seat, 'Are you thinking about planning a family trip?'"

[0121] The above is an embodiment of the present invention. This system allows the driver to receive advice and warnings in real time, improving safety and comfort.

[0122] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0123] Step 1:

[0124] Registering and saving user information

[0125] Input: The user enters basic information such as name, age, hobbies, preferences, and driving history into the vehicle's terminal.

[0126] Specific operation: The user enters each piece of information on the device screen.

[0127] Processing: The device sends the entered information to a server in the cloud. After receiving the information, the server stores it in a database and manages it for each user.

[0128] Output: Driver identification stored in a database.

[0129] Step 2:

[0130] Real-time data collection

[0131] Input: Data from each vehicle sensor (engine sensor, speed sensor, camera, etc.).

[0132] Specific operation: The device collects data in real time from sensors installed in the vehicle.

[0133] Processing: The device sends the collected data to a server in the cloud at regular intervals.

[0134] Output: Real-time sensor data sent to the server.

[0135] Step 3:

[0136] Data analysis

[0137] Input: Sensor data sent to the server.

[0138] Specific operation: The server analyzes the data using Python data analysis libraries (such as Pandas and NumPy).

[0139] Processing: Analyzes engine temperature, speed data, etc. to detect abnormalities and estimate driver fatigue.

[0140] Output: Analysis results include the presence or absence of abnormalities and the driver's fatigue state.

[0141] Step 4:

[0142] Generate alerts and advice

[0143] Input: Data analysis results (anomaly detection results, fatigue estimation results).

[0144] Specific behavior: The server generates appropriate alerts and advice based on the analysis results.

[0145] In case of engine abnormality: Generates a message saying "An engine abnormality has occurred. Please stop the vehicle in a safe place and request repairs."

[0146] For fatigue detection: Generate advice such as "Driving for long periods of time can cause fatigue, so we recommend that you take a break."

[0147] Processing: Sends generated alerts and advice to the device.

[0148] Output: Message data for generated alerts and advice.

[0149] Step 5:

[0150] User Notification

[0151] Input: The alert or advice message data sent by the server.

[0152] Specific behavior: The device notifies the user through voice and display.

[0153] In case of engine abnormality: An audio alert will be issued saying, "An engine abnormality has occurred. Please stop in a safe place and request repairs."

[0154] If fatigue is detected: A voice will advise, "Driving for long periods of time can cause fatigue, so we recommend that you take a break."

[0155] Action: Provide real-time alerts and advice to users.

[0156] Output: Proper alerts and advice to the driver.

[0157] Step 6:

[0158] Voice interaction processing

[0159] Input: The user speaks a question to the device, for example, "Where is the next rest stop?"

[0160] Specific operation: The device collects voice data and sends it to a server in the cloud.

[0161] Processing: The server converts the speech to text, analyzes the question, and generates the best answer.

[0162] Example: Generate an answer by getting information about the nearest rest stop.

[0163] Output: Sends the generated answer text data to the terminal.

[0164] Specific operation: The device converts the transmitted response data into voice and provides it to the user.

[0165] Output: The appropriate answer information provided to the user.

[0166] (Application example 1)

[0167] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0168] Autonomous vehicles require efficient registration of driver identification information and the provision of personalized driving experiences. It is also necessary to improve the safety and comfort of drivers and passengers by improving the accuracy of advice provided based on real-time monitoring of driving conditions and data analysis. While linking with various devices and utilizing generative AI models are effective ways to achieve these goals, many aspects of specific implementation remain unresolved.

[0169] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0170] In this invention, the server includes means for registering and storing the driver's identification information, means for collecting data in real time from vehicle sensors and monitoring the vehicle's status and driving situation, means for analyzing the collected data and providing the driver with appropriate information and advice, means for interacting with the driver and providing chat and entertainment based on the driver's hobbies and preferences, means for allocating individual topics to each family member, means for transmitting basic information about the user to a cloud server in the autonomous vehicle and providing analysis results and advice to a smart device in real time, and means for generating appropriate responses to the user's voice commands using a generative AI model. This enables the provision of a personalized driving experience based on the user's identification information, the provision of appropriate advice according to the situation of the autonomous vehicle, and advanced voice responses using the generative AI model.

[0171] "Driver identification information" is a series of information for identifying an individual driver, such as the user's name, age, driving history, etc.

[0172] "Real-time data collection" means using sensors and other input devices to instantly obtain data about the ongoing driving situation and vehicle status.

[0173] "Monitoring vehicle condition and driving status" means monitoring engine status, speed, location information, etc. through various sensors in the vehicle to determine whether it is operating normally.

[0174] "Analyzing collected data" means using algorithms and analytical tools based on the acquired data to detect vehicle abnormalities and the driver's condition.

[0175] "Providing appropriate information and advice" means providing drivers with advice for safe driving and comfortable driving based on the analysis results.

[0176] "Dialogue with the driver" means two-way communication with the driver using means such as voice recognition or text input.

[0177] "Providing chat and entertainment based on the driver's hobbies and preferences" means providing topics, music, and other entertainment content that meet the driver's individual preferences.

[0178] "Allocating individual topics to each family member" means bringing up topics that correspond to the interests and concerns of each passenger in the car and conducting individual conversations.

[0179] "Send to a server on the cloud" means uploading data to a remote server via the Internet.

[0180] A "generative AI model" is an algorithm based on machine learning and deep learning that is used to perform advanced tasks such as speech recognition and text generation.

[0181] "Generating an appropriate response to a voice command" means recognizing a voice instruction given by the user and providing an appropriate reaction or information in response to it.

[0182] "Smart devices" is a general term for mobile devices that can connect to the Internet, such as smartphones, tablets, and smart glasses.

[0183] "Personalized driving experience" refers to providing a driving environment and services that are customized based on a user's individual identity and preferences.

[0184] This invention is a system for enhancing the driver-specific experience in autonomous vehicles. The system registers the driver's identification information, monitors the vehicle's status and driving situation in real time, analyzes the collected data, and provides appropriate information and advice. Furthermore, it can use a generative AI model to interact with the driver and provide personalized conversations and entertainment.

[0185] The server first registers the driver's identification information in a database on the cloud. This information includes the user's name, age, driving history, etc. In addition, a terminal installed in the vehicle uses various sensors to collect vehicle status (engine status, speed, location information, etc.) in real time. This data is immediately sent to the server, which then analyzes it.

[0186] Based on the analysis results, the server generates advice and alerts appropriate to the situation and provides them to the driver via a smart device (such as a smartphone or smart glasses). For example, if an engine abnormality is detected, the server will issue an alert stating, "An engine abnormality has occurred. Please stop the vehicle in a safe place and request repairs."

[0187] Additionally, if the server detects that the driver is fatigued, it will provide advice such as, "Long-term driving can cause fatigue, so we recommend you take a break," ensuring the driver's safety. Furthermore, when the driver issues a voice command using the generative AI model, the server generates an appropriate response based on the analysis results and provides it to the driver.

[0188] For example, if a user asks, "Where is the next rest stop?", the smart device sends this information to the server, which uses data about the nearest rest stops to generate a response such as "There is a rest stop XX kilometers away."

[0189] Here is an example prompt:

[0190] "The user wants to know the nearest rest area from their current location. Their current location is [latitude, longitude]. What is the next available rest area?"

[0191] The hardware used includes smartphones, smart glasses, and autonomous vehicle sensors (cameras, GPS, microphones, etc.), while the software used includes cloud servers (AWS (registered trademark), Google (registered trademark) Cloud), machine learning models (TENSORFLOW (registered trademark), PyTorch), and real-time data analysis (Apache (registered trademark) Kafka).

[0192] The system will provide a personalized driving experience based on user identity, monitor vehicle status in real time and provide appropriate advice, and use generative AI models to provide advanced voice responses.

[0193] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0194] Step 1:

[0195] When a user gets into a vehicle, the device acquires the user's identification information and sends it to the cloud server. Basic information such as the user's name, age, and driving history is received as input, and based on this information, a new registration or update is made on the cloud. A user ID is generated as output.

[0196] Step 2:

[0197] The terminal collects data in real time from various sensors in the vehicle (engine status, speed, location information, etc.) and sends the collected data to a cloud server. The input includes signal data from each sensor. The server receives this and stores it in a database. The output is accumulated data.

[0198] Step 3:

[0199] The server analyzes the collected data and diagnoses the driver's condition. The stored sensor data is used as input. A machine learning model (e.g., TensorFlow, PyTorch) is used for data analysis to detect signs and abnormalities for safe driving. The analysis results are generated as output.

[0200] Step 4:

[0201] Based on the analysis results, the server automatically generates appropriate advice or alerts and notifies the driver via the terminal. The analysis results are used as input. For example, a message such as "There is an engine malfunction. Please stop the vehicle in a safe place and request repairs" is generated. As output, a voice or text notification is sent to the driver.

[0202] Step 5:

[0203] The server detects the driver's fatigue state from sensor data and provides advice such as, "You are showing signs of fatigue. We recommend that you take a break." Input includes vital data such as heart rate and reaction time. A machine learning model is used to analyze this data and detect signs of fatigue. The output is a notification encouraging the driver to take a break.

[0204] Step 6:

[0205] When a driver inputs a voice command via a smart device (smartphone or smart glasses), the device sends the information to the server. The user's voice command is used as input. Example prompt: "Where is the next rest stop?" The server uses a generative AI model to generate an appropriate response and returns it to the user. Information about rest stops is returned as output.

[0206] Step 7:

[0207] As part of the response generation process, the server performs database searches and calculations in the cloud. The input includes a prompt to the generative AI model. For example, the prompt might read, "The user wants to know the nearest rest area from their current location. Their current location is [latitude, longitude]. Please tell me where the next available rest area is." The server retrieves data on the nearest rest areas and generates an appropriate answer. The output is a response containing the location of the rest area.

[0208] These processing steps enable the system of the present invention to effectively collect and analyze driver-specific data to provide a safe and comfortable driving experience.

[0209] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0210] ---

[0211] This invention achieves even greater driving safety and comfort by collecting driver-specific data by registering and storing the driver's identification information, and combining this data with an emotion engine in a system that provides safe driving and a comfortable driving environment.

[0212] System Configuration

[0213] The system of the present invention includes a vehicle terminal used by the user, a cloud server, an interface for interacting with the driver, and an emotion engine. The terminal is installed in the vehicle and is responsible for the actual data collection and monitoring. The server analyzes and stores the data for long-term storage. The emotion engine has the function of recognizing the user's emotions and generating appropriate dialogue and responses based on those emotions. The users are the driver and passengers, and each receives dialogue and feedback tailored to their needs.

[0214] Program and processing flow

[0215] When a user first connects to the system, the server collects basic information about the user, including their name, age, hobbies, preferences, and driving history. Once the user begins driving, the device collects real-time data from the vehicle's sensors and sends it to the server. The server uses this data to analyze the vehicle's condition and driving situation, and provides appropriate information and advice under specific conditions.

[0216] As a specific example, if an engine abnormality is detected, the device immediately sends the abnormality data to the server. The server analyzes the received data and diagnoses the nature of the abnormality. The driver is then notified via the device with an audio alert saying, "An engine abnormality has occurred. Please stop the vehicle in a safe place and request repairs." By responding immediately in this way, serious vehicle breakdowns can be prevented.

[0217] The emotion engine also analyzes the user's emotions in real time and responds accordingly based on their emotional state. If the driver is feeling stressed, the emotion engine detects this and sends it to the server. The server then selects appropriate advice to reduce stress and music to help them relax, and provides this to the driver via the device. For example, if the emotion engine detects that the driver is stressed, it will notify the driver, "You seem to be feeling stressed at the moment. Would you like me to play some music of your choice to help you relax?"

[0218] Furthermore, when the whole family is in the car, the device uses microphones and speakers installed in each seat to assign individual topics to talk about. The emotion engine recognizes the emotional state of each member and provides appropriate dialogue. For example, if a child in the back seat looks sad, the device will ask, "What's wrong? Is there anything you'd like to talk about?" On the other hand, it will proactively provoke a conversation with the parent in the passenger seat by asking, "Why don't you think about your next family trip?"

[0219] This system realizes appropriate dialogue that takes into account the emotional states of the driver and passengers, providing a safer and more comfortable driving environment. The system provides real-time feedback according to the user's emotional state, not only improving driver safety but also creating a lively atmosphere inside the vehicle.

[0220] ---

[0221] The above is an embodiment of the present invention. By constructing a system according to this embodiment, it is possible to provide a comfortable and safe driving environment that takes into account the emotional states of the driver and passengers.

[0222] The processing flow will be explained below.

[0223] ---

[0224] Step 1:

[0225] The user enters the vehicle and logs in to the in-vehicle terminal. The terminal detects this and sends the user's login information to the server. The server uses this information to authenticate the user's profile and returns the necessary data to the terminal.

[0226] Step 2:

[0227] The device collects real-time data from the vehicle's sensors, including vehicle speed, engine status, fuel level, and interior temperature, and periodically transmits this data to a server.

[0228] Step 3:

[0229] The server receives and analyzes real-time data sent from the device, including engine anomaly detection, fuel level checks, and driver fatigue assessment.

[0230] Step 4:

[0231] The device uses the in-car camera and microphone to detect the user's facial expressions and tone of voice, and the emotion engine analyzes this data to identify the user's emotional state.

[0232] Step 5:

[0233] The server receives the emotion data sent from the emotion engine and provides appropriate information and advice to the driver based on the analysis results. For example, if the driver is under stress, the server will generate advice such as "Shall I play your favorite music to help you relax?" and provide it via the device.

[0234] Step 6:

[0235] While driving, the device uses a microphone to receive voice commands from the driver. For example, if the driver asks, "Where is the next rest stop?", the device sends the command to the server. The server analyzes and returns information about the nearest rest stop.

[0236] Step 7:

[0237] Based on the rest stop information returned from the server, the device notifies the driver, "The next rest stop is 5km away." In this way, answers to the driver's questions are provided in real time.

[0238] Step 8:

[0239] The server monitors the driver's fatigue level and generates advice to encourage rest if the driver has been driving for a long time. The device then communicates this to the driver. For example, it may say, "Long-term driving can cause fatigue, so we recommend that you take a break."

[0240] Step 9:

[0241] When the whole family is in the car, the device uses a microphone and speaker installed in each seat to individually assign topics to each member. The emotion engine recognizes the emotional state of each member and provides appropriate dialogue. For example, if a child in the back seat looks sad, the device will ask, "What's wrong? Is there anything you'd like to talk about?" On the other hand, it will proactively provoke a conversation with the parent in the passenger seat by asking, "Why don't you think about your next family trip?"

[0242] ---

[0243] The above are the processing steps for implementing the present invention. By building a system in accordance with these steps, it is possible to provide appropriate feedback in real time according to the emotional state of the driver and passengers, thereby realizing a safe and comfortable driving environment.

[0244] Example 2

[0245] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0246] Conventional vehicle driving assistance systems have limitations in providing safe driving and a comfortable driving environment for drivers, particularly in systems that take into account the driver's emotional state and passenger comfort. This can increase stress and discomfort while driving, potentially jeopardizing safe driving. Furthermore, the lack of a mechanism for providing individual conversations when the whole family is in the car makes it difficult to improve the in-car atmosphere. There is a need to address these issues and improve the overall driving experience for drivers and passengers.

[0247] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for registering and saving the driver's identification information, means for collecting data in real time from vehicle sensors and monitoring the vehicle state and driving situation, means for analyzing the collected data and providing the driver with appropriate information and advice, means for analyzing the emotional states of the driver and passengers using an emotion recognition engine, means for providing dialogue and entertainment based on the emotional states, means for allocating individual topics to each family member, and means for providing a method for the driver to relax when feeling stressed. This makes it possible to provide appropriate feedback and entertainment that takes into account the emotional states of the driver and passengers, thereby improving driving safety and comfort.

[0248] "Driver identification information" is information for identifying a driver, and includes the name, age, hobbies and preferences, driving history, and the like.

[0249] A "sensor" is a device for detecting various vehicle conditions and driving situations in real time, and includes a speed sensor, a brake sensor, an engine sensor, and the like.

[0250] "Real-time data" refers to instantaneous data collected from vehicle sensors during driving, including speed, braking status, engine temperature, and the like.

[0251] "Analysis" is the process of evaluating the driving situation and vehicle condition based on collected data and identifying problems and areas for improvement.

[0252] "Information and advice" refers to notifications and instructions provided to the driver based on the analysis results, including advice on safe driving and warnings of abnormalities.

[0253] An "emotion recognition engine" is software that analyzes the voice and facial expression data of the driver and passengers to determine their emotional state.

[0254] "Entertainment" refers to entertainment content provided to entertain the driver and passengers, and includes music, chat, games, and the like.

[0255] "Feedback" refers to reactions and responses provided to the driver and passengers based on analytical results and emotional state, and may take the form of advice or entertainment.

[0256] "All family members" refers to all passengers in the vehicle, and may include the driver.

[0257] "Individual topic allocation" means providing each passenger with an appropriate topic based on their interests and emotional state.

[0258] "Stress" refers to the mental strain or tension a driver feels while driving, and is detected by an emotion recognition engine.

[0259] "Relaxation methods" are means to reduce stress and relax the driver, and examples include music and suggestions for deep breathing.

[0260] System program generation

[0261] The server generates the system's program, which includes a series of processes for registering driver identification information, collecting real-time data, analyzing data, providing feedback, analyzing emotions, and conducting interactive dialogue.

[0262] Hardware and software used

[0263] Hardware: Devices installed in the vehicle (speed sensor, brake sensor, engine sensor, microphone, camera, speaker), cloud server

[0264] Software: Data analysis software, emotion recognition engine, interface application, cloud storage system

[0265] Program processing (natural language explanation)

[0266] Registering user information

[0267] The server collects basic information about the user when the user first connects to the system, including name, age, hobbies, driving history, etc. When the user logs in to the application, a screen for entering personal information is displayed, and the entered information is saved on the server.

[0268] Real-time data collection

[0269] The terminal collects real-time data from various sensors in the vehicle, including speed, braking status, engine temperature, and audio and video data from inside the vehicle. As soon as the driver starts driving, the sensors start operating and transmit the data to the terminal.

[0270] Data transmission and analysis

[0271] The device sends the collected data to a server in real time. The server analyzes the received data and evaluates the vehicle's condition and driving situation. If the speed data exceeds a certain range or abnormal data is detected from the engine, the server immediately detects the abnormality and performs analysis.

[0272] Providing Feedback

[0273] The server generates appropriate feedback based on the analysis results. The generated feedback is notified to the user via the terminal. For example, if an engine abnormality is detected, the server generates a message saying, "An engine abnormality has occurred. Please stop the vehicle in a safe place and request repairs," and sends it to the terminal. The terminal notifies the driver of this message as an audio alert.

[0274] Emotion analysis

[0275] The device uses a microphone and camera installed in the vehicle to collect the user's voice and facial expression data. The emotion recognition engine analyzes the user's emotions in real time based on the collected data. If the driver is feeling stressed, the emotion recognition engine detects this and sends the analysis results to the server.

[0276] Interactive Dialogue

[0277] The server and emotion recognition engine provide appropriate dialogue and entertainment based on the analysis results. For example, if the driver is feeling stressed, the device will notify them, "Your stress level seems to be increasing. Would you like me to play some relaxing music?" If a child in the back seat looks sad, the device will ask, "What's wrong? Is there anything you'd like to talk to me about?"

[0278] Examples and prompts

[0279] As a concrete example, consider a scenario where an engine malfunction occurs. The device detects the engine malfunction and sends the data to the server. The server analyzes the received data and notifies the driver with a voice alert saying, "An engine malfunction has occurred. Please stop the vehicle in a safe place and request repairs."

[0280] Example prompt sentence:

[0281] "Describe a program that uses previous driving data to generate safe driving advice."

[0282] As a process based on this prompt, the server provides the data analysis results as feedback and gives advice to the driver to encourage safe driving.

[0283] The above is an embodiment of the invention, which takes into consideration the emotional states of the driver and passengers and provides a safe and comfortable driving environment.

[0284] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0285] Step 1:

[0286] The server collects basic user information when the user first connects to the system, including name, age, hobbies, and driving history. The information is stored on the server and serves as the basis for future analysis and interactive dialogue.

[0287] Step 2:

[0288] The terminal collects data in real time from various sensors in the vehicle. Input data includes speed, braking status, engine temperature, and audio and video data inside the vehicle. The collected data is temporarily stored in the terminal and sent to the server as appropriate. This allows the server to manage detailed driving conditions.

[0289] Step 3:

[0290] The terminal sends the collected real-time data to a server. The input data is the output data from the vehicle's sensors, and once sent to the server, the data is analyzed immediately. The server analyzes the data and evaluates the driving situation and vehicle condition.

[0291] Step 4:

[0292] The server generates appropriate feedback based on the analysis results. The input data are the sensor data and analysis results sent to the server, and the output data is a feedback message for the driver. For example, the server generates a message saying, "An abnormality has occurred in the engine. Please stop the vehicle in a safe place and request repairs," and sends it to the terminal. The terminal notifies the driver of this as an audio alert.

[0293] Step 5:

[0294] The device uses the in-car microphone and camera to collect the user's voice and facial expression data. The input data is the voice and video from inside the car, and is sent to the emotion recognition engine. The emotion recognition engine analyzes this data and determines the user's emotional state.

[0295] Step 6:

[0296] The emotion recognition engine sends the result to the server based on the emotion analysis result. The input data is the analysis result by the emotion recognition engine, and the output data is the feedback of the user's emotional state to the server. If the driver is feeling stressed, the emotion recognition engine will detect this and provide the result to the server.

[0297] Step 7:

[0298] The server generates appropriate response feedback and entertainment based on the emotion analysis results. The input data is the emotional state analysis result from the emotion recognition engine, and the output data is voice feedback and entertainment content for the driver and passengers. For example, if the driver is feeling stressed, the server sends a message to the terminal saying, "Would you like me to play some relaxing music?" The terminal notifies the driver of this message by voice, and if the driver agrees, it plays relaxing music.

[0299] The above is a detailed explanation of the processing steps of the program of this system. At each step, the specific data processing and data calculation processes based on the input data are described in detail.

[0300] (Application example 2)

[0301] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0302] The purpose of this invention is to improve driving safety and comfort by analyzing the driver's emotional state in real time and providing appropriate feedback and advice based on that information. Conventional systems have difficulty responding appropriately to the driver's emotional state, and have been unable to effectively reduce stress and fatigue while driving. In addition, they have been unable to individually allocate topics of conversation, making it difficult for the whole family to enjoy a comfortable driving environment.

[0303] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting data from vehicle sensors in real time and analyzing the emotional state of the driver, means for providing appropriate feedback and advice to the driver based on the collected data, and means for suggesting music or relaxation methods based on the driver's emotional state. This makes it possible to respond in real time while taking the driver's emotional state into consideration, allowing the driver and all passengers to enjoy a safer and more comfortable driving environment.

[0304] "Driver identification information" includes individual information such as the driver's name, age, hobbies and preferences, and driving history.

[0305] "Vehicle sensors" refers to various sensor devices used to monitor the vehicle's condition and driving conditions in real time.

[0306] "Means for collecting data in real time" refers to technical devices and software for continuously collecting data from sensors installed in the vehicle.

[0307] "Means for analyzing collected data" refers to algorithms or systems for evaluating the vehicle condition and driving situation based on collected data and processing the results.

[0308] "Means for providing appropriate information and advice to drivers" refers to devices and systems for providing necessary information and driving advice to drivers.

[0309] "Means for communicating with the driver" refers to a voice recognition system or interface for directly communicating with the driver.

[0310] "Means for providing chat and entertainment based on the driver's interests and preferences" refers to a content generation system or interface for providing chat and entertainment based on the driver's interests and preferences.

[0311] "Means for individually allocating topics to each family member" refers to a dialogue system or interface that provides appropriate topics to each person in the car.

[0312] "Emotion recognition" refers to technology that analyzes a driver's emotional state based on data such as facial expressions and voice.

[0313] "Means for suggesting music or relaxation techniques based on the driver's emotional state" refers to a system that takes into account the driver's emotional state and provides appropriate music or relaxation techniques.

[0314] The Emotion Drive Assistant system of the present invention analyzes the emotional state of the driver in real time and provides appropriate feedback and advice based on that information, thereby improving driving safety and comfort. Hereinafter, an embodiment of the present invention will be described in detail.

[0315] System Configuration

[0316] The system includes a terminal in the vehicle used by the driver, a cloud server, an interface for interacting with the driver, and an emotion engine. The terminal is installed in the vehicle and is responsible for the actual data collection and monitoring, while the server analyzes and stores the data for long-term storage.

[0317] Program and processing flow

[0318] Hardware and Software

[0319] The main hardware used in this system includes:

[0320] Webcam: A camera that captures the driver's face in real time and collects data.

[0321] Smartphone: Serves as the control and interface for the entire system.

[0322] Sensors built into the vehicle: Various sensors to collect information such as speed, acceleration, engine status, etc.

[0323] The software used includes:

[0324] OpenCV: A library for processing webcam footage and performing face detection.

[0325] Keras: A machine learning library for running emotion recognition models.

[0326] requests: An HTTP library for communicating with the server.

[0327] Data processing and calculation

[0328] 1. Data Collection:

[0329] When the user starts driving, a web camera connected to the device captures video of the driver's face in real time, and various vehicle sensors collect driving situation data.

[0330] 2. Face detection and emotion recognition:

[0331] We use OpenCV to detect the driver's face from the captured video, and then input the detected face image into an emotion recognition model trained with Keras to analyze the driver's emotional state.

[0332] 3. Submitting Information and Feedback:

[0333] The analyzed emotional state is sent to the server using the requests library. The server analyzes the received data, generates appropriate feedback and advice, and conveys it to the driver via the terminal.

[0334] Specific examples and prompts for the generative AI model

[0335] Examples:

[0336] When a driver launches the EmotionDrive Assistant app and starts driving, the app begins to recognize emotions. If the driver is feeling stressed, the server receives that information and suggests appropriate relaxation methods (such as playing music or instructing breathing exercises) to the device.

[0337] Example prompt for a generative AI model:

[0338] "Design a system in which a smartphone application installed in an autonomous vehicle analyzes the emotional state of the driver in real time and provides appropriate feedback and advice based on that emotional state. The emotion recognition model uses Keras, and the video footage from the webcam is processed in real time."

[0339] This system enables real-time responses that take into account the driver's emotional state, allowing both the driver and passengers to enjoy a safer and more comfortable driving environment.

[0340] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0341] Step 1:

[0342] When the user starts driving, the web camera connected to the device captures the video of the driver's face in real time. The input is the video of the driver's face, and the output is the captured video. This video is used directly for the next step.

[0343] Step 2:

[0344] The device uses OpenCV to detect the driver's face from the captured video. The input of this step is the video captured in step 1, and the output is an image of the area where the face is detected. This face image is used in the next step for facial recognition.

[0345] Step 3:

[0346] The device inputs the facial image into an emotion recognition model trained with Keras to analyze the driver's emotional state. The input is the facial image obtained in step 2, and the output is the emotional state label and its probability. These data are saved for use in the next step.

[0347] Step 4:

[0348] The device uses the requests library to send data containing the analyzed emotional state to the server. The input is the emotional state label and probability, and the output is the emotional state data received by the server. The server receives this data.

[0349] Step 5:

[0350] The server analyzes the received data and generates appropriate feedback or advice. The input of this step is the emotional state data, and the output is the generated feedback or advice. This feedback is used in the next step.

[0351] Step 6:

[0352] The server sends the generated feedback and advice to the terminal. The input is the feedback and advice generated in step 5, and the output is the feedback and advice received by the terminal.

[0353] Step 7:

[0354] The device displays or outputs audio in an appropriate manner to convey the received feedback or advice to the driver. The input is the received feedback or advice, and the output is the actual feedback or advice provided to the driver. Specific actions include displaying the feedback or advice on a screen or announcing it via audio.

[0355] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0356] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0357] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0358] [Second embodiment]

[0359] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0360] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0361] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0362] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0363] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0364] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0365] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0366] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0367] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0368] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0369] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0370] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0371] ---

[0372] The present invention relates to a system that collects data specific to a driver by registering and storing the driver's identification information, and provides safe driving and a comfortable driving environment based on that data.

[0373] System Configuration

[0374] The system of the present invention includes a vehicle terminal used by the user, a cloud server, and an interface for interaction with the driver. The terminal is installed in the vehicle and is responsible for the actual data collection and monitoring. The server analyzes and stores the data for long-term storage. The users are the driver and passengers, and each receives appropriate dialogue and feedback.

[0375] Program and processing flow

[0376] When a user first connects to the system, the server collects basic information about the user, including their name, age, hobbies, preferences, and driving history. Once the user begins driving, the device collects real-time data from the vehicle's sensors and sends it to the server. The server uses this data to analyze the vehicle's condition and driving situation, and provides appropriate information and advice under specific conditions.

[0377] As a specific example, if an engine abnormality is detected, the device immediately sends the abnormality data to the server. The server analyzes the received data and diagnoses the nature of the abnormality. The driver is then notified via the device with an audio alert saying, "An engine abnormality has occurred. Please stop the vehicle in a safe place and request repairs." By responding immediately in this way, serious vehicle breakdowns can be prevented.

[0378] In addition, if the server detects fatigue in a user who has been driving for a long time, it will provide advice such as, "Long-term driving can cause fatigue, so we recommend that you take a break." This can reduce the risk of accidents caused by fatigued driving.

[0379] In addition, sensing devices inside the vehicle can be used to conduct voice interactions with the user and passengers. For example, if a user asks, "Where is the next rest stop?", the device sends this information to a server, which returns data about the nearest rest stop. The server then generates an appropriate response and relays it to the driver via the device.

[0380] When the whole family is in the car, the device can use the microphone and speaker at each seat to enable individual conversations. For example, it can prompt a child in the back seat with a question like, "What book have you read recently?", while asking a parent in the passenger seat, "Are you thinking about planning a family trip?" In this way, everyone can have appropriate topics to talk about, creating a lively atmosphere in the car.

[0381] ---

[0382] The above is an embodiment of the present invention. However, the present invention is not limited to this embodiment and may be modified as appropriate. For example, the type of data collected, the analysis method, and the content of the advice provided can be changed according to the system specifications and the needs of the user.

[0383] The processing flow will be explained below.

[0384] ---

[0385] Step 1:

[0386] The user enters the vehicle and logs in to the in-vehicle terminal. The terminal detects this and sends the user's login information to the server. The server uses this information to authenticate the user's profile and returns the necessary data to the terminal.

[0387] Step 2:

[0388] The device collects real-time data from the vehicle's sensors, including vehicle speed, engine status, fuel level, and interior temperature, and periodically transmits this data to a server.

[0389] Step 3:

[0390] The server receives and analyzes real-time data sent from the device, including engine anomaly detection, fuel level checks, and driver fatigue assessment.

[0391] Step 4:

[0392] Based on the analysis results from the server, the device will provide the driver with appropriate information and advice. For example, if an engine abnormality is detected, the device will issue a voice alert saying, "An engine abnormality has occurred. Please stop in a safe place and request repairs."

[0393] Step 5:

[0394] While driving, the device uses a microphone to receive voice commands from the driver. For example, if the driver asks, "Where is the next rest stop?", the device sends the command to the server. The server analyzes and returns information about the nearest rest stop.

[0395] Step 6:

[0396] Based on the rest stop information returned from the server, the device notifies the driver, "The next rest stop is 5km away." In this way, answers to the driver's questions are provided in real time.

[0397] Step 7:

[0398] The server monitors the driver's fatigue level and generates advice to encourage rest if the driver has been driving for a long time. The device then communicates this to the driver. For example, it may say, "Long-term driving can cause fatigue, so we recommend that you take a break."

[0399] Step 8:

[0400] When the whole family is in the car, the device uses a microphone and speaker installed in each seat to individually target each person with a topic, such as asking a child in the back seat, "What book have you read recently?", or speaking to a parent in the front seat, "Are you thinking about your next family trip?"

[0401] ---

[0402] The above are the processing steps for implementing the present invention. By building a system in accordance with these steps, it is possible to significantly improve the safety and comfort of the driver and passengers.

[0403] Example 1

[0404] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0405] In recent years, there has been a demand for improved safety and comfort in vehicle driving. However, conventional systems are unable to fully utilize individual driver information, making it difficult to provide appropriate advice and warnings in real time. They are also inadequate at immediately detecting and responding to driver fatigue or vehicle abnormalities. Furthermore, their ability to provide information and entertainment tailored to individual needs through two-way dialogue with the driver is limited. There is a need to resolve these issues and improve driver safety and comfort.

[0406] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0407] In this invention, the server includes a means for registering and storing the driver's identification information, a means for collecting data in real time from the vehicle's sensors and monitoring the vehicle's condition and driving situation, and a means for analyzing the collected data and providing the driver with appropriate information and advice. This makes it possible to provide appropriate advice based on the driver's individual information in real time. It also makes it possible to reduce the risk of accidents by detecting the driver's fatigue level and encouraging them to take a break, and to avoid serious breakdowns by immediately notifying the driver of vehicle abnormalities. Furthermore, it is possible to improve the comfort of the driving environment by interacting with the driver and providing information and entertainment based on the driver's hobbies and preferences.

[0408] 1. "Driver identification information" refers to data such as name, age, hobbies and preferences, and driving history used to identify and manage individual drivers.

[0409] 2. "Means for registering and storing" means a technology or device for inputting driver identification information and storing that data in a storage device.

[0410] 3. "Sensor" means a device used to detect and collect data about the vehicle's condition and driving situation. Examples include engine sensors, speed sensors, and cameras.

[0411] 4. "Means for collecting and monitoring data in real time" refers to technology that continuously and instantly collects data obtained from sensors and monitors the vehicle's condition and driving situation.

[0412] 5. "Means for analyzing and providing data" refers to the technology that processes collected data using an analysis program and, as a result, provides useful information and advice to the driver.

[0413] 6. "Means of dialogue" means technology that enables two-way communication with the driver via voice and text.

[0414] 7. "Means for providing conversation and entertainment based on the driver's interests and preferences" refers to technology that provides dialogue and entertainment based on the driver's individual information, according to the driver's interests and preferences.

[0415] 8. "Means for individual topic allocation" refers to technology that provides different conversation content to different passengers in the vehicle, enabling communication that is tailored to their individual interests and situations.

[0416] 9. "Means for issuing an alert when an engine abnormality is detected" refers to technology and devices that monitor the engine condition and issue a warning to the driver when an abnormality is detected.

[0417] 10. "Means for detecting fatigue and providing advice to encourage rest" refers to technology that determines the driver's level of fatigue from continuous driving data and provides information to encourage rest at an appropriate time.

[0418] 11. "Means for analyzing questions based on voice input and providing answers" refers to technology that converts questions entered by the driver through voice into text, analyzes the content of that text, and generates and provides the most appropriate answer.

[0419] The present invention relates to a system that collects data specific to a driver by registering and storing the driver's identification information, and provides safe driving and a comfortable driving environment based on that data.

[0420] System Configuration

[0421] The system of the present invention includes a terminal in the vehicle used by the user, a cloud server, and an interface for interaction with the driver. The terminal is installed in the vehicle and is responsible for the actual data collection and monitoring, while the server analyzes and stores the data for long-term storage.

[0422] Program and process description

[0423] When a user first connects to the system, the server collects basic information about the user, including their name, age, hobbies, preferences, and driving history. Once the user begins driving, the device collects real-time data from the vehicle's sensors and sends it to the server. The server uses this data to analyze the vehicle's condition and driving situation, and provides appropriate information and advice under specific conditions.

[0424] Specifically, the terminal uses the following hardware and software:

[0425] In-vehicle devices: for data collection and monitoring

[0426] Cloud server: for data analysis and long-term storage

[0427] Sensors (e.g. engine sensors, speed sensors, cameras): for real-time data collection and interaction

[0428] The server processes and calculates the collected data as follows:

[0429] 1. Receives data sent from each vehicle sensor and monitors it in real time.

[0430] 2. Use a data analysis program (e.g., Python script) to detect engine anomalies and fatigued driving.

[0431] 3. After detecting an anomaly, appropriate feedback and advice is generated and notified to the user in real time.

[0432] Specific examples

[0433] Examples of anomaly detection:

[0434] Situation: The server detects an abnormality in the engine sensor.

[0435] Processing: The abnormal data is analyzed, and an audio alert is sent to the user via the device stating, "An abnormality has occurred in the engine. Please stop the vehicle in a safe place and request repairs."

[0436] Prompt sentence to input to the generative AI model:

[0437] "An engine anomaly has been detected. Please generate a script to diagnose the anomaly and provide an audio alert to the user."

[0438] Examples of fatigue detection:

[0439] Situation: The server analyzes long-term driving data and estimates fatigue.

[0440] Action: Provide the user with the advice, "Driving for long periods of time can cause fatigue, so we recommend you take a break."

[0441] Prompt sentence to input to the generative AI model:

[0442] "If fatigue due to long driving is detected, generate a script that provides advice to encourage the user to take an appropriate break."

[0443] Examples of voice interactions:

[0444] Situation: A user asks the device, "Where is the next rest stop?"

[0445] Processing: The device sends this question to the server, which responds with information about the nearest rest area. The device then provides this information to the user.

[0446] Prompt sentence to input to the generative AI model:

[0447] "When a user asks, 'Where is the next rest stop?', generate a script that looks up information about the nearest rest stop and responds."

[0448] Examples of interactions with the whole family:

[0449] Situation: There are multiple passengers in the vehicle.

[0450] Processing: The device uses data from each seat's microphone and speaker to facilitate personalized conversations, asking the child in the back seat, "What book have you read recently?" and the parent in the front seat, "Are you thinking about planning a family trip?"

[0451] Prompt sentence to input to the generative AI model:

[0452] "Generate scripts that provide appropriate conversation topics for each passenger in the car. For example, ask the child in the back seat, 'What book have you read recently?' and the parent in the front seat, 'Are you thinking about planning a family trip?'"

[0453] The above is an embodiment of the present invention. This system allows the driver to receive advice and warnings in real time, improving safety and comfort.

[0454] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0455] Step 1:

[0456] Registering and saving user information

[0457] Input: The user enters basic information such as name, age, hobbies, preferences, and driving history into the vehicle's terminal.

[0458] Specific operation: The user enters each piece of information on the device screen.

[0459] Processing: The device sends the entered information to a server in the cloud. After receiving the information, the server stores it in a database and manages it for each user.

[0460] Output: Driver identification stored in a database.

[0461] Step 2:

[0462] Real-time data collection

[0463] Input: Data from each vehicle sensor (engine sensor, speed sensor, camera, etc.).

[0464] Specific operation: The device collects data in real time from sensors installed in the vehicle.

[0465] Processing: The device sends the collected data to a server in the cloud at regular intervals.

[0466] Output: Real-time sensor data sent to the server.

[0467] Step 3:

[0468] Data analysis

[0469] Input: Sensor data sent to the server.

[0470] Specific operation: The server analyzes the data using Python data analysis libraries (such as Pandas and NumPy).

[0471] Processing: Analyzes engine temperature, speed data, etc. to detect abnormalities and estimate driver fatigue.

[0472] Output: Analysis results include the presence or absence of abnormalities and the driver's fatigue state.

[0473] Step 4:

[0474] Generate alerts and advice

[0475] Input: Data analysis results (anomaly detection results, fatigue estimation results).

[0476] Specific behavior: The server generates appropriate alerts and advice based on the analysis results.

[0477] In case of engine abnormality: Generates a message saying "An engine abnormality has occurred. Please stop the vehicle in a safe place and request repairs."

[0478] For fatigue detection: Generate advice such as "Driving for long periods of time can cause fatigue, so we recommend that you take a break."

[0479] Processing: Sends generated alerts and advice to the device.

[0480] Output: Message data for generated alerts and advice.

[0481] Step 5:

[0482] User Notification

[0483] Input: The alert or advice message data sent by the server.

[0484] Specific behavior: The device notifies the user through voice and display.

[0485] In case of engine abnormality: An audio alert will be issued saying, "An engine abnormality has occurred. Please stop in a safe place and request repairs."

[0486] If fatigue is detected: A voice will advise, "Driving for long periods of time can cause fatigue, so we recommend that you take a break."

[0487] Action: Provide real-time alerts and advice to users.

[0488] Output: Proper alerts and advice to the driver.

[0489] Step 6:

[0490] Voice interaction processing

[0491] Input: The user speaks a question to the device, for example, "Where is the next rest stop?"

[0492] Specific operation: The device collects voice data and sends it to a server in the cloud.

[0493] Processing: The server converts the speech to text, analyzes the question, and generates the best answer.

[0494] Example: Generate an answer by getting information about the nearest rest stop.

[0495] Output: Sends the generated answer text data to the terminal.

[0496] Specific operation: The device converts the transmitted response data into voice and provides it to the user.

[0497] Output: The appropriate answer information provided to the user.

[0498] (Application example 1)

[0499] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0500] Autonomous vehicles require efficient registration of driver identification information and the provision of personalized driving experiences. It is also necessary to improve the safety and comfort of drivers and passengers by improving the accuracy of advice provided based on real-time monitoring of driving conditions and data analysis. While linking with various devices and utilizing generative AI models are effective ways to achieve these goals, many aspects of specific implementation remain unresolved.

[0501] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0502] In this invention, the server includes means for registering and storing the driver's identification information, means for collecting data in real time from vehicle sensors and monitoring the vehicle's status and driving situation, means for analyzing the collected data and providing the driver with appropriate information and advice, means for interacting with the driver and providing chat and entertainment based on the driver's hobbies and preferences, means for allocating individual topics to each family member, means for transmitting basic information about the user to a cloud server in the autonomous vehicle and providing analysis results and advice to a smart device in real time, and means for generating appropriate responses to the user's voice commands using a generative AI model. This enables the provision of a personalized driving experience based on the user's identification information, the provision of appropriate advice according to the situation of the autonomous vehicle, and advanced voice responses using the generative AI model.

[0503] "Driver identification information" is a series of information for identifying an individual driver, such as the user's name, age, driving history, etc.

[0504] "Real-time data collection" means using sensors and other input devices to instantly obtain data about the ongoing driving situation and vehicle status.

[0505] "Monitoring vehicle condition and driving status" means monitoring engine status, speed, location information, etc. through various sensors in the vehicle to determine whether it is operating normally.

[0506] "Analyzing collected data" means using algorithms and analytical tools based on the acquired data to detect vehicle abnormalities and the driver's condition.

[0507] "Providing appropriate information and advice" means providing drivers with advice for safe driving and comfortable driving based on the analysis results.

[0508] "Dialogue with the driver" means two-way communication with the driver using means such as voice recognition or text input.

[0509] "Providing chat and entertainment based on the driver's hobbies and preferences" means providing topics, music, and other entertainment content that meet the driver's individual preferences.

[0510] "Allocating individual topics to each family member" means bringing up topics that correspond to the interests and concerns of each passenger in the car and conducting individual conversations.

[0511] "Send to a server on the cloud" means uploading data to a remote server via the Internet.

[0512] A "generative AI model" is an algorithm based on machine learning and deep learning that is used to perform advanced tasks such as speech recognition and text generation.

[0513] "Generating an appropriate response to a voice command" means recognizing a voice instruction given by the user and providing an appropriate reaction or information in response to it.

[0514] "Smart devices" is a general term for mobile devices that can connect to the Internet, such as smartphones, tablets, and smart glasses.

[0515] "Personalized driving experience" refers to providing a driving environment and services that are customized based on a user's individual identity and preferences.

[0516] This invention is a system for enhancing the driver-specific experience in autonomous vehicles. The system registers the driver's identification information, monitors the vehicle's status and driving situation in real time, analyzes the collected data, and provides appropriate information and advice. Furthermore, it can use a generative AI model to interact with the driver and provide personalized conversations and entertainment.

[0517] The server first registers the driver's identification information in a database on the cloud. This information includes the user's name, age, driving history, etc. In addition, a terminal installed in the vehicle uses various sensors to collect vehicle status (engine status, speed, location information, etc.) in real time. This data is immediately sent to the server, which then analyzes it.

[0518] Based on the analysis results, the server generates advice and alerts appropriate to the situation and provides them to the driver via a smart device (such as a smartphone or smart glasses). For example, if an engine abnormality is detected, the server will issue an alert stating, "An engine abnormality has occurred. Please stop the vehicle in a safe place and request repairs."

[0519] Additionally, if the server detects that the driver is fatigued, it will provide advice such as, "Long-term driving can cause fatigue, so we recommend you take a break," ensuring the driver's safety. Furthermore, when the driver issues a voice command using the generative AI model, the server generates an appropriate response based on the analysis results and provides it to the driver.

[0520] For example, if a user asks, "Where is the next rest stop?", the smart device sends this information to the server, which uses data about the nearest rest stops to generate a response such as "There is a rest stop XX kilometers away."

[0521] Here is an example prompt:

[0522] "The user wants to know the nearest rest area from their current location. Their current location is [latitude, longitude]. What is the next available rest area?"

[0523] The hardware used includes smartphones, smart glasses, and autonomous vehicle sensors (cameras, GPS, microphones, etc.), while the software used includes cloud servers (AWS, Google Cloud), machine learning models (TensorFlow, PyTorch), and real-time data analysis (Apache Kafka).

[0524] The system will provide a personalized driving experience based on user identity, monitor vehicle status in real time and provide appropriate advice, and use generative AI models to provide advanced voice responses.

[0525] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0526] Step 1:

[0527] When a user gets into a vehicle, the device acquires the user's identification information and sends it to the cloud server. Basic information such as the user's name, age, and driving history is received as input, and based on this information, a new registration or update is made on the cloud. A user ID is generated as output.

[0528] Step 2:

[0529] The terminal collects data in real time from various sensors in the vehicle (engine status, speed, location information, etc.) and sends the collected data to a cloud server. The input includes signal data from each sensor. The server receives this and stores it in a database. The output is accumulated data.

[0530] Step 3:

[0531] The server analyzes the collected data and diagnoses the driver's condition. The stored sensor data is used as input. A machine learning model (e.g., TensorFlow, PyTorch) is used for data analysis to detect signs and abnormalities for safe driving. The analysis results are generated as output.

[0532] Step 4:

[0533] Based on the analysis results, the server automatically generates appropriate advice or alerts and notifies the driver via the terminal. The analysis results are used as input. For example, a message such as "There is an engine malfunction. Please stop the vehicle in a safe place and request repairs" is generated. As output, a voice or text notification is sent to the driver.

[0534] Step 5:

[0535] The server detects the driver's fatigue state from sensor data and provides advice such as, "You are showing signs of fatigue. We recommend that you take a break." Input includes vital data such as heart rate and reaction time. A machine learning model is used to analyze this data and detect signs of fatigue. The output is a notification encouraging the driver to take a break.

[0536] Step 6:

[0537] When a driver inputs a voice command via a smart device (smartphone or smart glasses), the device sends the information to the server. The user's voice command is used as input. Example prompt: "Where is the next rest stop?" The server uses a generative AI model to generate an appropriate response and returns it to the user. Information about rest stops is returned as output.

[0538] Step 7:

[0539] As part of the response generation process, the server performs database searches and calculations in the cloud. The input includes a prompt to the generative AI model. For example, the prompt might read, "The user wants to know the nearest rest area from their current location. Their current location is [latitude, longitude]. Please tell me where the next available rest area is." The server retrieves data on the nearest rest areas and generates an appropriate answer. The output is a response containing the location of the rest area.

[0540] These processing steps enable the system of the present invention to effectively collect and analyze driver-specific data to provide a safe and comfortable driving experience.

[0541] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0542] ---

[0543] This invention achieves even greater driving safety and comfort by collecting driver-specific data by registering and storing the driver's identification information, and combining this data with an emotion engine in a system that provides safe driving and a comfortable driving environment.

[0544] System Configuration

[0545] The system of the present invention includes a vehicle terminal used by the user, a cloud server, an interface for interacting with the driver, and an emotion engine. The terminal is installed in the vehicle and is responsible for the actual data collection and monitoring. The server analyzes and stores the data for long-term storage. The emotion engine has the function of recognizing the user's emotions and generating appropriate dialogue and responses based on those emotions. The users are the driver and passengers, and each receives dialogue and feedback tailored to their needs.

[0546] Program and processing flow

[0547] When a user first connects to the system, the server collects basic information about the user, including their name, age, hobbies, preferences, and driving history. Once the user begins driving, the device collects real-time data from the vehicle's sensors and sends it to the server. The server uses this data to analyze the vehicle's condition and driving situation, and provides appropriate information and advice under specific conditions.

[0548] As a specific example, if an engine abnormality is detected, the device immediately sends the abnormality data to the server. The server analyzes the received data and diagnoses the nature of the abnormality. The driver is then notified via the device with an audio alert saying, "An engine abnormality has occurred. Please stop the vehicle in a safe place and request repairs." By responding immediately in this way, serious vehicle breakdowns can be prevented.

[0549] The emotion engine also analyzes the user's emotions in real time and responds accordingly based on their emotional state. If the driver is feeling stressed, the emotion engine detects this and sends it to the server. The server then selects appropriate advice to reduce stress and music to help them relax, and provides this to the driver via the device. For example, if the emotion engine detects that the driver is stressed, it will notify the driver, "You seem to be feeling stressed at the moment. Would you like me to play some music of your choice to help you relax?"

[0550] Furthermore, when the whole family is in the car, the device uses microphones and speakers installed in each seat to assign individual topics to talk about. The emotion engine recognizes the emotional state of each member and provides appropriate dialogue. For example, if a child in the back seat looks sad, the device will ask, "What's wrong? Is there anything you'd like to talk about?" On the other hand, it will proactively provoke a conversation with the parent in the passenger seat by asking, "Why don't you think about your next family trip?"

[0551] This system realizes appropriate dialogue that takes into account the emotional states of the driver and passengers, providing a safer and more comfortable driving environment. The system provides real-time feedback according to the user's emotional state, not only improving driver safety but also creating a lively atmosphere inside the vehicle.

[0552] ---

[0553] The above is an embodiment of the present invention. By constructing a system according to this embodiment, it is possible to provide a comfortable and safe driving environment that takes into account the emotional states of the driver and passengers.

[0554] The processing flow will be explained below.

[0555] ---

[0556] Step 1:

[0557] The user enters the vehicle and logs in to the in-vehicle terminal. The terminal detects this and sends the user's login information to the server. The server uses this information to authenticate the user's profile and returns the necessary data to the terminal.

[0558] Step 2:

[0559] The device collects real-time data from the vehicle's sensors, including vehicle speed, engine status, fuel level, and interior temperature, and periodically transmits this data to a server.

[0560] Step 3:

[0561] The server receives and analyzes real-time data sent from the device, including engine anomaly detection, fuel level checks, and driver fatigue assessment.

[0562] Step 4:

[0563] The device uses the in-car camera and microphone to detect the user's facial expressions and tone of voice, and the emotion engine analyzes this data to identify the user's emotional state.

[0564] Step 5:

[0565] The server receives the emotion data sent from the emotion engine and provides appropriate information and advice to the driver based on the analysis results. For example, if the driver is under stress, the server will generate advice such as "Shall I play your favorite music to help you relax?" and provide it via the device.

[0566] Step 6:

[0567] While driving, the device uses a microphone to receive voice commands from the driver. For example, if the driver asks, "Where is the next rest stop?", the device sends the command to the server. The server analyzes and returns information about the nearest rest stop.

[0568] Step 7:

[0569] Based on the rest stop information returned from the server, the device notifies the driver, "The next rest stop is 5km away." In this way, answers to the driver's questions are provided in real time.

[0570] Step 8:

[0571] The server monitors the driver's fatigue level and generates advice to encourage rest if the driver has been driving for a long time. The device then communicates this to the driver. For example, it may say, "Long-term driving can cause fatigue, so we recommend that you take a break."

[0572] Step 9:

[0573] When the whole family is in the car, the device uses a microphone and speaker installed in each seat to individually assign topics to each member. The emotion engine recognizes the emotional state of each member and provides appropriate dialogue. For example, if a child in the back seat looks sad, the device will ask, "What's wrong? Is there anything you'd like to talk about?" On the other hand, it will proactively provoke a conversation with the parent in the passenger seat by asking, "Why don't you think about your next family trip?"

[0574] ---

[0575] The above are the processing steps for implementing the present invention. By building a system in accordance with these steps, it is possible to provide appropriate feedback in real time according to the emotional state of the driver and passengers, thereby realizing a safe and comfortable driving environment.

[0576] Example 2

[0577] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0578] Conventional vehicle driving assistance systems have limitations in providing safe driving and a comfortable driving environment for drivers, particularly in systems that take into account the driver's emotional state and passenger comfort. This can increase stress and discomfort while driving, potentially jeopardizing safe driving. Furthermore, the lack of a mechanism for providing individual conversations when the whole family is in the car makes it difficult to improve the in-car atmosphere. There is a need to address these issues and improve the overall driving experience for drivers and passengers.

[0579] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for registering and saving the driver's identification information, means for collecting data in real time from vehicle sensors and monitoring the vehicle state and driving situation, means for analyzing the collected data and providing the driver with appropriate information and advice, means for analyzing the emotional states of the driver and passengers using an emotion recognition engine, means for providing dialogue and entertainment based on the emotional states, means for allocating individual topics to each family member, and means for providing a method for the driver to relax when feeling stressed. This makes it possible to provide appropriate feedback and entertainment that takes into account the emotional states of the driver and passengers, thereby improving driving safety and comfort.

[0580] "Driver identification information" is information for identifying a driver, and includes the name, age, hobbies and preferences, driving history, and the like.

[0581] A "sensor" is a device for detecting various vehicle conditions and driving situations in real time, and includes a speed sensor, a brake sensor, an engine sensor, and the like.

[0582] "Real-time data" refers to instantaneous data collected from vehicle sensors during driving, including speed, braking status, engine temperature, and the like.

[0583] "Analysis" is the process of evaluating the driving situation and vehicle condition based on collected data and identifying problems and areas for improvement.

[0584] "Information and advice" refers to notifications and instructions provided to the driver based on the analysis results, including advice on safe driving and warnings of abnormalities.

[0585] An "emotion recognition engine" is software that analyzes the voice and facial expression data of the driver and passengers to determine their emotional state.

[0586] "Entertainment" refers to entertainment content provided to entertain the driver and passengers, and includes music, chat, games, and the like.

[0587] "Feedback" refers to reactions and responses provided to the driver and passengers based on analytical results and emotional state, and may take the form of advice or entertainment.

[0588] "All family members" refers to all passengers in the vehicle, and may include the driver.

[0589] "Individual topic allocation" means providing each passenger with an appropriate topic based on their interests and emotional state.

[0590] "Stress" refers to the mental strain or tension a driver feels while driving, and is detected by an emotion recognition engine.

[0591] "Relaxation methods" are means to reduce stress and relax the driver, and examples include music and suggestions for deep breathing.

[0592] System program generation

[0593] The server generates the system's program, which includes a series of processes for registering driver identification information, collecting real-time data, analyzing data, providing feedback, analyzing emotions, and conducting interactive dialogue.

[0594] Hardware and software used

[0595] Hardware: Devices installed in the vehicle (speed sensor, brake sensor, engine sensor, microphone, camera, speaker), cloud server

[0596] Software: Data analysis software, emotion recognition engine, interface application, cloud storage system

[0597] Program processing (natural language explanation)

[0598] Registering user information

[0599] The server collects basic information about the user when the user first connects to the system, including name, age, hobbies, driving history, etc. When the user logs in to the application, a screen for entering personal information is displayed, and the entered information is saved on the server.

[0600] Real-time data collection

[0601] The terminal collects real-time data from various sensors in the vehicle, including speed, braking status, engine temperature, and audio and video data from inside the vehicle. As soon as the driver starts driving, the sensors start operating and transmit the data to the terminal.

[0602] Data transmission and analysis

[0603] The device sends the collected data to a server in real time. The server analyzes the received data and evaluates the vehicle's condition and driving situation. If the speed data exceeds a certain range or abnormal data is detected from the engine, the server immediately detects the abnormality and performs analysis.

[0604] Providing Feedback

[0605] The server generates appropriate feedback based on the analysis results. The generated feedback is notified to the user via the terminal. For example, if an engine abnormality is detected, the server generates a message saying, "An engine abnormality has occurred. Please stop the vehicle in a safe place and request repairs," and sends it to the terminal. The terminal notifies the driver of this message as an audio alert.

[0606] Emotion analysis

[0607] The device uses a microphone and camera installed in the vehicle to collect the user's voice and facial expression data. The emotion recognition engine analyzes the user's emotions in real time based on the collected data. If the driver is feeling stressed, the emotion recognition engine detects this and sends the analysis results to the server.

[0608] Interactive Dialogue

[0609] The server and emotion recognition engine provide appropriate dialogue and entertainment based on the analysis results. For example, if the driver is feeling stressed, the device will notify them, "Your stress level seems to be increasing. Would you like me to play some relaxing music?" If a child in the back seat looks sad, the device will ask, "What's wrong? Is there anything you'd like to talk to me about?"

[0610] Examples and prompts

[0611] As a concrete example, consider a scenario where an engine malfunction occurs. The device detects the engine malfunction and sends the data to the server. The server analyzes the received data and notifies the driver with a voice alert saying, "An engine malfunction has occurred. Please stop the vehicle in a safe place and request repairs."

[0612] Example prompt sentence:

[0613] "Describe a program that uses previous driving data to generate safe driving advice."

[0614] As a process based on this prompt, the server provides the data analysis results as feedback and gives advice to the driver to encourage safe driving.

[0615] The above is an embodiment of the invention, which takes into consideration the emotional states of the driver and passengers and provides a safe and comfortable driving environment.

[0616] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0617] Step 1:

[0618] The server collects basic user information when the user first connects to the system, including name, age, hobbies, and driving history. The information is stored on the server and serves as the basis for future analysis and interactive dialogue.

[0619] Step 2:

[0620] The terminal collects data in real time from various sensors in the vehicle. Input data includes speed, braking status, engine temperature, and audio and video data inside the vehicle. The collected data is temporarily stored in the terminal and sent to the server as appropriate. This allows the server to manage detailed driving conditions.

[0621] Step 3:

[0622] The terminal sends the collected real-time data to a server. The input data is the output data from the vehicle's sensors, and once sent to the server, the data is analyzed immediately. The server analyzes the data and evaluates the driving situation and vehicle condition.

[0623] Step 4:

[0624] The server generates appropriate feedback based on the analysis results. The input data are the sensor data and analysis results sent to the server, and the output data is a feedback message for the driver. For example, the server generates a message saying, "An abnormality has occurred in the engine. Please stop the vehicle in a safe place and request repairs," and sends it to the terminal. The terminal notifies the driver of this as an audio alert.

[0625] Step 5:

[0626] The device uses the in-car microphone and camera to collect the user's voice and facial expression data. The input data is the voice and video from inside the car, and is sent to the emotion recognition engine. The emotion recognition engine analyzes this data and determines the user's emotional state.

[0627] Step 6:

[0628] The emotion recognition engine sends the result to the server based on the emotion analysis result. The input data is the analysis result by the emotion recognition engine, and the output data is the feedback of the user's emotional state to the server. If the driver is feeling stressed, the emotion recognition engine will detect this and provide the result to the server.

[0629] Step 7:

[0630] The server generates appropriate response feedback and entertainment based on the emotion analysis results. The input data is the emotional state analysis result from the emotion recognition engine, and the output data is voice feedback and entertainment content for the driver and passengers. For example, if the driver is feeling stressed, the server sends a message to the terminal saying, "Would you like me to play some relaxing music?" The terminal notifies the driver of this message by voice, and if the driver agrees, it plays relaxing music.

[0631] The above is a detailed explanation of the processing steps of the program of this system. At each step, the specific data processing and data calculation processes based on the input data are described in detail.

[0632] (Application example 2)

[0633] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0634] The purpose of this invention is to improve driving safety and comfort by analyzing the driver's emotional state in real time and providing appropriate feedback and advice based on that information. Conventional systems have difficulty responding appropriately to the driver's emotional state, and have been unable to effectively reduce stress and fatigue while driving. In addition, they have been unable to individually allocate topics of conversation, making it difficult for the whole family to enjoy a comfortable driving environment.

[0635] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting data from vehicle sensors in real time and analyzing the emotional state of the driver, means for providing appropriate feedback and advice to the driver based on the collected data, and means for suggesting music or relaxation methods based on the driver's emotional state. This makes it possible to respond in real time while taking the driver's emotional state into consideration, allowing the driver and all passengers to enjoy a safer and more comfortable driving environment.

[0636] "Driver identification information" includes individual information such as the driver's name, age, hobbies and preferences, and driving history.

[0637] "Vehicle sensors" refers to various sensor devices used to monitor the vehicle's condition and driving conditions in real time.

[0638] "Means for collecting data in real time" refers to technical devices and software for continuously collecting data from sensors installed in the vehicle.

[0639] "Means for analyzing collected data" refers to algorithms or systems for evaluating the vehicle condition and driving situation based on collected data and processing the results.

[0640] "Means for providing appropriate information and advice to drivers" refers to devices and systems for providing necessary information and driving advice to drivers.

[0641] "Means for communicating with the driver" refers to a voice recognition system or interface for directly communicating with the driver.

[0642] "Means for providing chat and entertainment based on the driver's interests and preferences" refers to a content generation system or interface for providing chat and entertainment based on the driver's interests and preferences.

[0643] "Means for individually allocating topics to each family member" refers to a dialogue system or interface that provides appropriate topics to each person in the car.

[0644] "Emotion recognition" refers to technology that analyzes a driver's emotional state based on data such as facial expressions and voice.

[0645] "Means for suggesting music or relaxation techniques based on the driver's emotional state" refers to a system that takes into account the driver's emotional state and provides appropriate music or relaxation techniques.

[0646] The Emotion Drive Assistant system of the present invention analyzes the emotional state of the driver in real time and provides appropriate feedback and advice based on that information, thereby improving driving safety and comfort. Hereinafter, an embodiment of the present invention will be described in detail.

[0647] System Configuration

[0648] The system includes a terminal in the vehicle used by the driver, a cloud server, an interface for interacting with the driver, and an emotion engine. The terminal is installed in the vehicle and is responsible for the actual data collection and monitoring, while the server analyzes and stores the data for long-term storage.

[0649] Program and processing flow

[0650] Hardware and Software

[0651] The main hardware used in this system includes:

[0652] Webcam: A camera that captures the driver's face in real time and collects data.

[0653] Smartphone: Serves as the control and interface for the entire system.

[0654] Sensors built into the vehicle: Various sensors to collect information such as speed, acceleration, engine status, etc.

[0655] The software used includes:

[0656] OpenCV: A library for processing webcam footage and performing face detection.

[0657] Keras: A machine learning library for running emotion recognition models.

[0658] requests: An HTTP library for communicating with the server.

[0659] Data processing and calculation

[0660] 1. Data Collection:

[0661] When the user starts driving, a web camera connected to the device captures video of the driver's face in real time, and various vehicle sensors collect driving situation data.

[0662] 2. Face detection and emotion recognition:

[0663] We use OpenCV to detect the driver's face from the captured video, and then input the detected face image into an emotion recognition model trained with Keras to analyze the driver's emotional state.

[0664] 3. Submitting Information and Feedback:

[0665] The analyzed emotional state is sent to the server using the requests library. The server analyzes the received data, generates appropriate feedback and advice, and conveys it to the driver via the terminal.

[0666] Specific examples and prompts for the generative AI model

[0667] Examples:

[0668] When a driver launches the EmotionDrive Assistant app and starts driving, the app begins to recognize emotions. If the driver is feeling stressed, the server receives that information and suggests appropriate relaxation methods (such as playing music or instructing breathing exercises) to the device.

[0669] Example prompt for a generative AI model:

[0670] "Design a system in which a smartphone application installed in an autonomous vehicle analyzes the emotional state of the driver in real time and provides appropriate feedback and advice based on that emotional state. The emotion recognition model uses Keras, and the video footage from the webcam is processed in real time."

[0671] This system enables real-time responses that take into account the driver's emotional state, allowing both the driver and passengers to enjoy a safer and more comfortable driving environment.

[0672] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0673] Step 1:

[0674] When the user starts driving, the web camera connected to the device captures the video of the driver's face in real time. The input is the video of the driver's face, and the output is the captured video. This video is used directly for the next step.

[0675] Step 2:

[0676] The device uses OpenCV to detect the driver's face from the captured video. The input of this step is the video captured in step 1, and the output is an image of the area where the face is detected. This face image is used in the next step for facial recognition.

[0677] Step 3:

[0678] The device inputs the facial image into an emotion recognition model trained with Keras to analyze the driver's emotional state. The input is the facial image obtained in step 2, and the output is the emotional state label and its probability. These data are saved for use in the next step.

[0679] Step 4:

[0680] The device uses the requests library to send data containing the analyzed emotional state to the server. The input is the emotional state label and probability, and the output is the emotional state data received by the server. The server receives this data.

[0681] Step 5:

[0682] The server analyzes the received data and generates appropriate feedback or advice. The input of this step is the emotional state data, and the output is the generated feedback or advice. This feedback is used in the next step.

[0683] Step 6:

[0684] The server sends the generated feedback and advice to the terminal. The input is the feedback and advice generated in step 5, and the output is the feedback and advice received by the terminal.

[0685] Step 7:

[0686] The device displays or outputs audio in an appropriate manner to convey the received feedback or advice to the driver. The input is the received feedback or advice, and the output is the actual feedback or advice provided to the driver. Specific actions include displaying the feedback or advice on a screen or announcing it via audio.

[0687] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0688] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0689] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0690] [Third embodiment]

[0691] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0692] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0693] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0694] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0695] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0696] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0697] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0698] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0699] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0700] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0701] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0702] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0703] ---

[0704] The present invention relates to a system that collects data specific to a driver by registering and storing the driver's identification information, and provides safe driving and a comfortable driving environment based on that data.

[0705] System Configuration

[0706] The system of the present invention includes a vehicle terminal used by the user, a cloud server, and an interface for interaction with the driver. The terminal is installed in the vehicle and is responsible for the actual data collection and monitoring. The server analyzes and stores the data for long-term storage. The users are the driver and passengers, and each receives appropriate dialogue and feedback.

[0707] Program and processing flow

[0708] When a user first connects to the system, the server collects basic information about the user, including their name, age, hobbies, preferences, and driving history. Once the user begins driving, the device collects real-time data from the vehicle's sensors and sends it to the server. The server uses this data to analyze the vehicle's condition and driving situation, and provides appropriate information and advice under specific conditions.

[0709] As a specific example, if an engine abnormality is detected, the device immediately sends the abnormality data to the server. The server analyzes the received data and diagnoses the nature of the abnormality. The driver is then notified via the device with an audio alert saying, "An engine abnormality has occurred. Please stop the vehicle in a safe place and request repairs." By responding immediately in this way, serious vehicle breakdowns can be prevented.

[0710] In addition, if the server detects fatigue in a user who has been driving for a long time, it will provide advice such as, "Long-term driving can cause fatigue, so we recommend that you take a break." This can reduce the risk of accidents caused by fatigued driving.

[0711] In addition, sensing devices inside the vehicle can be used to conduct voice interactions with the user and passengers. For example, if a user asks, "Where is the next rest stop?", the device sends this information to a server, which returns data about the nearest rest stop. The server then generates an appropriate response and relays it to the driver via the device.

[0712] When the whole family is in the car, the device can use the microphone and speaker at each seat to enable individual conversations. For example, it can prompt a child in the back seat with a question like, "What book have you read recently?", while asking a parent in the passenger seat, "Are you thinking about planning a family trip?" In this way, everyone can have appropriate topics to talk about, creating a lively atmosphere in the car.

[0713] ---

[0714] The above is an embodiment of the present invention. However, the present invention is not limited to this embodiment and may be modified as appropriate. For example, the type of data collected, the analysis method, and the content of the advice provided can be changed according to the system specifications and the needs of the user.

[0715] The processing flow will be explained below.

[0716] ---

[0717] Step 1:

[0718] The user enters the vehicle and logs in to the in-vehicle terminal. The terminal detects this and sends the user's login information to the server. The server uses this information to authenticate the user's profile and returns the necessary data to the terminal.

[0719] Step 2:

[0720] The device collects real-time data from the vehicle's sensors, including vehicle speed, engine status, fuel level, and interior temperature, and periodically transmits this data to a server.

[0721] Step 3:

[0722] The server receives and analyzes real-time data sent from the device, including engine anomaly detection, fuel level checks, and driver fatigue assessment.

[0723] Step 4:

[0724] Based on the analysis results from the server, the device will provide the driver with appropriate information and advice. For example, if an engine abnormality is detected, the device will issue a voice alert saying, "An engine abnormality has occurred. Please stop in a safe place and request repairs."

[0725] Step 5:

[0726] While driving, the device uses a microphone to receive voice commands from the driver. For example, if the driver asks, "Where is the next rest stop?", the device sends the command to the server. The server analyzes and returns information about the nearest rest stop.

[0727] Step 6:

[0728] Based on the rest stop information returned from the server, the device notifies the driver, "The next rest stop is 5km away." In this way, answers to the driver's questions are provided in real time.

[0729] Step 7:

[0730] The server monitors the driver's fatigue level and generates advice to encourage rest if the driver has been driving for a long time. The device then communicates this to the driver. For example, it may say, "Long-term driving can cause fatigue, so we recommend that you take a break."

[0731] Step 8:

[0732] When the whole family is in the car, the device uses a microphone and speaker installed in each seat to individually target each person with a topic, such as asking a child in the back seat, "What book have you read recently?", or speaking to a parent in the front seat, "Are you thinking about your next family trip?"

[0733] ---

[0734] The above are the processing steps for implementing the present invention. By building a system in accordance with these steps, it is possible to significantly improve the safety and comfort of the driver and passengers.

[0735] Example 1

[0736] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0737] In recent years, there has been a demand for improved safety and comfort in vehicle driving. However, conventional systems are unable to fully utilize individual driver information, making it difficult to provide appropriate advice and warnings in real time. They are also inadequate at immediately detecting and responding to driver fatigue or vehicle abnormalities. Furthermore, their ability to provide information and entertainment tailored to individual needs through two-way dialogue with the driver is limited. There is a need to resolve these issues and improve driver safety and comfort.

[0738] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0739] In this invention, the server includes a means for registering and storing the driver's identification information, a means for collecting data in real time from the vehicle's sensors and monitoring the vehicle's condition and driving situation, and a means for analyzing the collected data and providing the driver with appropriate information and advice. This makes it possible to provide appropriate advice based on the driver's individual information in real time. It also makes it possible to reduce the risk of accidents by detecting the driver's fatigue level and encouraging them to take a break, and to avoid serious breakdowns by immediately notifying the driver of vehicle abnormalities. Furthermore, it is possible to improve the comfort of the driving environment by interacting with the driver and providing information and entertainment based on the driver's hobbies and preferences.

[0740] 1. "Driver identification information" refers to data such as name, age, hobbies and preferences, and driving history used to identify and manage individual drivers.

[0741] 2. "Means for registering and storing" means a technology or device for inputting driver identification information and storing that data in a storage device.

[0742] 3. "Sensor" means a device used to detect and collect data about the vehicle's condition and driving situation. Examples include engine sensors, speed sensors, and cameras.

[0743] 4. "Means for collecting and monitoring data in real time" refers to technology that continuously and instantly collects data obtained from sensors and monitors the vehicle's condition and driving situation.

[0744] 5. "Means for analyzing and providing data" refers to the technology that processes collected data using an analysis program and, as a result, provides useful information and advice to the driver.

[0745] 6. "Means of dialogue" means technology that enables two-way communication with the driver via voice and text.

[0746] 7. "Means for providing conversation and entertainment based on the driver's interests and preferences" refers to technology that provides dialogue and entertainment based on the driver's individual information, according to the driver's interests and preferences.

[0747] 8. "Means for individual topic allocation" refers to technology that provides different conversation content to different passengers in the vehicle, enabling communication that is tailored to their individual interests and situations.

[0748] 9. "Means for issuing an alert when an engine abnormality is detected" refers to technology and devices that monitor the engine condition and issue a warning to the driver when an abnormality is detected.

[0749] 10. "Means for detecting fatigue and providing advice to encourage rest" refers to technology that determines the driver's level of fatigue from continuous driving data and provides information to encourage rest at an appropriate time.

[0750] 11. "Means for analyzing questions based on voice input and providing answers" refers to technology that converts questions entered by the driver through voice into text, analyzes the content of that text, and generates and provides the most appropriate answer.

[0751] The present invention relates to a system that collects data specific to a driver by registering and storing the driver's identification information, and provides safe driving and a comfortable driving environment based on that data.

[0752] System Configuration

[0753] The system of the present invention includes a terminal in the vehicle used by the user, a cloud server, and an interface for interaction with the driver. The terminal is installed in the vehicle and is responsible for the actual data collection and monitoring, while the server analyzes and stores the data for long-term storage.

[0754] Program and process description

[0755] When a user first connects to the system, the server collects basic information about the user, including their name, age, hobbies, preferences, and driving history. Once the user begins driving, the device collects real-time data from the vehicle's sensors and sends it to the server. The server uses this data to analyze the vehicle's condition and driving situation, and provides appropriate information and advice under specific conditions.

[0756] Specifically, the terminal uses the following hardware and software:

[0757] In-vehicle devices: for data collection and monitoring

[0758] Cloud server: for data analysis and long-term storage

[0759] Sensors (e.g. engine sensors, speed sensors, cameras): for real-time data collection and interaction

[0760] The server processes and calculates the collected data as follows:

[0761] 1. Receives data sent from each vehicle sensor and monitors it in real time.

[0762] 2. Use a data analysis program (e.g., Python script) to detect engine anomalies and fatigued driving.

[0763] 3. After detecting an anomaly, appropriate feedback and advice is generated and notified to the user in real time.

[0764] Specific examples

[0765] Examples of anomaly detection:

[0766] Situation: The server detects an abnormality in the engine sensor.

[0767] Processing: The abnormal data is analyzed, and an audio alert is sent to the user via the device stating, "An abnormality has occurred in the engine. Please stop the vehicle in a safe place and request repairs."

[0768] Prompt sentence to input to the generative AI model:

[0769] "An engine anomaly has been detected. Please generate a script to diagnose the anomaly and provide an audio alert to the user."

[0770] Examples of fatigue detection:

[0771] Situation: The server analyzes long-term driving data and estimates fatigue.

[0772] Action: Provide the user with the advice, "Driving for long periods of time can cause fatigue, so we recommend you take a break."

[0773] Prompt sentence to input to the generative AI model:

[0774] "If fatigue due to long driving is detected, generate a script that provides advice to encourage the user to take an appropriate break."

[0775] Examples of voice interactions:

[0776] Situation: A user asks the device, "Where is the next rest stop?"

[0777] Processing: The device sends this question to the server, which responds with information about the nearest rest area. The device then provides this information to the user.

[0778] Prompt sentence to input to the generative AI model:

[0779] "When a user asks, 'Where is the next rest stop?', generate a script that looks up information about the nearest rest stop and responds."

[0780] Examples of interactions with the whole family:

[0781] Situation: There are multiple passengers in the vehicle.

[0782] Processing: The device uses data from each seat's microphone and speaker to facilitate personalized conversations, asking the child in the back seat, "What book have you read recently?" and the parent in the front seat, "Are you thinking about planning a family trip?"

[0783] Prompt sentence to input to the generative AI model:

[0784] "Generate scripts that provide appropriate conversation topics for each passenger in the car. For example, ask the child in the back seat, 'What book have you read recently?' and the parent in the front seat, 'Are you thinking about planning a family trip?'"

[0785] The above is an embodiment of the present invention. This system allows the driver to receive advice and warnings in real time, improving safety and comfort.

[0786] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0787] Step 1:

[0788] Registering and saving user information

[0789] Input: The user enters basic information such as name, age, hobbies, preferences, and driving history into the vehicle's terminal.

[0790] Specific operation: The user enters each piece of information on the device screen.

[0791] Processing: The device sends the entered information to a server in the cloud. After receiving the information, the server stores it in a database and manages it for each user.

[0792] Output: Driver identification stored in a database.

[0793] Step 2:

[0794] Real-time data collection

[0795] Input: Data from each vehicle sensor (engine sensor, speed sensor, camera, etc.).

[0796] Specific operation: The device collects data in real time from sensors installed in the vehicle.

[0797] Processing: The device sends the collected data to a server in the cloud at regular intervals.

[0798] Output: Real-time sensor data sent to the server.

[0799] Step 3:

[0800] Data analysis

[0801] Input: Sensor data sent to the server.

[0802] Specific operation: The server analyzes the data using Python data analysis libraries (such as Pandas and NumPy).

[0803] Processing: Analyzes engine temperature, speed data, etc. to detect abnormalities and estimate driver fatigue.

[0804] Output: Analysis results include the presence or absence of abnormalities and the driver's fatigue state.

[0805] Step 4:

[0806] Generate alerts and advice

[0807] Input: Data analysis results (anomaly detection results, fatigue estimation results).

[0808] Specific behavior: The server generates appropriate alerts and advice based on the analysis results.

[0809] In case of engine abnormality: Generates a message saying "An engine abnormality has occurred. Please stop the vehicle in a safe place and request repairs."

[0810] For fatigue detection: Generate advice such as "Driving for long periods of time can cause fatigue, so we recommend that you take a break."

[0811] Processing: Sends generated alerts and advice to the device.

[0812] Output: Message data for generated alerts and advice.

[0813] Step 5:

[0814] User Notification

[0815] Input: The alert or advice message data sent by the server.

[0816] Specific behavior: The device notifies the user through voice and display.

[0817] In case of engine abnormality: An audio alert will be issued saying, "An engine abnormality has occurred. Please stop in a safe place and request repairs."

[0818] If fatigue is detected: A voice will advise, "Driving for long periods of time can cause fatigue, so we recommend that you take a break."

[0819] Action: Provide real-time alerts and advice to users.

[0820] Output: Proper alerts and advice to the driver.

[0821] Step 6:

[0822] Voice interaction processing

[0823] Input: The user speaks a question to the device, for example, "Where is the next rest stop?"

[0824] Specific operation: The device collects voice data and sends it to a server in the cloud.

[0825] Processing: The server converts the speech to text, analyzes the question, and generates the best answer.

[0826] Example: Generate an answer by getting information about the nearest rest stop.

[0827] Output: Sends the generated answer text data to the terminal.

[0828] Specific operation: The device converts the transmitted response data into voice and provides it to the user.

[0829] Output: The appropriate answer information provided to the user.

[0830] (Application example 1)

[0831] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0832] Autonomous vehicles require efficient registration of driver identification information and the provision of personalized driving experiences. It is also necessary to improve the safety and comfort of drivers and passengers by improving the accuracy of advice provided based on real-time monitoring of driving conditions and data analysis. While linking with various devices and utilizing generative AI models are effective ways to achieve these goals, many aspects of specific implementation remain unresolved.

[0833] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0834] In this invention, the server includes means for registering and storing the driver's identification information, means for collecting data in real time from vehicle sensors and monitoring the vehicle's status and driving situation, means for analyzing the collected data and providing the driver with appropriate information and advice, means for interacting with the driver and providing chat and entertainment based on the driver's hobbies and preferences, means for allocating individual topics to each family member, means for transmitting basic information about the user to a cloud server in the autonomous vehicle and providing analysis results and advice to a smart device in real time, and means for generating appropriate responses to the user's voice commands using a generative AI model. This enables the provision of a personalized driving experience based on the user's identification information, the provision of appropriate advice according to the situation of the autonomous vehicle, and advanced voice responses using the generative AI model.

[0835] "Driver identification information" is a series of information for identifying an individual driver, such as the user's name, age, driving history, etc.

[0836] "Real-time data collection" means using sensors and other input devices to instantly obtain data about the ongoing driving situation and vehicle status.

[0837] "Monitoring vehicle condition and driving status" means monitoring engine status, speed, location information, etc. through various sensors in the vehicle to determine whether it is operating normally.

[0838] "Analyzing collected data" means using algorithms and analytical tools based on the acquired data to detect vehicle abnormalities and the driver's condition.

[0839] "Providing appropriate information and advice" means providing drivers with advice for safe driving and comfortable driving based on the analysis results.

[0840] "Dialogue with the driver" means two-way communication with the driver using means such as voice recognition or text input.

[0841] "Providing chat and entertainment based on the driver's hobbies and preferences" means providing topics, music, and other entertainment content that meet the driver's individual preferences.

[0842] "Allocating individual topics to each family member" means bringing up topics that correspond to the interests and concerns of each passenger in the car and conducting individual conversations.

[0843] "Send to a server on the cloud" means uploading data to a remote server via the Internet.

[0844] A "generative AI model" is an algorithm based on machine learning and deep learning that is used to perform advanced tasks such as speech recognition and text generation.

[0845] "Generating an appropriate response to a voice command" means recognizing a voice instruction given by the user and providing an appropriate reaction or information in response to it.

[0846] "Smart devices" is a general term for mobile devices that can connect to the Internet, such as smartphones, tablets, and smart glasses.

[0847] "Personalized driving experience" refers to providing a driving environment and services that are customized based on a user's individual identity and preferences.

[0848] This invention is a system for enhancing the driver-specific experience in autonomous vehicles. The system registers the driver's identification information, monitors the vehicle's status and driving situation in real time, analyzes the collected data, and provides appropriate information and advice. Furthermore, it can use a generative AI model to interact with the driver and provide personalized conversations and entertainment.

[0849] The server first registers the driver's identification information in a database on the cloud. This information includes the user's name, age, driving history, etc. In addition, a terminal installed in the vehicle uses various sensors to collect vehicle status (engine status, speed, location information, etc.) in real time. This data is immediately sent to the server, which then analyzes it.

[0850] Based on the analysis results, the server generates advice and alerts appropriate to the situation and provides them to the driver via a smart device (such as a smartphone or smart glasses). For example, if an engine abnormality is detected, the server will issue an alert stating, "An engine abnormality has occurred. Please stop the vehicle in a safe place and request repairs."

[0851] Additionally, if the server detects that the driver is fatigued, it will provide advice such as, "Long-term driving can cause fatigue, so we recommend you take a break," ensuring the driver's safety. Furthermore, when the driver issues a voice command using the generative AI model, the server generates an appropriate response based on the analysis results and provides it to the driver.

[0852] For example, if a user asks, "Where is the next rest stop?", the smart device sends this information to the server, which uses data about the nearest rest stops to generate a response such as "There is a rest stop XX kilometers away."

[0853] Here is an example prompt:

[0854] "The user wants to know the nearest rest area from their current location. Their current location is [latitude, longitude]. What is the next available rest area?"

[0855] The hardware used includes smartphones, smart glasses, and autonomous vehicle sensors (cameras, GPS, microphones, etc.), while the software used includes cloud servers (AWS, Google Cloud), machine learning models (TensorFlow, PyTorch), and real-time data analysis (Apache Kafka).

[0856] The system will provide a personalized driving experience based on user identity, monitor vehicle status in real time and provide appropriate advice, and use generative AI models to provide advanced voice responses.

[0857] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0858] Step 1:

[0859] When a user gets into a vehicle, the device acquires the user's identification information and sends it to the cloud server. Basic information such as the user's name, age, and driving history is received as input, and based on this information, a new registration or update is made on the cloud. A user ID is generated as output.

[0860] Step 2:

[0861] The terminal collects data in real time from various sensors in the vehicle (engine status, speed, location information, etc.) and sends the collected data to a cloud server. The input includes signal data from each sensor. The server receives this and stores it in a database. The output is accumulated data.

[0862] Step 3:

[0863] The server analyzes the collected data and diagnoses the driver's condition. The stored sensor data is used as input. A machine learning model (e.g., TensorFlow, PyTorch) is used for data analysis to detect signs and abnormalities for safe driving. The analysis results are generated as output.

[0864] Step 4:

[0865] Based on the analysis results, the server automatically generates appropriate advice or alerts and notifies the driver via the terminal. The analysis results are used as input. For example, a message such as "There is an engine malfunction. Please stop the vehicle in a safe place and request repairs" is generated. As output, a voice or text notification is sent to the driver.

[0866] Step 5:

[0867] The server detects the driver's fatigue state from sensor data and provides advice such as, "You are showing signs of fatigue. We recommend that you take a break." Input includes vital data such as heart rate and reaction time. A machine learning model is used to analyze this data and detect signs of fatigue. The output is a notification encouraging the driver to take a break.

[0868] Step 6:

[0869] When a driver inputs a voice command via a smart device (smartphone or smart glasses), the device sends the information to the server. The user's voice command is used as input. Example prompt: "Where is the next rest stop?" The server uses a generative AI model to generate an appropriate response and returns it to the user. Information about rest stops is returned as output.

[0870] Step 7:

[0871] As part of the response generation process, the server performs database searches and calculations in the cloud. The input includes a prompt to the generative AI model. For example, the prompt might read, "The user wants to know the nearest rest area from their current location. Their current location is [latitude, longitude]. Please tell me where the next available rest area is." The server retrieves data on the nearest rest areas and generates an appropriate answer. The output is a response containing the location of the rest area.

[0872] These processing steps enable the system of the present invention to effectively collect and analyze driver-specific data to provide a safe and comfortable driving experience.

[0873] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0874] ---

[0875] This invention achieves even greater driving safety and comfort by collecting driver-specific data by registering and storing the driver's identification information, and combining this data with an emotion engine in a system that provides safe driving and a comfortable driving environment.

[0876] System Configuration

[0877] The system of the present invention includes a vehicle terminal used by the user, a cloud server, an interface for interacting with the driver, and an emotion engine. The terminal is installed in the vehicle and is responsible for the actual data collection and monitoring. The server analyzes and stores the data for long-term storage. The emotion engine has the function of recognizing the user's emotions and generating appropriate dialogue and responses based on those emotions. The users are the driver and passengers, and each receives dialogue and feedback tailored to their needs.

[0878] Program and processing flow

[0879] When a user first connects to the system, the server collects basic information about the user, including their name, age, hobbies, preferences, and driving history. Once the user begins driving, the device collects real-time data from the vehicle's sensors and sends it to the server. The server uses this data to analyze the vehicle's condition and driving situation, and provides appropriate information and advice under specific conditions.

[0880] As a specific example, if an engine abnormality is detected, the device immediately sends the abnormality data to the server. The server analyzes the received data and diagnoses the nature of the abnormality. The driver is then notified via the device with an audio alert saying, "An engine abnormality has occurred. Please stop the vehicle in a safe place and request repairs." By responding immediately in this way, serious vehicle breakdowns can be prevented.

[0881] The emotion engine also analyzes the user's emotions in real time and responds accordingly based on their emotional state. If the driver is feeling stressed, the emotion engine detects this and sends it to the server. The server then selects appropriate advice to reduce stress and music to help them relax, and provides this to the driver via the device. For example, if the emotion engine detects that the driver is stressed, it will notify the driver, "You seem to be feeling stressed at the moment. Would you like me to play some music of your choice to help you relax?"

[0882] Furthermore, when the whole family is in the car, the device uses microphones and speakers installed in each seat to assign individual topics to talk about. The emotion engine recognizes the emotional state of each member and provides appropriate dialogue. For example, if a child in the back seat looks sad, the device will ask, "What's wrong? Is there anything you'd like to talk about?" On the other hand, it will proactively provoke a conversation with the parent in the passenger seat by asking, "Why don't you think about your next family trip?"

[0883] This system realizes appropriate dialogue that takes into account the emotional states of the driver and passengers, providing a safer and more comfortable driving environment. The system provides real-time feedback according to the user's emotional state, not only improving driver safety but also creating a lively atmosphere inside the vehicle.

[0884] ---

[0885] The above is an embodiment of the present invention. By constructing a system according to this embodiment, it is possible to provide a comfortable and safe driving environment that takes into account the emotional states of the driver and passengers.

[0886] The processing flow will be explained below.

[0887] ---

[0888] Step 1:

[0889] The user enters the vehicle and logs in to the in-vehicle terminal. The terminal detects this and sends the user's login information to the server. The server uses this information to authenticate the user's profile and returns the necessary data to the terminal.

[0890] Step 2:

[0891] The device collects real-time data from the vehicle's sensors, including vehicle speed, engine status, fuel level, and interior temperature, and periodically transmits this data to a server.

[0892] Step 3:

[0893] The server receives and analyzes real-time data sent from the device, including engine anomaly detection, fuel level checks, and driver fatigue assessment.

[0894] Step 4:

[0895] The device uses the in-car camera and microphone to detect the user's facial expressions and tone of voice, and the emotion engine analyzes this data to identify the user's emotional state.

[0896] Step 5:

[0897] The server receives the emotion data sent from the emotion engine and provides appropriate information and advice to the driver based on the analysis results. For example, if the driver is under stress, the server will generate advice such as "Shall I play your favorite music to help you relax?" and provide it via the device.

[0898] Step 6:

[0899] While driving, the device uses a microphone to receive voice commands from the driver. For example, if the driver asks, "Where is the next rest stop?", the device sends the command to the server. The server analyzes and returns information about the nearest rest stop.

[0900] Step 7:

[0901] Based on the rest stop information returned from the server, the device notifies the driver, "The next rest stop is 5km away." In this way, answers to the driver's questions are provided in real time.

[0902] Step 8:

[0903] The server monitors the driver's fatigue level and generates advice to encourage rest if the driver has been driving for a long time. The device then communicates this to the driver. For example, it may say, "Long-term driving can cause fatigue, so we recommend that you take a break."

[0904] Step 9:

[0905] When the whole family is in the car, the device uses a microphone and speaker installed in each seat to individually assign topics to each member. The emotion engine recognizes the emotional state of each member and provides appropriate dialogue. For example, if a child in the back seat looks sad, the device will ask, "What's wrong? Is there anything you'd like to talk about?" On the other hand, it will proactively provoke a conversation with the parent in the passenger seat by asking, "Why don't you think about your next family trip?"

[0906] ---

[0907] The above are the processing steps for implementing the present invention. By building a system in accordance with these steps, it is possible to provide appropriate feedback in real time according to the emotional state of the driver and passengers, thereby realizing a safe and comfortable driving environment.

[0908] Example 2

[0909] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0910] Conventional vehicle driving assistance systems have limitations in providing safe driving and a comfortable driving environment for drivers, particularly in systems that take into account the driver's emotional state and passenger comfort. This can increase stress and discomfort while driving, potentially jeopardizing safe driving. Furthermore, the lack of a mechanism for providing individual conversations when the whole family is in the car makes it difficult to improve the in-car atmosphere. There is a need to address these issues and improve the overall driving experience for drivers and passengers.

[0911] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for registering and saving the driver's identification information, means for collecting data in real time from vehicle sensors and monitoring the vehicle state and driving situation, means for analyzing the collected data and providing the driver with appropriate information and advice, means for analyzing the emotional states of the driver and passengers using an emotion recognition engine, means for providing dialogue and entertainment based on the emotional states, means for allocating individual topics to each family member, and means for providing a method for the driver to relax when feeling stressed. This makes it possible to provide appropriate feedback and entertainment that takes into account the emotional states of the driver and passengers, thereby improving driving safety and comfort.

[0912] "Driver identification information" is information for identifying a driver, and includes the name, age, hobbies and preferences, driving history, and the like.

[0913] A "sensor" is a device for detecting various vehicle conditions and driving situations in real time, and includes a speed sensor, a brake sensor, an engine sensor, and the like.

[0914] "Real-time data" refers to instantaneous data collected from vehicle sensors during driving, including speed, braking status, engine temperature, and the like.

[0915] "Analysis" is the process of evaluating the driving situation and vehicle condition based on collected data and identifying problems and areas for improvement.

[0916] "Information and advice" refers to notifications and instructions provided to the driver based on the analysis results, including advice on safe driving and warnings of abnormalities.

[0917] An "emotion recognition engine" is software that analyzes the voice and facial expression data of the driver and passengers to determine their emotional state.

[0918] "Entertainment" refers to entertainment content provided to entertain the driver and passengers, and includes music, chat, games, and the like.

[0919] "Feedback" refers to reactions and responses provided to the driver and passengers based on analytical results and emotional state, and may take the form of advice or entertainment.

[0920] "All family members" refers to all passengers in the vehicle, and may include the driver.

[0921] "Individual topic allocation" means providing each passenger with an appropriate topic based on their interests and emotional state.

[0922] "Stress" refers to the mental strain or tension a driver feels while driving, and is detected by an emotion recognition engine.

[0923] "Relaxation methods" are means to reduce stress and relax the driver, and examples include music and suggestions for deep breathing.

[0924] System program generation

[0925] The server generates the system's program, which includes a series of processes for registering driver identification information, collecting real-time data, analyzing data, providing feedback, analyzing emotions, and conducting interactive dialogue.

[0926] Hardware and software used

[0927] Hardware: Devices installed in the vehicle (speed sensor, brake sensor, engine sensor, microphone, camera, speaker), cloud server

[0928] Software: Data analysis software, emotion recognition engine, interface application, cloud storage system

[0929] Program processing (natural language explanation)

[0930] Registering user information

[0931] The server collects basic information about the user when the user first connects to the system, including name, age, hobbies, driving history, etc. When the user logs in to the application, a screen for entering personal information is displayed, and the entered information is saved on the server.

[0932] Real-time data collection

[0933] The terminal collects real-time data from various sensors in the vehicle, including speed, braking status, engine temperature, and audio and video data from inside the vehicle. As soon as the driver starts driving, the sensors start operating and transmit the data to the terminal.

[0934] Data transmission and analysis

[0935] The device sends the collected data to a server in real time. The server analyzes the received data and evaluates the vehicle's condition and driving situation. If the speed data exceeds a certain range or abnormal data is detected from the engine, the server immediately detects the abnormality and performs analysis.

[0936] Providing Feedback

[0937] The server generates appropriate feedback based on the analysis results. The generated feedback is notified to the user via the terminal. For example, if an engine abnormality is detected, the server generates a message saying, "An engine abnormality has occurred. Please stop the vehicle in a safe place and request repairs," and sends it to the terminal. The terminal notifies the driver of this message as an audio alert.

[0938] Emotion analysis

[0939] The device uses a microphone and camera installed in the vehicle to collect the user's voice and facial expression data. The emotion recognition engine analyzes the user's emotions in real time based on the collected data. If the driver is feeling stressed, the emotion recognition engine detects this and sends the analysis results to the server.

[0940] Interactive Dialogue

[0941] The server and emotion recognition engine provide appropriate dialogue and entertainment based on the analysis results. For example, if the driver is feeling stressed, the device will notify them, "Your stress level seems to be increasing. Would you like me to play some relaxing music?" If a child in the back seat looks sad, the device will ask, "What's wrong? Is there anything you'd like to talk to me about?"

[0942] Examples and prompts

[0943] As a concrete example, consider a scenario where an engine malfunction occurs. The device detects the engine malfunction and sends the data to the server. The server analyzes the received data and notifies the driver with a voice alert saying, "An engine malfunction has occurred. Please stop the vehicle in a safe place and request repairs."

[0944] Example prompt sentence:

[0945] "Describe a program that uses previous driving data to generate safe driving advice."

[0946] As a process based on this prompt, the server provides the data analysis results as feedback and gives advice to the driver to encourage safe driving.

[0947] The above is an embodiment of the invention, which takes into consideration the emotional states of the driver and passengers and provides a safe and comfortable driving environment.

[0948] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0949] Step 1:

[0950] The server collects basic user information when the user first connects to the system, including name, age, hobbies, and driving history. The information is stored on the server and serves as the basis for future analysis and interactive dialogue.

[0951] Step 2:

[0952] The terminal collects data in real time from various sensors in the vehicle. Input data includes speed, braking status, engine temperature, and audio and video data inside the vehicle. The collected data is temporarily stored in the terminal and sent to the server as appropriate. This allows the server to manage detailed driving conditions.

[0953] Step 3:

[0954] The terminal sends the collected real-time data to a server. The input data is the output data from the vehicle's sensors, and once sent to the server, the data is analyzed immediately. The server analyzes the data and evaluates the driving situation and vehicle condition.

[0955] Step 4:

[0956] The server generates appropriate feedback based on the analysis results. The input data are the sensor data and analysis results sent to the server, and the output data is a feedback message for the driver. For example, the server generates a message saying, "An abnormality has occurred in the engine. Please stop the vehicle in a safe place and request repairs," and sends it to the terminal. The terminal notifies the driver of this as an audio alert.

[0957] Step 5:

[0958] The device uses the in-car microphone and camera to collect the user's voice and facial expression data. The input data is the voice and video from inside the car, and is sent to the emotion recognition engine. The emotion recognition engine analyzes this data and determines the user's emotional state.

[0959] Step 6:

[0960] The emotion recognition engine sends the result to the server based on the emotion analysis result. The input data is the analysis result by the emotion recognition engine, and the output data is the feedback of the user's emotional state to the server. If the driver is feeling stressed, the emotion recognition engine will detect this and provide the result to the server.

[0961] Step 7:

[0962] The server generates appropriate response feedback and entertainment based on the emotion analysis results. The input data is the emotional state analysis result from the emotion recognition engine, and the output data is voice feedback and entertainment content for the driver and passengers. For example, if the driver is feeling stressed, the server sends a message to the terminal saying, "Would you like me to play some relaxing music?" The terminal notifies the driver of this message by voice, and if the driver agrees, it plays relaxing music.

[0963] The above is a detailed explanation of the processing steps of the program of this system. At each step, the specific data processing and data calculation processes based on the input data are described in detail.

[0964] (Application example 2)

[0965] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0966] The purpose of this invention is to improve driving safety and comfort by analyzing the driver's emotional state in real time and providing appropriate feedback and advice based on that information. Conventional systems have difficulty responding appropriately to the driver's emotional state, and have been unable to effectively reduce stress and fatigue while driving. In addition, they have been unable to individually allocate topics of conversation, making it difficult for the whole family to enjoy a comfortable driving environment.

[0967] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting data from vehicle sensors in real time and analyzing the emotional state of the driver, means for providing appropriate feedback and advice to the driver based on the collected data, and means for suggesting music or relaxation methods based on the driver's emotional state. This makes it possible to respond in real time while taking the driver's emotional state into consideration, allowing the driver and all passengers to enjoy a safer and more comfortable driving environment.

[0968] "Driver identification information" includes individual information such as the driver's name, age, hobbies and preferences, and driving history.

[0969] "Vehicle sensors" refers to various sensor devices used to monitor the vehicle's condition and driving conditions in real time.

[0970] "Means for collecting data in real time" refers to technical devices and software for continuously collecting data from sensors installed in the vehicle.

[0971] "Means for analyzing collected data" refers to algorithms or systems for evaluating the vehicle condition and driving situation based on collected data and processing the results.

[0972] "Means for providing appropriate information and advice to drivers" refers to devices and systems for providing necessary information and driving advice to drivers.

[0973] "Means for communicating with the driver" refers to a voice recognition system or interface for directly communicating with the driver.

[0974] "Means for providing chat and entertainment based on the driver's interests and preferences" refers to a content generation system or interface for providing chat and entertainment based on the driver's interests and preferences.

[0975] "Means for individually allocating topics to each family member" refers to a dialogue system or interface that provides appropriate topics to each person in the car.

[0976] "Emotion recognition" refers to technology that analyzes a driver's emotional state based on data such as facial expressions and voice.

[0977] "Means for suggesting music or relaxation techniques based on the driver's emotional state" refers to a system that takes into account the driver's emotional state and provides appropriate music or relaxation techniques.

[0978] The Emotion Drive Assistant system of the present invention analyzes the emotional state of the driver in real time and provides appropriate feedback and advice based on that information, thereby improving driving safety and comfort. Hereinafter, an embodiment of the present invention will be described in detail.

[0979] System Configuration

[0980] The system includes a terminal in the vehicle used by the driver, a cloud server, an interface for interacting with the driver, and an emotion engine. The terminal is installed in the vehicle and is responsible for the actual data collection and monitoring, while the server analyzes and stores the data for long-term storage.

[0981] Program and processing flow

[0982] Hardware and Software

[0983] The main hardware used in this system includes:

[0984] Webcam: A camera that captures the driver's face in real time and collects data.

[0985] Smartphone: Serves as the control and interface for the entire system.

[0986] Sensors built into the vehicle: Various sensors to collect information such as speed, acceleration, engine status, etc.

[0987] The software used includes:

[0988] OpenCV: A library for processing webcam footage and performing face detection.

[0989] Keras: A machine learning library for running emotion recognition models.

[0990] requests: An HTTP library for communicating with the server.

[0991] Data processing and calculation

[0992] 1. Data Collection:

[0993] When the user starts driving, a web camera connected to the device captures video of the driver's face in real time, and various vehicle sensors collect driving situation data.

[0994] 2. Face detection and emotion recognition:

[0995] We use OpenCV to detect the driver's face from the captured video, and then input the detected face image into an emotion recognition model trained with Keras to analyze the driver's emotional state.

[0996] 3. Submitting Information and Feedback:

[0997] The analyzed emotional state is sent to the server using the requests library. The server analyzes the received data, generates appropriate feedback and advice, and conveys it to the driver via the terminal.

[0998] Specific examples and prompts for the generative AI model

[0999] Examples:

[1000] When a driver launches the EmotionDrive Assistant app and starts driving, the app begins to recognize emotions. If the driver is feeling stressed, the server receives that information and suggests appropriate relaxation methods (such as playing music or instructing breathing exercises) to the device.

[1001] Example prompt for a generative AI model:

[1002] "Design a system in which a smartphone application installed in an autonomous vehicle analyzes the emotional state of the driver in real time and provides appropriate feedback and advice based on that emotional state. The emotion recognition model uses Keras, and the video footage from the webcam is processed in real time."

[1003] This system enables real-time responses that take into account the driver's emotional state, allowing both the driver and passengers to enjoy a safer and more comfortable driving environment.

[1004] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1005] Step 1:

[1006] When the user starts driving, the web camera connected to the device captures the video of the driver's face in real time. The input is the video of the driver's face, and the output is the captured video. This video is used directly for the next step.

[1007] Step 2:

[1008] The device uses OpenCV to detect the driver's face from the captured video. The input of this step is the video captured in step 1, and the output is an image of the area where the face is detected. This face image is used in the next step for facial recognition.

[1009] Step 3:

[1010] The device inputs the facial image into an emotion recognition model trained with Keras to analyze the driver's emotional state. The input is the facial image obtained in step 2, and the output is the emotional state label and its probability. These data are saved for use in the next step.

[1011] Step 4:

[1012] The device uses the requests library to send data containing the analyzed emotional state to the server. The input is the emotional state label and probability, and the output is the emotional state data received by the server. The server receives this data.

[1013] Step 5:

[1014] The server analyzes the received data and generates appropriate feedback or advice. The input of this step is the emotional state data, and the output is the generated feedback or advice. This feedback is used in the next step.

[1015] Step 6:

[1016] The server sends the generated feedback and advice to the terminal. The input is the feedback and advice generated in step 5, and the output is the feedback and advice received by the terminal.

[1017] Step 7:

[1018] The device displays or outputs audio in an appropriate manner to convey the received feedback or advice to the driver. The input is the received feedback or advice, and the output is the actual feedback or advice provided to the driver. Specific actions include displaying the feedback or advice on a screen or announcing it via audio.

[1019] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1020] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1021] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1022] [Fourth embodiment]

[1023] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1024] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1026] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1027] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1028] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1030] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1031] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1032] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1034] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1036] ---

[1037] The present invention relates to a system that collects data specific to a driver by registering and storing the driver's identification information, and provides safe driving and a comfortable driving environment based on that data.

[1038] System Configuration

[1039] The system of the present invention includes a vehicle terminal used by the user, a cloud server, and an interface for interaction with the driver. The terminal is installed in the vehicle and is responsible for the actual data collection and monitoring. The server analyzes and stores the data for long-term storage. The users are the driver and passengers, and each receives appropriate dialogue and feedback.

[1040] Program and processing flow

[1041] When a user first connects to the system, the server collects basic information about the user, including their name, age, hobbies, preferences, and driving history. Once the user begins driving, the device collects real-time data from the vehicle's sensors and sends it to the server. The server uses this data to analyze the vehicle's condition and driving situation, and provides appropriate information and advice under specific conditions.

[1042] As a specific example, if an engine abnormality is detected, the device immediately sends the abnormality data to the server. The server analyzes the received data and diagnoses the nature of the abnormality. The driver is then notified via the device with an audio alert saying, "An engine abnormality has occurred. Please stop the vehicle in a safe place and request repairs." By responding immediately in this way, serious vehicle breakdowns can be prevented.

[1043] In addition, if the server detects fatigue in a user who has been driving for a long time, it will provide advice such as, "Long-term driving can cause fatigue, so we recommend that you take a break." This can reduce the risk of accidents caused by fatigued driving.

[1044] In addition, sensing devices inside the vehicle can be used to conduct voice interactions with the user and passengers. For example, if a user asks, "Where is the next rest stop?", the device sends this information to a server, which returns data about the nearest rest stop. The server then generates an appropriate response and relays it to the driver via the device.

[1045] When the whole family is in the car, the device can use the microphone and speaker at each seat to enable individual conversations. For example, it can prompt a child in the back seat with a question like, "What book have you read recently?", while asking a parent in the passenger seat, "Are you thinking about planning a family trip?" In this way, everyone can have appropriate topics to talk about, creating a lively atmosphere in the car.

[1046] ---

[1047] The above is an embodiment of the present invention. However, the present invention is not limited to this embodiment and may be modified as appropriate. For example, the type of data collected, the analysis method, and the content of the advice provided can be changed according to the system specifications and the needs of the user.

[1048] The processing flow will be explained below.

[1049] ---

[1050] Step 1:

[1051] The user enters the vehicle and logs in to the in-vehicle terminal. The terminal detects this and sends the user's login information to the server. The server uses this information to authenticate the user's profile and returns the necessary data to the terminal.

[1052] Step 2:

[1053] The device collects real-time data from the vehicle's sensors, including vehicle speed, engine status, fuel level, and interior temperature, and periodically transmits this data to a server.

[1054] Step 3:

[1055] The server receives and analyzes real-time data sent from the device, including engine anomaly detection, fuel level checks, and driver fatigue assessment.

[1056] Step 4:

[1057] Based on the analysis results from the server, the device will provide the driver with appropriate information and advice. For example, if an engine abnormality is detected, the device will issue a voice alert saying, "An engine abnormality has occurred. Please stop in a safe place and request repairs."

[1058] Step 5:

[1059] While driving, the device uses a microphone to receive voice commands from the driver. For example, if the driver asks, "Where is the next rest stop?", the device sends the command to the server. The server analyzes and returns information about the nearest rest stop.

[1060] Step 6:

[1061] Based on the rest stop information returned from the server, the device notifies the driver, "The next rest stop is 5km away." In this way, answers to the driver's questions are provided in real time.

[1062] Step 7:

[1063] The server monitors the driver's fatigue level and generates advice to encourage rest if the driver has been driving for a long time. The device then communicates this to the driver. For example, it may say, "Long-term driving can cause fatigue, so we recommend that you take a break."

[1064] Step 8:

[1065] When the whole family is in the car, the device uses a microphone and speaker installed in each seat to individually target each person with a topic, such as asking a child in the back seat, "What book have you read recently?", or speaking to a parent in the front seat, "Are you thinking about your next family trip?"

[1066] ---

[1067] The above are the processing steps for implementing the present invention. By building a system in accordance with these steps, it is possible to significantly improve the safety and comfort of the driver and passengers.

[1068] Example 1

[1069] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1070] In recent years, there has been a demand for improved safety and comfort in vehicle driving. However, conventional systems are unable to fully utilize individual driver information, making it difficult to provide appropriate advice and warnings in real time. They are also inadequate at immediately detecting and responding to driver fatigue or vehicle abnormalities. Furthermore, their ability to provide information and entertainment tailored to individual needs through two-way dialogue with the driver is limited. There is a need to resolve these issues and improve driver safety and comfort.

[1071] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1072] In this invention, the server includes a means for registering and storing the driver's identification information, a means for collecting data in real time from the vehicle's sensors and monitoring the vehicle's condition and driving situation, and a means for analyzing the collected data and providing the driver with appropriate information and advice. This makes it possible to provide appropriate advice based on the driver's individual information in real time. It also makes it possible to reduce the risk of accidents by detecting the driver's fatigue level and encouraging them to take a break, and to avoid serious breakdowns by immediately notifying the driver of vehicle abnormalities. Furthermore, it is possible to improve the comfort of the driving environment by interacting with the driver and providing information and entertainment based on the driver's hobbies and preferences.

[1073] 1. "Driver identification information" refers to data such as name, age, hobbies and preferences, and driving history used to identify and manage individual drivers.

[1074] 2. "Means for registering and storing" means a technology or device for inputting driver identification information and storing that data in a storage device.

[1075] 3. "Sensor" means a device used to detect and collect data about the vehicle's condition and driving situation. Examples include engine sensors, speed sensors, and cameras.

[1076] 4. "Means for collecting and monitoring data in real time" refers to technology that continuously and instantly collects data obtained from sensors and monitors the vehicle's condition and driving situation.

[1077] 5. "Means for analyzing and providing data" refers to the technology that processes collected data using an analysis program and, as a result, provides useful information and advice to the driver.

[1078] 6. "Means of dialogue" means technology that enables two-way communication with the driver via voice and text.

[1079] 7. "Means for providing conversation and entertainment based on the driver's interests and preferences" refers to technology that provides dialogue and entertainment based on the driver's individual information, according to the driver's interests and preferences.

[1080] 8. "Means for individual topic allocation" refers to technology that provides different conversation content to different passengers in the vehicle, enabling communication that is tailored to their individual interests and situations.

[1081] 9. "Means for issuing an alert when an engine abnormality is detected" refers to technology and devices that monitor the engine condition and issue a warning to the driver when an abnormality is detected.

[1082] 10. "Means for detecting fatigue and providing advice to encourage rest" refers to technology that determines the driver's level of fatigue from continuous driving data and provides information to encourage rest at an appropriate time.

[1083] 11. "Means for analyzing questions based on voice input and providing answers" refers to technology that converts questions entered by the driver through voice into text, analyzes the content of that text, and generates and provides the most appropriate answer.

[1084] The present invention relates to a system that collects data specific to a driver by registering and storing the driver's identification information, and provides safe driving and a comfortable driving environment based on that data.

[1085] System Configuration

[1086] The system of the present invention includes a terminal in the vehicle used by the user, a cloud server, and an interface for interaction with the driver. The terminal is installed in the vehicle and is responsible for the actual data collection and monitoring, while the server analyzes and stores the data for long-term storage.

[1087] Program and process description

[1088] When a user first connects to the system, the server collects basic information about the user, including their name, age, hobbies, preferences, and driving history. Once the user begins driving, the device collects real-time data from the vehicle's sensors and sends it to the server. The server uses this data to analyze the vehicle's condition and driving situation, and provides appropriate information and advice under specific conditions.

[1089] Specifically, the terminal uses the following hardware and software:

[1090] In-vehicle devices: for data collection and monitoring

[1091] Cloud server: for data analysis and long-term storage

[1092] Sensors (e.g. engine sensors, speed sensors, cameras): for real-time data collection and interaction

[1093] The server processes and calculates the collected data as follows:

[1094] 1. Receives data sent from each vehicle sensor and monitors it in real time.

[1095] 2. Use a data analysis program (e.g., Python script) to detect engine anomalies and fatigued driving.

[1096] 3. After detecting an anomaly, appropriate feedback and advice is generated and notified to the user in real time.

[1097] Specific examples

[1098] Examples of anomaly detection:

[1099] Situation: The server detects an abnormality in the engine sensor.

[1100] Processing: The abnormal data is analyzed, and an audio alert is sent to the user via the device stating, "An abnormality has occurred in the engine. Please stop the vehicle in a safe place and request repairs."

[1101] Prompt sentence to input to the generative AI model:

[1102] "An engine anomaly has been detected. Please generate a script to diagnose the anomaly and provide an audio alert to the user."

[1103] Examples of fatigue detection:

[1104] Situation: The server analyzes long-term driving data and estimates fatigue.

[1105] Action: Provide the user with the advice, "Driving for long periods of time can cause fatigue, so we recommend you take a break."

[1106] Prompt sentence to input to the generative AI model:

[1107] "If fatigue due to long driving is detected, generate a script that provides advice to encourage the user to take an appropriate break."

[1108] Examples of voice interactions:

[1109] Situation: A user asks the device, "Where is the next rest stop?"

[1110] Processing: The device sends this question to the server, which responds with information about the nearest rest area. The device then provides this information to the user.

[1111] Prompt sentence to input to the generative AI model:

[1112] "When a user asks, 'Where is the next rest stop?', generate a script that looks up information about the nearest rest stop and responds."

[1113] Examples of interactions with the whole family:

[1114] Situation: There are multiple passengers in the vehicle.

[1115] Processing: The device uses data from each seat's microphone and speaker to facilitate personalized conversations, asking the child in the back seat, "What book have you read recently?" and the parent in the front seat, "Are you thinking about planning a family trip?"

[1116] Prompt sentence to input to the generative AI model:

[1117] "Generate scripts that provide appropriate conversation topics for each passenger in the car. For example, ask the child in the back seat, 'What book have you read recently?' and the parent in the front seat, 'Are you thinking about planning a family trip?'"

[1118] The above is an embodiment of the present invention. This system allows the driver to receive advice and warnings in real time, improving safety and comfort.

[1119] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1120] Step 1:

[1121] Registering and saving user information

[1122] Input: The user enters basic information such as name, age, hobbies, preferences, and driving history into the vehicle's terminal.

[1123] Specific operation: The user enters each piece of information on the device screen.

[1124] Processing: The device sends the entered information to a server in the cloud. After receiving the information, the server stores it in a database and manages it for each user.

[1125] Output: Driver identification stored in a database.

[1126] Step 2:

[1127] Real-time data collection

[1128] Input: Data from each vehicle sensor (engine sensor, speed sensor, camera, etc.).

[1129] Specific operation: The device collects data in real time from sensors installed in the vehicle.

[1130] Processing: The device sends the collected data to a server in the cloud at regular intervals.

[1131] Output: Real-time sensor data sent to the server.

[1132] Step 3:

[1133] Data analysis

[1134] Input: Sensor data sent to the server.

[1135] Specific operation: The server analyzes the data using Python data analysis libraries (such as Pandas and NumPy).

[1136] Processing: Analyzes engine temperature, speed data, etc. to detect abnormalities and estimate driver fatigue.

[1137] Output: Analysis results include the presence or absence of abnormalities and the driver's fatigue state.

[1138] Step 4:

[1139] Generate alerts and advice

[1140] Input: Data analysis results (anomaly detection results, fatigue estimation results).

[1141] Specific behavior: The server generates appropriate alerts and advice based on the analysis results.

[1142] In case of engine abnormality: Generates a message saying "An engine abnormality has occurred. Please stop the vehicle in a safe place and request repairs."

[1143] For fatigue detection: Generate advice such as "Driving for long periods of time can cause fatigue, so we recommend that you take a break."

[1144] Processing: Sends generated alerts and advice to the device.

[1145] Output: Message data for generated alerts and advice.

[1146] Step 5:

[1147] User Notification

[1148] Input: The alert or advice message data sent by the server.

[1149] Specific behavior: The device notifies the user through voice and display.

[1150] In case of engine abnormality: An audio alert will be issued saying, "An engine abnormality has occurred. Please stop in a safe place and request repairs."

[1151] If fatigue is detected: A voice will advise, "Driving for long periods of time can cause fatigue, so we recommend that you take a break."

[1152] Action: Provide real-time alerts and advice to users.

[1153] Output: Proper alerts and advice to the driver.

[1154] Step 6:

[1155] Voice interaction processing

[1156] Input: The user speaks a question to the device, for example, "Where is the next rest stop?"

[1157] Specific operation: The device collects voice data and sends it to a server in the cloud.

[1158] Processing: The server converts the speech to text, analyzes the question, and generates the best answer.

[1159] Example: Generate an answer by getting information about the nearest rest stop.

[1160] Output: Sends the generated answer text data to the terminal.

[1161] Specific operation: The device converts the transmitted response data into voice and provides it to the user.

[1162] Output: The appropriate answer information provided to the user.

[1163] (Application example 1)

[1164] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1165] Autonomous vehicles require efficient registration of driver identification information and the provision of personalized driving experiences. It is also necessary to improve the safety and comfort of drivers and passengers by improving the accuracy of advice provided based on real-time monitoring of driving conditions and data analysis. While linking with various devices and utilizing generative AI models are effective ways to achieve these goals, many aspects of specific implementation remain unresolved.

[1166] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1167] In this invention, the server includes means for registering and storing the driver's identification information, means for collecting data in real time from vehicle sensors and monitoring the vehicle's status and driving situation, means for analyzing the collected data and providing the driver with appropriate information and advice, means for interacting with the driver and providing chat and entertainment based on the driver's hobbies and preferences, means for allocating individual topics to each family member, means for transmitting basic information about the user to a cloud server in the autonomous vehicle and providing analysis results and advice to a smart device in real time, and means for generating appropriate responses to the user's voice commands using a generative AI model. This enables the provision of a personalized driving experience based on the user's identification information, the provision of appropriate advice according to the situation of the autonomous vehicle, and advanced voice responses using the generative AI model.

[1168] "Driver identification information" is a series of information for identifying an individual driver, such as the user's name, age, driving history, etc.

[1169] "Real-time data collection" means using sensors and other input devices to instantly obtain data about the ongoing driving situation and vehicle status.

[1170] "Monitoring vehicle condition and driving status" means monitoring engine status, speed, location information, etc. through various sensors in the vehicle to determine whether it is operating normally.

[1171] "Analyzing collected data" means using algorithms and analytical tools based on the acquired data to detect vehicle abnormalities and the driver's condition.

[1172] "Providing appropriate information and advice" means providing drivers with advice for safe driving and comfortable driving based on the analysis results.

[1173] "Dialogue with the driver" means two-way communication with the driver using means such as voice recognition or text input.

[1174] "Providing chat and entertainment based on the driver's hobbies and preferences" means providing topics, music, and other entertainment content that meet the driver's individual preferences.

[1175] "Allocating individual topics to each family member" means bringing up topics that correspond to the interests and concerns of each passenger in the car and conducting individual conversations.

[1176] "Send to a server on the cloud" means uploading data to a remote server via the Internet.

[1177] A "generative AI model" is an algorithm based on machine learning and deep learning that is used to perform advanced tasks such as speech recognition and text generation.

[1178] "Generating an appropriate response to a voice command" means recognizing a voice instruction given by the user and providing an appropriate reaction or information in response to it.

[1179] "Smart devices" is a general term for mobile devices that can connect to the Internet, such as smartphones, tablets, and smart glasses.

[1180] "Personalized driving experience" refers to providing a driving environment and services that are customized based on a user's individual identity and preferences.

[1181] This invention is a system for enhancing the driver-specific experience in autonomous vehicles. The system registers the driver's identification information, monitors the vehicle's status and driving situation in real time, analyzes the collected data, and provides appropriate information and advice. Furthermore, it can use a generative AI model to interact with the driver and provide personalized conversations and entertainment.

[1182] The server first registers the driver's identification information in a database on the cloud. This information includes the user's name, age, driving history, etc. In addition, a terminal installed in the vehicle uses various sensors to collect vehicle status (engine status, speed, location information, etc.) in real time. This data is immediately sent to the server, which then analyzes it.

[1183] Based on the analysis results, the server generates advice and alerts appropriate to the situation and provides them to the driver via a smart device (such as a smartphone or smart glasses). For example, if an engine abnormality is detected, the server will issue an alert stating, "An engine abnormality has occurred. Please stop the vehicle in a safe place and request repairs."

[1184] Additionally, if the server detects that the driver is fatigued, it will provide advice such as, "Long-term driving can cause fatigue, so we recommend you take a break," ensuring the driver's safety. Furthermore, when the driver issues a voice command using the generative AI model, the server generates an appropriate response based on the analysis results and provides it to the driver.

[1185] For example, if a user asks, "Where is the next rest stop?", the smart device sends this information to the server, which uses data about the nearest rest stops to generate a response such as "There is a rest stop XX kilometers away."

[1186] Here is an example prompt:

[1187] "The user wants to know the nearest rest area from their current location. Their current location is [latitude, longitude]. What is the next available rest area?"

[1188] The hardware used includes smartphones, smart glasses, and autonomous vehicle sensors (cameras, GPS, microphones, etc.), while the software used includes cloud servers (AWS, Google Cloud), machine learning models (TensorFlow, PyTorch), and real-time data analysis (Apache Kafka).

[1189] The system will provide a personalized driving experience based on user identity, monitor vehicle status in real time and provide appropriate advice, and use generative AI models to provide advanced voice responses.

[1190] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1191] Step 1:

[1192] When a user gets into a vehicle, the device acquires the user's identification information and sends it to the cloud server. Basic information such as the user's name, age, and driving history is received as input, and based on this information, a new registration or update is made on the cloud. A user ID is generated as output.

[1193] Step 2:

[1194] The terminal collects data in real time from various sensors in the vehicle (engine status, speed, location information, etc.) and sends the collected data to a cloud server. The input includes signal data from each sensor. The server receives this and stores it in a database. The output is accumulated data.

[1195] Step 3:

[1196] The server analyzes the collected data and diagnoses the driver's condition. The stored sensor data is used as input. A machine learning model (e.g., TensorFlow, PyTorch) is used for data analysis to detect signs and abnormalities for safe driving. The analysis results are generated as output.

[1197] Step 4:

[1198] Based on the analysis results, the server automatically generates appropriate advice or alerts and notifies the driver via the terminal. The analysis results are used as input. For example, a message such as "There is an engine malfunction. Please stop the vehicle in a safe place and request repairs" is generated. As output, a voice or text notification is sent to the driver.

[1199] Step 5:

[1200] The server detects the driver's fatigue state from sensor data and provides advice such as, "You are showing signs of fatigue. We recommend that you take a break." Input includes vital data such as heart rate and reaction time. A machine learning model is used to analyze this data and detect signs of fatigue. The output is a notification encouraging the driver to take a break.

[1201] Step 6:

[1202] When a driver inputs a voice command via a smart device (smartphone or smart glasses), the device sends the information to the server. The user's voice command is used as input. Example prompt: "Where is the next rest stop?" The server uses a generative AI model to generate an appropriate response and returns it to the user. Information about rest stops is returned as output.

[1203] Step 7:

[1204] As part of the response generation process, the server performs database searches and calculations in the cloud. The input includes a prompt to the generative AI model. For example, the prompt might read, "The user wants to know the nearest rest area from their current location. Their current location is [latitude, longitude]. Please tell me where the next available rest area is." The server retrieves data on the nearest rest areas and generates an appropriate answer. The output is a response containing the location of the rest area.

[1205] These processing steps enable the system of the present invention to effectively collect and analyze driver-specific data to provide a safe and comfortable driving experience.

[1206] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1207] ---

[1208] This invention achieves even greater driving safety and comfort by collecting driver-specific data by registering and storing the driver's identification information, and combining this data with an emotion engine in a system that provides safe driving and a comfortable driving environment.

[1209] System Configuration

[1210] The system of the present invention includes a vehicle terminal used by the user, a cloud server, an interface for interacting with the driver, and an emotion engine. The terminal is installed in the vehicle and is responsible for the actual data collection and monitoring. The server analyzes and stores the data for long-term storage. The emotion engine has the function of recognizing the user's emotions and generating appropriate dialogue and responses based on those emotions. The users are the driver and passengers, and each receives dialogue and feedback tailored to their needs.

[1211] Program and processing flow

[1212] When a user first connects to the system, the server collects basic information about the user, including their name, age, hobbies, preferences, and driving history. Once the user begins driving, the device collects real-time data from the vehicle's sensors and sends it to the server. The server uses this data to analyze the vehicle's condition and driving situation, and provides appropriate information and advice under specific conditions.

[1213] As a specific example, if an engine abnormality is detected, the device immediately sends the abnormality data to the server. The server analyzes the received data and diagnoses the nature of the abnormality. The driver is then notified via the device with an audio alert saying, "An engine abnormality has occurred. Please stop the vehicle in a safe place and request repairs." By responding immediately in this way, serious vehicle breakdowns can be prevented.

[1214] The emotion engine also analyzes the user's emotions in real time and responds accordingly based on their emotional state. If the driver is feeling stressed, the emotion engine detects this and sends it to the server. The server then selects appropriate advice to reduce stress and music to help them relax, and provides this to the driver via the device. For example, if the emotion engine detects that the driver is stressed, it will notify the driver, "You appear to be feeling stressed at the moment. Would you like us to play your favorite music to help you relax?"

[1215] Furthermore, when the whole family is in the car, the device uses microphones and speakers installed in each seat to assign individual topics to talk about. The emotion engine recognizes the emotional state of each member and provides appropriate dialogue. For example, if a child in the back seat looks sad, the device will ask, "What's wrong? Is there anything you'd like to talk about?" On the other hand, it will proactively provoke a conversation with the parent in the passenger seat by asking, "Why don't you think about your next family trip?"

[1216] This system realizes appropriate dialogue that takes into account the emotional states of the driver and passengers, providing a safer and more comfortable driving environment. The system provides real-time feedback according to the user's emotional state, not only improving driver safety but also creating a lively atmosphere inside the vehicle.

[1217] ---

[1218] The above is an embodiment of the present invention. By constructing a system according to this embodiment, it is possible to provide a comfortable and safe driving environment that takes into account the emotional states of the driver and passengers.

[1219] The processing flow will be explained below.

[1220] ---

[1221] Step 1:

[1222] The user enters the vehicle and logs in to the in-vehicle terminal. The terminal detects this and sends the user's login information to the server. The server uses this information to authenticate the user's profile and returns the necessary data to the terminal.

[1223] Step 2:

[1224] The device collects real-time data from the vehicle's sensors, including vehicle speed, engine status, fuel level, and interior temperature, and periodically transmits this data to a server.

[1225] Step 3:

[1226] The server receives and analyzes real-time data sent from the device, including engine anomaly detection, fuel level checks, and driver fatigue assessment.

[1227] Step 4:

[1228] The device uses the in-car camera and microphone to detect the user's facial expressions and tone of voice, and the emotion engine analyzes this data to identify the user's emotional state.

[1229] Step 5:

[1230] The server receives the emotion data sent from the emotion engine and provides appropriate information and advice to the driver based on the analysis results. For example, if the driver is under stress, the server will generate advice such as "Shall I play your favorite music to help you relax?" and provide it via the device.

[1231] Step 6:

[1232] While driving, the device uses a microphone to receive voice commands from the driver. For example, if the driver asks, "Where is the next rest stop?", the device sends the command to the server. The server analyzes and returns information about the nearest rest stop.

[1233] Step 7:

[1234] Based on the rest stop information returned from the server, the device notifies the driver, "The next rest stop is 5km away." In this way, answers to the driver's questions are provided in real time.

[1235] Step 8:

[1236] The server monitors the driver's fatigue level and generates advice to encourage rest if the driver has been driving for a long time. The device then communicates this to the driver. For example, it may say, "Long-term driving can cause fatigue, so we recommend that you take a break."

[1237] Step 9:

[1238] When the whole family is in the car, the device uses a microphone and speaker installed in each seat to individually assign topics to each member. The emotion engine recognizes the emotional state of each member and provides appropriate dialogue. For example, if a child in the back seat looks sad, the device will ask, "What's wrong? Is there anything you'd like to talk about?" On the other hand, it will proactively provoke a conversation with the parent in the passenger seat by asking, "Why don't you think about your next family trip?"

[1239] ---

[1240] The above are the processing steps for implementing the present invention. By building a system in accordance with these steps, it is possible to provide appropriate feedback in real time according to the emotional state of the driver and passengers, thereby realizing a safe and comfortable driving environment.

[1241] Example 2

[1242] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1243] Conventional vehicle driving assistance systems have limitations in providing safe driving and a comfortable driving environment for drivers, particularly in systems that take into account the driver's emotional state and passenger comfort. This can increase stress and discomfort while driving, potentially jeopardizing safe driving. Furthermore, the lack of a mechanism for providing individual conversations when the whole family is in the car makes it difficult to improve the in-car atmosphere. There is a need to address these issues and improve the overall driving experience for drivers and passengers.

[1244] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for registering and saving the driver's identification information, means for collecting data in real time from vehicle sensors and monitoring the vehicle state and driving situation, means for analyzing the collected data and providing the driver with appropriate information and advice, means for analyzing the emotional states of the driver and passengers using an emotion recognition engine, means for providing dialogue and entertainment based on the emotional states, means for allocating individual topics to each family member, and means for providing a method for the driver to relax when feeling stressed. This makes it possible to provide appropriate feedback and entertainment that takes into account the emotional states of the driver and passengers, thereby improving driving safety and comfort.

[1245] "Driver identification information" is information for identifying a driver, and includes the name, age, hobbies and preferences, driving history, and the like.

[1246] A "sensor" is a device for detecting various vehicle conditions and driving situations in real time, and includes a speed sensor, a brake sensor, an engine sensor, and the like.

[1247] "Real-time data" refers to instantaneous data collected from vehicle sensors during driving, including speed, braking status, engine temperature, and the like.

[1248] "Analysis" is the process of evaluating the driving situation and vehicle condition based on collected data and identifying problems and areas for improvement.

[1249] "Information and advice" refers to notifications and instructions provided to the driver based on the analysis results, including advice on safe driving and warnings of abnormalities.

[1250] An "emotion recognition engine" is software that analyzes the voice and facial expression data of the driver and passengers to determine their emotional state.

[1251] "Entertainment" refers to entertainment content provided to entertain the driver and passengers, and includes music, chat, games, and the like.

[1252] "Feedback" refers to reactions and responses provided to the driver and passengers based on analytical results and emotional state, and may take the form of advice or entertainment.

[1253] "All family members" refers to all passengers in the vehicle, and may include the driver.

[1254] "Individual topic allocation" means providing each passenger with an appropriate topic based on their interests and emotional state.

[1255] "Stress" refers to the mental strain or tension a driver feels while driving, and is detected by an emotion recognition engine.

[1256] "Relaxation methods" are means to reduce stress and relax the driver, and examples include music and suggestions for deep breathing.

[1257] System program generation

[1258] The server generates the system's program, which includes a series of processes for registering driver identification information, collecting real-time data, analyzing data, providing feedback, analyzing emotions, and conducting interactive dialogue.

[1259] Hardware and software used

[1260] Hardware: Devices installed in the vehicle (speed sensor, brake sensor, engine sensor, microphone, camera, speaker), cloud server

[1261] Software: Data analysis software, emotion recognition engine, interface application, cloud storage system

[1262] Program processing (natural language explanation)

[1263] Registering user information

[1264] The server collects basic information about the user when the user first connects to the system, including name, age, hobbies, driving history, etc. When the user logs in to the application, a screen for entering personal information is displayed, and the entered information is saved on the server.

[1265] Real-time data collection

[1266] The terminal collects real-time data from various sensors in the vehicle, including speed, braking status, engine temperature, and audio and video data from inside the vehicle. As soon as the driver starts driving, the sensors start operating and transmit the data to the terminal.

[1267] Data transmission and analysis

[1268] The device sends the collected data to a server in real time. The server analyzes the received data and evaluates the vehicle's condition and driving situation. If the speed data exceeds a certain range or abnormal data is detected from the engine, the server immediately detects the abnormality and performs analysis.

[1269] Providing Feedback

[1270] The server generates appropriate feedback based on the analysis results. The generated feedback is notified to the user via the terminal. For example, if an engine abnormality is detected, the server generates a message saying, "An engine abnormality has occurred. Please stop the vehicle in a safe place and request repairs," and sends it to the terminal. The terminal notifies the driver of this message as an audio alert.

[1271] Emotion analysis

[1272] The device uses a microphone and camera installed in the vehicle to collect the user's voice and facial expression data. The emotion recognition engine analyzes the user's emotions in real time based on the collected data. If the driver is feeling stressed, the emotion recognition engine detects this and sends the analysis results to the server.

[1273] Interactive Dialogue

[1274] The server and emotion recognition engine provide appropriate dialogue and entertainment based on the analysis results. For example, if the driver is feeling stressed, the device will notify them, "Your stress level seems to be increasing. Would you like me to play some relaxing music?" If a child in the back seat looks sad, the device will ask, "What's wrong? Is there anything you'd like to talk to me about?"

[1275] Examples and prompts

[1276] As a concrete example, consider a scenario where an engine malfunction occurs. The device detects the engine malfunction and sends the data to the server. The server analyzes the received data and notifies the driver with a voice alert saying, "An engine malfunction has occurred. Please stop the vehicle in a safe place and request repairs."

[1277] Example prompt sentence:

[1278] "Describe a program that uses previous driving data to generate safe driving advice."

[1279] As a process based on this prompt, the server provides the data analysis results as feedback and gives advice to the driver to encourage safe driving.

[1280] The above is an embodiment of the invention, which takes into consideration the emotional states of the driver and passengers and provides a safe and comfortable driving environment.

[1281] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1282] Step 1:

[1283] The server collects basic user information when the user first connects to the system, including name, age, hobbies, and driving history. The information is stored on the server and serves as the basis for future analysis and interactive dialogue.

[1284] Step 2:

[1285] The terminal collects data in real time from various sensors in the vehicle. Input data includes speed, braking status, engine temperature, and audio and video data inside the vehicle. The collected data is temporarily stored in the terminal and sent to the server as appropriate. This allows the server to manage detailed driving conditions.

[1286] Step 3:

[1287] The terminal sends the collected real-time data to a server. The input data is the output data from the vehicle's sensors, and once sent to the server, the data is analyzed immediately. The server analyzes the data and evaluates the driving situation and vehicle condition.

[1288] Step 4:

[1289] The server generates appropriate feedback based on the analysis results. The input data are the sensor data and analysis results sent to the server, and the output data is a feedback message for the driver. For example, the server generates a message saying, "An abnormality has occurred in the engine. Please stop the vehicle in a safe place and request repairs," and sends it to the terminal. The terminal notifies the driver of this as an audio alert.

[1290] Step 5:

[1291] The device uses the in-car microphone and camera to collect the user's voice and facial expression data. The input data is the voice and video from inside the car, and is sent to the emotion recognition engine. The emotion recognition engine analyzes this data and determines the user's emotional state.

[1292] Step 6:

[1293] The emotion recognition engine sends the result to the server based on the emotion analysis result. The input data is the analysis result by the emotion recognition engine, and the output data is the feedback of the user's emotional state to the server. If the driver is feeling stressed, the emotion recognition engine will detect this and provide the result to the server.

[1294] Step 7:

[1295] The server generates appropriate response feedback and entertainment based on the emotion analysis results. The input data is the emotional state analysis result from the emotion recognition engine, and the output data is voice feedback and entertainment content for the driver and passengers. For example, if the driver is feeling stressed, the server sends a message to the terminal saying, "Would you like me to play some relaxing music?" The terminal notifies the driver of this message by voice, and if the driver agrees, it plays relaxing music.

[1296] The above is a detailed explanation of the processing steps of the program of this system. At each step, the specific data processing and data calculation processes based on the input data are described in detail.

[1297] (Application example 2)

[1298] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1299] The purpose of this invention is to improve driving safety and comfort by analyzing the driver's emotional state in real time and providing appropriate feedback and advice based on that information. Conventional systems have difficulty responding appropriately to the driver's emotional state, and have been unable to effectively reduce stress and fatigue while driving. In addition, they have been unable to individually allocate topics of conversation, making it difficult for the whole family to enjoy a comfortable driving environment.

[1300] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting data from vehicle sensors in real time and analyzing the emotional state of the driver, means for providing appropriate feedback and advice to the driver based on the collected data, and means for suggesting music or relaxation methods based on the driver's emotional state. This makes it possible to respond in real time while taking the driver's emotional state into consideration, allowing the driver and all passengers to enjoy a safer and more comfortable driving environment.

[1301] "Driver identification information" includes individual information such as the driver's name, age, hobbies and preferences, and driving history.

[1302] "Vehicle sensors" refers to various sensor devices used to monitor the vehicle's condition and driving conditions in real time.

[1303] "Means for collecting data in real time" refers to technical devices and software for continuously collecting data from sensors installed in the vehicle.

[1304] "Means for analyzing collected data" refers to algorithms or systems for evaluating the vehicle condition and driving situation based on collected data and processing the results.

[1305] "Means for providing appropriate information and advice to drivers" refers to devices and systems for providing necessary information and driving advice to drivers.

[1306] "Means for communicating with the driver" refers to a voice recognition system or interface for directly communicating with the driver.

[1307] "Means for providing chat and entertainment based on the driver's interests and preferences" refers to a content generation system or interface for providing chat and entertainment based on the driver's interests and preferences.

[1308] "Means for individually allocating topics to each family member" refers to a dialogue system or interface that provides appropriate topics to each person in the car.

[1309] "Emotion recognition" refers to technology that analyzes a driver's emotional state based on data such as facial expressions and voice.

[1310] "Means for suggesting music or relaxation techniques based on the driver's emotional state" refers to a system that takes into account the driver's emotional state and provides appropriate music or relaxation techniques.

[1311] The Emotion Drive Assistant system of the present invention analyzes the emotional state of the driver in real time and provides appropriate feedback and advice based on that information, thereby improving driving safety and comfort. Hereinafter, an embodiment of the present invention will be described in detail.

[1312] System Configuration

[1313] The system includes a terminal in the vehicle used by the driver, a cloud server, an interface for interacting with the driver, and an emotion engine. The terminal is installed in the vehicle and is responsible for the actual data collection and monitoring, while the server analyzes and stores the data for long-term storage.

[1314] Program and processing flow

[1315] Hardware and Software

[1316] The main hardware used in this system includes:

[1317] Webcam: A camera that captures the driver's face in real time and collects data.

[1318] Smartphone: Serves as the control and interface for the entire system.

[1319] Sensors built into the vehicle: Various sensors to collect information such as speed, acceleration, engine status, etc.

[1320] The software used includes:

[1321] OpenCV: A library for processing webcam footage and performing face detection.

[1322] Keras: A machine learning library for running emotion recognition models.

[1323] requests: An HTTP library for communicating with the server.

[1324] Data processing and calculation

[1325] 1. Data Collection:

[1326] When the user starts driving, a web camera connected to the device captures video of the driver's face in real time, and various vehicle sensors collect driving situation data.

[1327] 2. Face detection and emotion recognition:

[1328] We use OpenCV to detect the driver's face from the captured video, and then input the detected face image into an emotion recognition model trained with Keras to analyze the driver's emotional state.

[1329] 3. Submitting Information and Feedback:

[1330] The analyzed emotional state is sent to the server using the requests library. The server analyzes the received data, generates appropriate feedback and advice, and conveys it to the driver via the terminal.

[1331] Specific examples and prompts for the generative AI model

[1332] Examples:

[1333] When a driver launches the EmotionDrive Assistant app and starts driving, the app begins to recognize emotions. If the driver is feeling stressed, the server receives that information and suggests appropriate relaxation methods (such as playing music or instructing breathing exercises) to the device.

[1334] Example prompt for a generative AI model:

[1335] "Design a system in which a smartphone application installed in an autonomous vehicle analyzes the emotional state of the driver in real time and provides appropriate feedback and advice based on that emotional state. The emotion recognition model uses Keras, and the video footage from the webcam is processed in real time."

[1336] This system enables real-time responses that take into account the driver's emotional state, allowing both the driver and passengers to enjoy a safer and more comfortable driving environment.

[1337] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1338] Step 1:

[1339] When the user starts driving, the web camera connected to the device captures the video of the driver's face in real time. The input is the video of the driver's face, and the output is the captured video. This video is used directly for the next step.

[1340] Step 2:

[1341] The device uses OpenCV to detect the driver's face from the captured video. The input of this step is the video captured in step 1, and the output is an image of the area where the face is detected. This face image is used in the next step for facial recognition.

[1342] Step 3:

[1343] The device inputs the facial image into an emotion recognition model trained with Keras to analyze the driver's emotional state. The input is the facial image obtained in step 2, and the output is the emotional state label and its probability. These data are saved for use in the next step.

[1344] Step 4:

[1345] The device uses the requests library to send data containing the analyzed emotional state to the server. The input is the emotional state label and probability, and the output is the emotional state data received by the server. The server receives this data.

[1346] Step 5:

[1347] The server analyzes the received data and generates appropriate feedback or advice. The input of this step is the emotional state data, and the output is the generated feedback or advice. This feedback is used in the next step.

[1348] Step 6:

[1349] The server sends the generated feedback and advice to the terminal. The input is the feedback and advice generated in step 5, and the output is the feedback and advice received by the terminal.

[1350] Step 7:

[1351] The device displays or outputs audio in an appropriate manner to convey the received feedback or advice to the driver. The input is the received feedback or advice, and the output is the actual feedback or advice provided to the driver. Specific actions include displaying the feedback or advice on a screen or announcing it via audio.

[1352] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1353] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1354] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1355] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1356] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1357] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1358] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1359] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1360] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1361] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1362] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1363] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1364] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1365] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1366] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1367] The hardware resource for executing a specific process can be any of the following types of processors: A processor, for example, is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1368] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1369] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1370] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1371] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1372] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1373] The following is further disclosed regarding the above embodiment.

[1374] (Claim 1)

[1375] means for registering and storing driver identification information;

[1376] means for collecting data in real time from sensors in the vehicle to monitor the vehicle's condition and driving status;

[1377] A means of analyzing the collected data and providing appropriate information and advice to drivers;

[1378] A means for interacting with the driver and providing chat and entertainment based on the driver's hobbies and preferences;

[1379] A way to assign individual topics to each family member,

[1380] A system including:

[1381] (Claim 2)

[1382] 10. The system of claim 1, further comprising means for notifying a driver of an alert when an abnormality in the vehicle state is detected.

[1383] (Claim 3)

[1384] 10. The system of claim 1, further comprising means for detecting a driver fatigue state and providing advice to encourage rest.

[1385] "Example 1"

[1386] (Claim 1)

[1387] means for registering and storing driver identification information;

[1388] means for collecting data in real time from sensors in the vehicle to monitor the vehicle's condition and driving status;

[1389] A means of analyzing the collected data and providing appropriate information and advice to drivers;

[1390] A means for interacting with the driver and providing chat and entertainment based on the driver's hobbies and preferences;

[1391] A way to assign individual topics to each family member,

[1392] a means for notifying the driver of an alert in real time when an engine abnormality is detected;

[1393] A means of analyzing long-term driving, detecting driver fatigue, and providing advice to encourage rest;

[1394] means for analyzing the driver's question based on the voice input and providing an appropriate answer;

[1395] A system including:

[1396] (Claim 2)

[1397] 10. The system of claim 1, further comprising means for notifying a driver of an alert when an abnormality in the vehicle state is detected.

[1398] (Claim 3)

[1399] 10. The system of claim 1, further comprising means for detecting a driver fatigue state and providing advice to encourage rest.

[1400] "Application Example 1"

[1401] (Claim 1)

[1402] means for registering and storing driver identification information;

[1403] means for collecting data in real time from sensors in the vehicle to monitor the vehicle's condition and driving status;

[1404] A means of analyzing the collected data and providing appropriate information and advice to drivers;

[1405] A means for interacting with the driver and providing chat and entertainment based on the driver's hobbies and preferences;

[1406] A way to assign individual topics to each family member,

[1407] In an autonomous vehicle, a means for transmitting basic user information to a server on the cloud and providing analysis results and advice to a smart device in real time;

[1408] a means for generating appropriate responses to a user's voice commands using a generative AI model;

[1409] A system including:

[1410] (Claim 2)

[1411] 10. The system of claim 1, further comprising means for notifying a driver of an alert when an abnormality in the vehicle state is detected.

[1412] (Claim 3)

[1413] 10. The system of claim 1, further comprising means for detecting a driver fatigue state and providing advice to encourage rest.

[1414] "Example 2: Combining Emotion Engines"

[1415] (Claim 1)

[1416] means for registering and storing driver identification information;

[1417] means for collecting data in real time from sensors in the vehicle to monitor the vehicle's condition and driving status;

[1418] A means of analyzing the collected data and providing appropriate information and advice to drivers;

[1419] means for analyzing the emotional states of a driver and passengers using an emotion recognition engine;

[1420] a means for providing interaction and entertainment based on emotional states;

[1421] A way to assign individual topics to each family member,

[1422] A means for providing a method of relaxation for drivers when they feel stressed;

[1423] A system including:

[1424] (Claim 2)

[1425] 10. The system of claim 1, further comprising means for notifying a driver of an alert when an abnormality in the vehicle state is detected.

[1426] (Claim 3)

[1427] 10. The system of claim 1, further comprising means for detecting a driver fatigue state and providing advice to encourage rest.

[1428] "Application example 2 when combining emotion engines"

[1429] (Claim 1)

[1430] means for registering and storing driver identification information;

[1431] means for collecting data in real time from sensors in the vehicle to monitor the vehicle's condition and driving status;

[1432] A means of analyzing the collected data and providing appropriate information and advice to drivers;

[1433] A means for interacting with the driver and providing chat and entertainment based on the driver's hobbies and preferences;

[1434] A way to assign individual topics to each family member,

[1435] A means to analyze the driver's emotional state in real time through emotion recognition and provide appropriate feedback and advice;

[1436] means for suggesting music or relaxation techniques based on the driver's emotional state;

[1437] A system including:

[1438] (Claim 2)

[1439] 10. The system of claim 1, further comprising means for notifying a driver of an alert when an abnormality in the vehicle state is detected.

[1440] (Claim 3)

[1441] 10. The system of claim 1, further comprising means for detecting a driver fatigue state and providing advice to encourage rest. [Explanation of symbols]

[1442] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for registering and storing driver identification information; means for collecting data in real time from sensors in the vehicle to monitor the vehicle's condition and driving status; A means of analyzing the collected data and providing appropriate information and advice to drivers; A means for interacting with the driver and providing chat and entertainment based on the driver's hobbies and preferences; A way to assign individual topics to each family member, A system including:

2. The system according to claim 1 , further comprising means for notifying the driver of an alert when an abnormality in the vehicle state is detected.

3. The system of claim 1 further comprising means for detecting a driver's fatigue state and providing advice to encourage rest.

Images