system

Abstract

Provide a system. A method for collecting sensor data from a vehicle for driving skill evaluation includes: a means for analyzing sensor data and using a generative model to perform driver fatigue assessment; A means for proposing a transition to an autonomous driving mode based on the analysis results; and means for notifying the driver of the proposal; A means of collecting human feedback to help improve the system; and Safe driving support system including.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Current autonomous driving technology often lacks the ability to evaluate the driver's driving skill and fatigue level in real time. As a result, the risk of traffic accidents caused by driver fatigue or driving errors cannot be sufficiently reduced. Furthermore, it is difficult for autonomous vehicles to communicate appropriately with people around them in the city, preventing sufficient social acceptance of autonomous driving technology. Furthermore, the lack of functionality to effectively utilize driver feedback to improve the system poses a challenge, making it difficult to continuously improve autonomous driving systems. [Means for solving the problem]

[0005] The present invention provides a safe driving support system that collects sensor data from a vehicle to evaluate driving skills and analyzes that data using a generative model. Specifically, sensor data is collected and analyzed in real time to evaluate the driver's driving skills and fatigue level. Based on the analysis results, a transition to autonomous driving mode is suggested and the driver is notified via audio or visual means. Furthermore, the driver's verbal feedback is converted into text data using speech recognition technology to help improve the system. Furthermore, during autonomous driving in urban areas, speech and gestures are recognized and appropriate responses and instructions are provided through a generative model. In this way, the present invention reduces the risk of traffic accidents and increases social acceptance of autonomous driving technology.

[0006] "Sensors" are devices installed in self-driving cars, and come in a variety of types, such as cameras, lidar, GPS, and fatigue monitoring sensors, and are used to detect the surrounding situation and the driver's condition.

[0007] A "generative model" is an artificial intelligence (AI) algorithm that analyzes driving skills and fatigue levels based on collected sensor data and generates suggestions and alerts.

[0008] "Driving skill" refers to a driver's ability to drive a vehicle safely and efficiently, and is assessed based on data such as speed, frequency of braking, and steering angle.

[0009] "Fatigue level" refers to the degree of fatigue a driver experiences while driving, and is assessed based on biological information such as eye movement, facial expression, and heart rate.

[0010] "Autonomous driving mode" refers to a state in which a vehicle operates fully or partially automatically, and is a function designed to reduce human driving operations.

[0011] "Suggestion" refers to notifications or warnings that encourage the driver to switch to autonomous driving mode based on the analysis results of the generative model.

[0012] "Verbal feedback" refers to answers and opinions given by the driver to the system via voice, and is information that can be used to improve the system and assist the driver.

[0013] "Speech recognition technology" refers to technology for converting voice data into text and is used to analyze verbal feedback.

[0014] "Autonomous driving in urban areas" refers to driving conditions when an autonomous vehicle is traveling on public roads such as in urban or residential areas, where interaction with pedestrians and other vehicles is required.

[0015] "Responses and instructions" are appropriate actions and messages suggested by the generative model through communication with people around the vehicle, intended to support the operation of the autonomous vehicle and the safety of those around it.

[0016] "Improvement" refers to continuous efforts to improve the performance and safety of the automated driving system based on driver feedback and analysis of sensor data. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The safe driving support system of this invention collects driving data in real time from sensors installed in the vehicle and analyzes it using an AI model to evaluate the driver's driving skill and fatigue level. Based on the results of this evaluation, it proposes an appropriate transition to autonomous driving mode and notifies the driver. It also collects verbal feedback from the driver and uses it to retrain the AI ​​model, thereby ensuring continuous improvement of the system. Furthermore, it supports communication with surrounding people when driving in urban areas.

[0039] System Overview

[0040] Device:

[0041] The vehicle is equipped with multiple sensors, including cameras, lidar, GPS, and fatigue monitoring sensors. The device collects data from these sensors and transmits it to a server. For example, a sensor that monitors the driver's eye movements provides important information for assessing the driver's fatigue level.

[0042] server:

[0043] The server receives data sent from the device and inputs it into the generative model. The generative model uses AI algorithms to analyze driving skill and fatigue level in real time. Based on the analysis results, it determines whether switching to autonomous driving mode is necessary and generates a recommendation. The server also receives feedback from the driver to help improve the system.

[0044] User:

[0045] The user is the driver and receives suggestions and warnings from the device. They also provide verbal feedback to contribute to system improvement. For example, after driving for a long time on a highway, the user may receive a notification from the device saying, "We recommend switching to autonomous driving mode." By responding "yes," the user can switch to autonomous driving mode.

[0046] Program processing

[0047] The processing contents of the program will be specifically explained below.

[0048] Sensor data collection:

[0049] The device collects real-time data from various sensors in the vehicle. For example, camera data provides visual information, lidar data provides distance information to surrounding objects, and fatigue monitoring sensors monitor the driver's facial expressions and eye movements to assess fatigue level.

[0050] Data transmission and analysis:

[0051] The device sends the collected data to a server. The server inputs the received data into a generative model for analysis. This analysis evaluates the driver's driving skill and fatigue level. For example, if the driver blinks frequently, the generative model will determine that the driver is highly fatigued.

[0052] Proposal generation and notification:

[0053] The server generates suggestions and warnings and sends them to the device, which then notifies the driver via audio or visual notification. For example, a notification might say, "Your fatigue level is increasing. We recommend switching to autonomous driving mode."

[0054] Processing verbal feedback:

[0055] The device uses voice recognition technology to convert verbal feedback from the driver into text data and sends it to the server. The server analyzes this feedback and uses it to retrain the generative model. For example, if the driver verbally commands, "Please switch to autonomous driving mode," the system converts this into text data and sends it to the server.

[0056] Communication in urban areas:

[0057] During autonomous driving in cities, the system recognizes speech and gestures and provides appropriate responses through generative models, such as an audio message to pedestrians about to cross the street saying, "Please go ahead."

[0058] In this way, by using the system of the present invention, it is possible to increase social acceptance of autonomous driving technology while ensuring driver safety.

[0059] The processing flow will be explained below.

[0060] Program processing steps

[0061] Step 1: Initialize the sensor

[0062] Device:

[0063] 1. Launch an integrated system for autonomous vehicles.

[0064] 2. Initialize the vehicle's camera, lidar, GPS, and fatigue monitoring sensors.

[0065] 3. Start collecting data from the sensors.

[0066] Step 2: Collect sensor data

[0067] Device:

[0068] 1. Collect data in real time from each sensor on the vehicle.

[0069] Camera: Captures visual information about the surroundings.

[0070] Lidar: Obtains distance information to obstacles.

[0071] GPS: Get your current location.

[0072] Fatigue monitoring sensor: Monitors the driver's eye movements and facial expressions.

[0073] Step 3: Sending data

[0074] Device → Server:

[0075] 1. Collected sensor data is periodically sent to a server via batch processing or real-time streaming.

[0076] 2. Data includes speed, frequency of braking, steering angle, eye movements, etc.

[0077] Step 4: Data analysis

[0078] server:

[0079] 1. Input the received sensor data into the generative model.

[0080] 2. A generative model analyzes the data and assesses driving skill and fatigue in real time.

[0081] The analysis results include detecting driving behavior patterns and signs of fatigue.

[0082] Step 5: Generate proposals

[0083] server:

[0084] 1. Evaluate the driver's condition based on the analysis results.

[0085] For example, if the fatigue level is high, the system will determine that the fatigue level is high.

[0086] 2. Generate a proposal to transition to autonomous driving mode.

[0087] For example, it generates a message saying, "We recommend switching to autonomous driving mode."

[0088] Step 6: Proposal Notification

[0089] Server → Device:

[0090] 1. Send generated suggestions and warnings to the device.

[0091] Device:

[0092] 1. Notify the driver of suggestions via display and audio feedback.

[0093] Step 7: Gathering human feedback

[0094] User:

[0095] 1. Providing verbal feedback on proposals and notifications.

[0096] For example, reply, "Yes, please."

[0097] Device:

[0098] 1. Convert verbal feedback into text data using speech recognition technology.

[0099] 2. Send the textual feedback to the server.

[0100] Step 8: Analyze feedback and improve the system

[0101] server:

[0102] 1. Analyze the verbal feedback received.

[0103] 2. Retrain the generative model based on the feedback results and improve the system.

[0104] Step 9: Communicating in the city

[0105] Device → Server:

[0106] 1. Collecting data to recognize speech and gestures while driving autonomously around town.

[0107] 2. Send the collected data to the server.

[0108] server:

[0109] 1. The generative model generates an appropriate response based on the received data.

[0110] For example, if a pedestrian raises their hand, the system understands their intention and determines the appropriate action to take.

[0111] 2. Send the generated responses and instructions to the terminal.

[0112] Device:

[0113] 1. Notify people around you of received instructions via voice or display.

[0114] For example, instructions such as "Please go ahead" are announced through the speaker.

[0115] Through these detailed steps, the system of the present invention can ensure driver safety and increase social acceptance of autonomous driving technology.

[0116] Example 1

[0117] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0118] In modern society, long-distance driving and driving in urban areas place a heavy burden on drivers, and the risk of accidents increases as fatigue accumulates. To solve these problems and achieve safe and efficient driving, a system is needed that monitors the driver's condition in real time and suggests switching to autonomous driving mode as necessary. It is also important to be able to communicate smoothly with people around you in the city.

[0119] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0120] In this invention, the server includes a means for collecting sensor data from the vehicle to evaluate driving skills, a means for analyzing the sensor data and using a generative model to evaluate the driver's fatigue level, a means for proposing switching to autonomous driving mode based on the analysis results, a means for collecting user feedback to help improve the system, and a means for recognizing speech and gestures and providing a response based on the surrounding situation. This makes it possible to accurately grasp the driver's fatigue level and propose switching to autonomous driving mode when necessary. In addition, appropriate responses can be provided even in urban areas, facilitating communication with surrounding people.

[0121] "Sensor data" refers to information collected by various sensors to understand the situation inside and around the vehicle.

[0122] "Driving ability assessment" refers to the measurement and analysis of a driver's driving ability and skills.

[0123] "Fatigue assessment" is the process of monitoring and measuring the driver's fatigue state.

[0124] A "generative model" is an AI algorithm that learns from large amounts of data and analyzes and evaluates driving skills and fatigue levels.

[0125] "Autonomous driving mode" refers to a mode in which the vehicle automatically performs driving operations, minimizing driver involvement.

[0126] "Suggestion" refers to notifications to the driver to change their driving mode or to issue warnings based on the analysis results.

[0127] "Notification" is the act of conveying information to the driver through audio or visual means.

[0128] "Feedback" refers to the evaluations and opinions that the driver provides to the system, which are used to improve the system.

[0129] "Speech recognition" refers to the technology that uses voice recognition technology to understand and process the driver's verbal instructions and feedback.

[0130] "Gesture recognition" is a technology that detects gestures and hand movements made by drivers and pedestrians, and enables the system to provide an appropriate response based on those gestures.

[0131] The safe driving support system of this invention is a system that evaluates the driver's driving skill and fatigue level by collecting driving data in real time from various sensors installed in the vehicle and analyzing it based on an AI-generated model. The system consists of the following components.

[0132] Hardware Configuration

[0133] The device collects data from multiple sensors installed in the vehicle and transmits it to a server. These include a camera, lidar, GPS, and a fatigue monitoring sensor. For example, the camera monitors the driver's face and road conditions, while the lidar measures the distance to surrounding objects. The fatigue monitoring sensor monitors the driver's facial expressions and eye movements and evaluates their fatigue level based on the data.

[0134] Software Configuration

[0135] The server receives the data sent from the device and analyzes it in real time using an AI generative model. A generative model is an AI algorithm that has been trained on a large amount of driving data in advance. Specifically, the server analyzes camera data using an image recognition algorithm (for example, face detection using deep learning) to evaluate the driver's fatigue level. Lidar data and GPS data are also integrated to evaluate the driving environment and road conditions.

[0136] System operation example

[0137] For example, if camera data shows that a driver is blinking frequently after driving for a long period of time, the server's generative model will assess the driver's fatigue level as high. Based on this result, the server will generate a recommendation such as "Your fatigue level is increasing. We recommend switching to autonomous driving mode," and send it to the device. The device will then notify the driver of this recommendation via voice and display.

[0138] The driver provides verbal feedback, such as "Please switch to autonomous driving mode." This verbal feedback is captured by a microphone installed on the device and converted into text data using voice recognition technology. The server receives this text data and uses it to retrain the system. In this way, the system is continuously improved.

[0139] During autonomous driving in urban areas, the device understands the surrounding situation based on data from lidar and cameras. For example, if a pedestrian is detected approaching a crosswalk, the generative model will suggest an appropriate voice message such as "Please go ahead," and the device will notify the driver by voice.

[0140] Prompt Sentence Examples

[0141] "Please explain the process of the program that generates a notification suggesting switching to autonomous driving mode based on fatigue assessment after a long period of driving."

[0142] "Please tell me about the function of the system that allows vehicles traveling in urban areas to provide appropriate audio notifications to pedestrians."

[0143] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0144] Step 1: Collect sensor data

[0145] The device collects data in real time from multiple sensors installed on the vehicle. The inputs are raw data from cameras, lidar, GPS, and fatigue monitoring sensors. These data include cameras capturing images of the driver's face and road conditions, lidar measuring the distance to surrounding objects, GPS acquiring location information, and fatigue monitoring sensors monitoring the driver's facial expressions and eye movements. As an output, these data are shaped into a single integrated data stream, ready to be sent to a server.

[0146] Step 2: Send and analyze data

[0147] The device sends the collected sensor data to the server. The input is the integrated data stream collected and shaped in step 1. The server receives this data stream and inputs it into the generative AI model. The generative AI model uses image recognition algorithms and pattern recognition algorithms to analyze the driver's fatigue level from the camera data and evaluate the driving environment and road conditions from other sensor data. The output is an evaluation of the driver's fatigue level and driving skills.

[0148] Step 3: Proposal generation and notification

[0149] The server generates suggestions and warnings based on the analysis results of step 2. The input is data on the driver's fatigue level assessment and driving skill assessment. The server generates suggestions and warning messages such as "Your fatigue level is increasing. We recommend switching to autonomous driving mode." As output, these messages are sent to the terminal. The terminal notifies the driver of this suggestion via voice or display. Specifically, a warning message is displayed on the display and voice guidance is played from the speaker.

[0150] Step 4: Processing verbal feedback

[0151] The user provides verbal feedback in response to notifications and suggestions from the device. The input is the driver's verbal instruction (e.g., "Please switch to autonomous driving mode"). The device captures this voice feedback and converts it into text data using voice recognition technology. As output, this text data is sent to the server and used for retraining the system. Specifically, the microphone captures the voice, and the internal voice recognition module converts it into text.

[0152] Step 5: Communicating in urban areas

[0153] During autonomous driving in urban areas, the device understands the surrounding situation based on data from the lidar and camera. The input is real-time data from the lidar and camera. For example, if a pedestrian is detected when approaching a crosswalk, the generative AI model will suggest an appropriate voice message such as "Please go ahead." The device will notify the driver of this suggestion by voice. The output is appropriate communication with the driver and pedestrian. In concrete terms, the camera and lidar scan the surrounding situation, and the speaker plays the voice message.

[0154] (Application example 1)

[0155] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0156] Autonomous vehicles are required to accurately assess the driver's fatigue level and driving skills in real time and make appropriate suggestions based on the assessment. However, conventional safe driving support systems are unable to monitor the driver's eye movements and facial expressions in detail, making it difficult to accurately grasp the driver's condition. Furthermore, the lack of effective means to collect and analyze feedback from the driver makes continuous system improvement difficult. Furthermore, there is an issue of being unable to effectively support communication with humans in urban areas during autonomous driving.

[0157] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0158] In this invention, the server includes means for collecting sensor data from the vehicle to evaluate driving skills, means for analyzing the sensor data and using a generative model to evaluate the driver's fatigue level, means for monitoring the driver's eye movements and facial expressions and evaluating them in real time, means for proposing switching to autonomous driving mode based on the analysis results, means for notifying the driver of the suggestion, means for collecting verbal feedback from the driver and converting it into text data using speech recognition technology, means for analyzing the converted text data and using it to improve the system, and means for recognizing speech and gestures during autonomous driving in urban areas and providing appropriate responses. This enables the system to accurately grasp the driver's condition, make appropriate suggestions, and effectively collect and analyze feedback from the driver, enabling continuous system improvement and communication support in urban areas.

[0159] "Driving skill assessment" is the process of measuring and assessing a driver's level of driving skill and ability.

[0160] "Sensor data" refers to information obtained from various sensors installed on the vehicle, including camera images, distance data from LIDAR, and GPS information.

[0161] A "generative model" is an artificial intelligence (AI) algorithm used to analyze collected data and assess the driver's state.

[0162] "Fatigue assessment" is the measurement and evaluation of a driver's level of fatigue, usually by analyzing eye movements and facial expressions.

[0163] "Autonomous driving mode" refers to a mode in which a vehicle drives autonomously, responding to road conditions and the surrounding environment without driver intervention.

[0164] "Notification Means" means the method used to communicate a suggestion or warning to the driver, including a visual display or audio notification.

[0165] "Feedback collection" is the process of collecting opinions and reactions from drivers, which allows for improvements to the system.

[0166] "Voice recognition technology" is a technology that converts voice data into text data, which allows the driver's verbal feedback to be analyzed.

[0167] "Communication in urban areas" refers to an automated vehicle's ability to communicate appropriately with other vehicles and pedestrians in urban areas, including through speech and gesture interaction.

[0168] overview

[0169] The safe driving support system of this invention aims to evaluate driving skills, monitor fatigue levels, collect feedback, appropriately transition to autonomous driving mode, and provide communication support in urban areas. The system collects data from multiple sensors installed in the vehicle and analyzes it with a generative AI model. Based on the analysis results, it makes appropriate suggestions to the driver and collects verbal feedback from the driver for use in retraining the system.

[0170] System Configuration

[0171] 1. Hardware Configuration

[0172] Smart glasses: Monitor the driver's eye movements and facial expressions and provide visual and audio notifications.

[0173] In-vehicle sensors: camera, lidar, GPS, fatigue monitoring sensors.

[0174] Server: A server for running generative AI models for data analysis.

[0175] 2. Software Configuration

[0176] AI Model: A generative AI model built using TENSORFLOW® or PyTorch.

[0177] Speech recognition technology: Uses Google® Speech-to-Text API or equivalent technology.

[0178] Data communication: Data is transmitted between the smart glasses and the server via Wi-Fi or Bluetooth.

[0179] Process Overview

[0180] 1. Data Collection

[0181] The smart glasses and in-vehicle sensors collect real-time data, including the driver's eye movements, facial expressions, camera footage of the surrounding environment, distance information from LiDAR, and location information from GPS.

[0182] 2. Data Transmission

[0183] The collected data is sent to a server via Wi-Fi or Bluetooth.

[0184] 3. Analysis using AI models

[0185] The server inputs the received data into a generative AI model to evaluate the driver's fatigue level and driving skill. For example, if frequent blinking is observed, it is determined that the driver is highly fatigued.

[0186] 4. Proposals and Notifications

[0187] Based on the analysis, it determines whether a transition to autonomous driving mode is necessary and generates recommendations, which are communicated to the driver via visual and audio notifications via the smart glasses.

[0188] 5. Feedback Collection

[0189] Verbal feedback from the driver is collected by the microphone in the smart glasses and converted into text data using voice recognition technology, which is then sent to a server and used to retrain the system.

[0190] 6. Communication in urban areas

[0191] During autonomous driving, the system supports communication with pedestrians and other vehicles in urban areas. For example, it has a function to tell pedestrians "please go ahead."

[0192] Specific examples

[0193] When a driver is driving for a long period of time, the smart glasses monitor the driver's eye movements, and if frequent blinking is detected, a notification will be displayed saying, "Your level of fatigue is increasing. We recommend switching to autonomous driving mode."

[0194] When the driver issues a voice command to "switch to autonomous driving," the system recognizes the voice and switches to autonomous driving mode.

[0195] Prompt Sentence Examples

[0196] Please enter your driving data and fatigue evaluation data in the following format.

[0197] {

[0198] "Driving data": {

[0199] "Speed": "80",

[0200] "camera": "visual information data",

[0201] "lidar": "distance information data",

[0202] "GPS": "Location data"

[0203] },

[0204] "Fatigue Level": "High"

[0205] }

[0206] Generate appropriate suggestions.

[0207] {

[0208] "Suggestion": "We recommend switching to Autopilot mode",

[0209] "Notification Method": "Visual and Audio"

[0210] }

[0211] In this way, the invention can improve driver safety and comfort and provide reliable support during long drives and automated driving in urban areas.

[0212] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0213] Step 1: Data collection

[0214] The terminal collects data in real time from various sensors installed inside the vehicle. Specifically, the camera captures visual information about the vehicle's surroundings, the lidar captures distance information from objects, the GPS captures location information, and the smart glasses capture the driver's eye movements. This collected data is the initial input.

[0215] Step 2: Send data

[0216] The data collected by the device is sent to the server via Wi-Fi or Bluetooth. The input here is the sensor data sent from the device, and the output is the data received by the server.

[0217] Step 3: Data analysis

[0218] The server inputs the received sensor data into the generative AI model to evaluate the driver's driving skill and fatigue level. The input is sensor data, which is calculated and analyzed by the generative AI model. The output is an evaluation of the driver's driving skill and fatigue level.

[0219] Step 4: Proposal Generation

[0220] The server determines whether switching to autonomous driving mode is necessary based on the analysis results and generates a proposal. The input is the result of the data analysis, and the output is the proposal. This proposal may include content such as "We recommend switching to autonomous driving mode."

[0221] Step 5: Notification

[0222] The device communicates the suggestions generated by the server to the driver through the smart glasses with visual and audio notifications. The input is the suggestion from the server, and the output is the notification to the driver. Specifically, the display will say "We recommend switching to autonomous driving mode," and the same message will be conveyed via audio.

[0223] Step 6: Gather feedback

[0224] The user (driver) provides verbal feedback through the microphone in the smart glasses. For example, they give a voice command such as "switch to automatic driving." This becomes the input data. The device converts this voice feedback into text using speech recognition technology and sends it to the server. The output is a string of feedback data.

[0225] Step 7: Feedback analysis

[0226] The server analyzes the character string feedback data sent from the device and uses it to retrain the generative AI model. The input is the feedback data, and the output is the updated results of the model. Specifically, the parameters of the generative AI model are adjusted based on the driver's feedback.

[0227] Step 8: Communicating in urban areas

[0228] The device uses data from sensors to support communication with pedestrians and other vehicles while the vehicle is driving autonomously in urban areas. The input is data from lidar and cameras, which the server analyzes and generates appropriate responses such as "please go ahead." The output is a notification to the driver or pedestrian. For example, if a pedestrian is standing at a crosswalk, the device will notify them by voice, "please go ahead."

[0229] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0230] The safe driving support system of this invention uses in-vehicle sensors to collect driving data in real time and analyzes it using an AI generation model and emotion engine to evaluate the driver's driving skill, fatigue level, and emotional state. Based on the results of this evaluation, the system proposes an appropriate transition to autonomous driving mode and notifies the driver. Furthermore, verbal feedback from the driver is collected and used to retrain the AI ​​generation model, thereby ensuring continuous improvement of the system. Furthermore, when driving in urban areas, the system supports communication with surrounding people.

[0231] System Overview

[0232] Device:

[0233] The vehicle is equipped with multiple sensors, including cameras, lidar, GPS, fatigue monitoring sensors, and an emotion engine that detects the driver's emotional state. The device collects data from these sensors and transmits it to a server. For example, a sensor monitors the driver's eye movements to assess the driver's fatigue level, and facial expressions and voice tone to assess emotions.

[0234] server:

[0235] The server receives data sent from the device and inputs it into the generative AI model and emotion engine. The generative model analyzes the data and evaluates driving skill and fatigue level in real time. The emotion engine analyzes the driver's emotional state and detects stress and anxiety. Based on the results of this analysis, it determines whether switching to autonomous driving mode is necessary and generates a recommendation to do so. The server also receives feedback from the driver to help improve the system.

[0236] User:

[0237] The user is the driver and receives suggestions and warnings from the device. They can also provide verbal feedback to contribute to system improvements. For example, if a user receives a notification from the device while driving on a highway saying, "We recommend switching to autonomous driving mode," they can respond "yes" to switch to autonomous driving mode. If the user feels stressed while driving, the emotion engine can also return advice such as "relax."

[0238] Program processing

[0239] Sensor data collection

[0240] Device:

[0241] 1. Collect data in real time from various sensors on the vehicle.

[0242] Camera: Captures visual information about the surroundings.

[0243] Lidar: Obtains distance information to obstacles.

[0244] GPS: Get your current location.

[0245] Fatigue monitoring sensor: Detects the driver's eye movements and facial expressions.

[0246] Emotion Engine: Identifies the driver's emotional state from their tone of voice and facial expressions.

[0247] Data transmission and analysis

[0248] Device → Server:

[0249] 1. Collected sensor data is periodically sent to the server.

[0250] Data includes speed, frequency of brake use, steering angle, eye movements and voice tones.

[0251] server:

[0252] 1. Input the received data into the generative model and emotion engine.

[0253] 2. The generative model analyzes driving skill and fatigue level in real time.

[0254] For example, if a driver blinks frequently, the generative model will determine that the driver is highly fatigued.

[0255] 3. The emotion engine analyzes the driver's emotional state from their tone of voice and facial expressions.

[0256] For example, if a person's voice is trembling or their facial expression shows signs of tension, it is determined that they are feeling stressed.

[0257] Proposal generation and notification

[0258] server:

[0259] 1. Evaluate the driver's condition based on the analysis results.

[0260] For example, if the driver is highly fatigued or stressed, it is recommended to switch to autonomous driving mode.

[0261] 2. Generate a suggestion: "We recommend transitioning to autonomous driving mode."

[0262] Server → Device:

[0263] 1. Sending generated suggestions and warnings to the device.

[0264] Device:

[0265] 1. Notify the driver of suggestions via display and audio feedback.

[0266] For example, it may notify you, "Your stress level is high. We recommend switching to autonomous driving mode."

[0267] Processing verbal feedback

[0268] User:

[0269] 1. Providing verbal feedback on proposals and notifications.

[0270] For example, reply, "Yes, please."

[0271] Device:

[0272] 1. Convert verbal feedback into text data using speech recognition technology.

[0273] 2. Send the textual feedback to the server.

[0274] Feedback analysis and system improvement

[0275] server:

[0276] 1. Analyze the verbal feedback received.

[0277] 2. Based on the feedback results, the generative model and emotion engine are retrained to improve the system.

[0278] Communication in urban areas

[0279] Device → Server:

[0280] 1. Collecting data for speech and gesture recognition during autonomous driving in urban areas.

[0281] 2. Send the collected data to the server.

[0282] server:

[0283] 1. The generative model generates an appropriate response based on the received data.

[0284] For example, if a pedestrian raises their hand, the system understands their intention and determines the appropriate action to take.

[0285] 2. Send the generated responses and instructions to the terminal.

[0286] Device:

[0287] 1. Notify people around you of received instructions via voice or display.

[0288] For example, instructions such as "Please go ahead" are announced through the speaker.

[0289] In this way, by using the system of the present invention, it is possible to increase social acceptance of autonomous driving technology while ensuring the safety and comfort of drivers.

[0290] The processing flow will be explained below.

[0291] Program processing steps

[0292] Step 1: Initialize the sensor

[0293] Device:

[0294] 1. Launch an integrated system for autonomous vehicles.

[0295] 2. Initialize the vehicle's cameras, lidar, GPS, fatigue monitoring sensors, and emotion engine.

[0296] 3. Start collecting data from the sensors.

[0297] Step 2: Collect sensor data

[0298] Device:

[0299] 1. Collect data in real time from various sensors on the vehicle.

[0300] Camera: Captures visual information about the surroundings.

[0301] Lidar: Obtains distance information to obstacles and other vehicles.

[0302] GPS: Get your current location.

[0303] Fatigue monitoring sensor: Detects the driver's eye movements and facial expressions.

[0304] Emotion Engine: Identifies the driver's emotional state from their tone of voice and facial expressions.

[0305] Step 3: Sending data

[0306] Device → Server:

[0307] 1. Collected sensor data is sent to the server at regular intervals.

[0308] Data includes speed, frequency of brake use, steering angle, eye movements and voice tones.

[0309] Step 4: Data analysis

[0310] server:

[0311] 1. Input the received sensor data into the generative model and emotion engine.

[0312] 2. The generative model analyzes driving skill and fatigue level in real time.

[0313] Driving behavior patterns are analyzed and compared with normal driving behavior.

[0314] Detect signs of fatigue (e.g., frequent blinking).

[0315] 3. The emotion engine analyzes the driver's emotional state from their tone of voice and facial expressions.

[0316] Stress is detected when the voice tone is high and the voice is trembling.

[0317] If the facial muscles are tense, assess anxiety or tension.

[0318] Step 5: Generate proposals

[0319] server:

[0320] 1. Evaluate the driver's condition based on the analysis results.

[0321] For example, if the level of fatigue is high, it is determined that "the driver is fatigued."

[0322] Emotional states such as stress and tension are also assessed.

[0323] 2. Generate a proposal to transition to autonomous driving mode.

[0324] For example, create a message that reads, "Your fatigue and stress levels are high, so we recommend switching to autonomous driving mode."

[0325] Step 6: Proposal Notification

[0326] Server → Device:

[0327] 1. Send generated suggestions and warnings to the device.

[0328] Device:

[0329] 1. Notify the driver of suggestions via display and audio feedback.

[0330] For example, a voice notification may say, "You are highly fatigued, so we recommend switching to autonomous driving mode."

[0331] Step 7: Gathering human feedback

[0332] User:

[0333] 1. Providing verbal feedback on proposals and notifications.

[0334] For example, reply, "Yes, please."

[0335] Device:

[0336] 1. Convert verbal feedback into text data using speech recognition technology.

[0337] 2. Send the textual feedback to the server.

[0338] Step 8: Analyze feedback and improve the system

[0339] server:

[0340] 1. Analyze the verbal feedback received.

[0341] 2. Based on the feedback results, the generative model and emotion engine are retrained to improve the system.

[0342] For example, if the driver instructs the system to "switch to autonomous driving mode," the generative model is updated accordingly.

[0343] Step 9: Communicating in the city

[0344] Device → Server:

[0345] 1. Collecting data for speech and gesture recognition during autonomous driving in urban areas.

[0346] 2. Send the collected data to the server.

[0347] server:

[0348] 1. The generative model generates an appropriate response based on the received data.

[0349] For example, if a pedestrian raises their hand, the system understands their intention and determines the appropriate action to take.

[0350] 2. Send the generated responses and instructions to the terminal.

[0351] Device:

[0352] 1. Notify people around you of received instructions via voice or display.

[0353] For example, instructions such as "Please go ahead" are announced through the speaker.

[0354] Through these detailed steps, the system of the present invention can ensure driver safety and comfort and increase social acceptance of autonomous driving technology.

[0355] Example 2

[0356] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0357] Conventional driving assistance systems have struggled to assess the driver's driving skill, fatigue level, and emotional state in real time and provide appropriate driving assistance based on that assessment. Furthermore, there was a lack of a way to incorporate driver feedback into system improvements, making continuous system improvement difficult. Furthermore, they lacked a function to support communication with surrounding people when the vehicle is stopped in urban areas.

[0358] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0359] In this invention, the server includes means for collecting sensor data from the vehicle to evaluate driving skill, means for analyzing the sensor data and using a generative model to evaluate the driver's fatigue level, means for using an emotion engine to analyze the driver's emotional state, means for proposing transition to autonomous driving mode based on the analysis results, means for notifying the driver of the suggestion, means for converting verbal feedback from the driver into text data using speech recognition technology, means for collecting driver feedback and using it to improve the system, and means for supporting communication with surrounding people when the vehicle is stopped. This enables the driver's driving skill, fatigue level, and emotional state to be evaluated in real time, and appropriate driving assistance based on the evaluation. Furthermore, collecting feedback and using it to improve the system promotes continuous improvement. Furthermore, smooth communication with surrounding people when the vehicle is stopped in urban areas is possible.

[0360] "Sensor data" refers to data obtained from multiple sensors installed in a vehicle and used to evaluate the driver's driving skill, fatigue level, and emotional state.

[0361] A "generative model" is a machine learning algorithm that analyzes collected sensor data to evaluate a driver's driving skill and fatigue level.

[0362] The "emotion engine" is an algorithm that analyzes the driver's emotional state from their tone of voice and facial expressions, and identifies emotions such as stress and anxiety.

[0363] "Autonomous driving mode" is a mode in which the system controls the vehicle on behalf of the driver, and switches automatically depending on the driver's level of fatigue and emotional state.

[0364] "Suggestions" refer to instructions or advice that the system notifies the driver based on the analysis results, including, for example, a notification recommending switching to autonomous driving mode.

[0365] "Feedback" is the verbal or other responses or opinions from the driver that are collected and used to improve the system.

[0366] "Voice recognition technology" is a technology that converts the driver's voice into a digital signal and analyzes it as text data.

[0367] "System improvement" is the process of retraining the generative model and emotion engine based on collected feedback to improve the overall performance of the safe driving support system.

[0368] "Communication Support" is an assistance function that facilitates smooth interactions with people around the vehicle when stopped, and provides appropriate responses and instructions via voice and display.

[0369] This invention relates to a safe driving support system that uses in-vehicle sensors to collect driving data and analyzes it using a generative AI model and emotion engine to evaluate the driver's driving skill, fatigue level, and emotional state. Based on the analysis results, the system proposes an appropriate transition to an autonomous driving mode and notifies the driver. Furthermore, verbal feedback from the driver is collected and used to retrain the system, ensuring continuous improvement. Furthermore, when driving in urban areas, the system supports communication with surrounding people.

[0370] Hardware and software used

[0371] Hardware:

[0372] Camera: Captures visual information about the vehicle's surroundings.

[0373] Lidar: Obtains distance information to obstacles.

[0374] GPS: Get your current location.

[0375] Fatigue monitoring sensor: Detects eye movements and facial expressions.

[0376] Emotion Engine: Identify emotional states from voice tone and facial expressions.

[0377] software:

[0378] Generative AI model: Uses collected data to analyze driving skills and fatigue levels in real time.

[0379] Speech recognition technology: Analyzes verbal feedback and converts it into text data.

[0380] Explanation of program processing

[0381] Device:

[0382] Various sensors installed inside the vehicle collect data in real time and send it to a server. Specifically, the camera captures visual information about the surroundings, the lidar measures the distance to obstacles, and the GPS tracks location information. In addition, a fatigue monitoring sensor monitors the driver's eye movements and facial expressions, and an emotion engine analyzes the driver's emotional state from voice tone and facial expressions. For example, if the driver blinks frequently, it is determined that the driver is highly fatigued.

[0383] server:

[0384] The server receives sensor data sent from the device and inputs it into the generative AI model and emotion engine for analysis. The generative AI model evaluates driving skill and fatigue level, and the emotion engine analyzes the driver's emotional state. Based on the analysis results, it determines whether switching to autonomous driving mode is necessary and generates a proposal to do so. The proposal or warning is sent to the device and notified to the driver. If the driver answers "yes," the system switches to autonomous driving mode. In addition, verbal feedback from the driver is converted into text data using voice recognition technology and used to improve the system. For example, the system can switch to autonomous driving by replying "yes" to a notification that says, "We recommend switching to autonomous driving mode."

[0385] User:

[0386] The driver responds to the suggestions and warnings provided and provides verbal feedback. For example, if the driver receives a notification saying, "Your fatigue level is increasing. We recommend switching to autonomous driving mode," and responds, "Yes, please," the system will switch to autonomous driving mode. Additionally, when the vehicle is stopped in urban areas, the system will provide appropriate instructions to nearby pedestrians via voice, such as, "Please go ahead and let the vehicle pass."

[0387] Examples of concrete examples and prompts

[0388] Examples:

[0389] If the driver feels fatigued and blinks more frequently while driving on a highway, the system will issue a voice notification saying, "Your fatigue level is high, so we recommend switching to autonomous driving mode." If the driver replies "yes," the system will switch to autonomous driving mode.

[0390] Example prompt sentence:

[0391] "Analyze data from vehicle sensors as follows and suggest transitioning to autonomous driving mode: When the driver is highly fatigued and stressed."

[0392] As described above, the safe driving support system of the present invention can provide advanced support for driver safety and comfort by combining a wide range of sensors with an advanced AI model and emotion engine.

[0393] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0394] Step 1:

[0395] Sensor data collection

[0396] Device: Collects data in real time from various sensors installed in the vehicle. Specifically, the camera captures visual information of the surroundings, the lidar measures the distance to obstacles, and the GPS acquires the current location information. The fatigue monitoring sensor monitors the driver's eye movements and facial expressions, and the emotion engine analyzes the driver's emotional state from voice tone and facial expressions.

[0397] Inputs: Data from camera, lidar, GPS, fatigue monitoring sensors, emotion engine.

[0398] Output: Camera visual information, rider distance information, GPS location information, fatigue level information, emotional state information.

[0399] Step 2:

[0400] Sending collected data

[0401] Device → Server: Collected sensor data is periodically sent to the server, including speed information, braking frequency, steering angle, eye movements, and audio tones.

[0402] Input: Data collected from various sensors.

[0403] Output: Sensor data sent to the server.

[0404] Step 3:

[0405] Data analysis

[0406] Server: The received data is input into the generative AI model and emotion engine to analyze driving skill and fatigue level. The generative AI model evaluates driving skill and determines fatigue level based on factors such as frequent blinking. The emotion engine analyzes the driver's emotional state based on voice tone and facial expressions. For example, if the voice is trembling, it is interpreted as being stressed.

[0407] Input: Sensor data sent to the server.

[0408] Output: Driving skill evaluation results, fatigue evaluation results, emotional state evaluation results.

[0409] Step 4:

[0410] Proposal Generation

[0411] Server: Based on the analysis results, evaluates the driver's condition and determines whether switching to autonomous driving mode is necessary. For example, if the driver's fatigue level or stress level is high, a recommendation is generated to "recommend switching to autonomous driving mode."

[0412] Input: Driving skill evaluation results, fatigue evaluation results, emotional state evaluation results.

[0413] Output: Proposal to transition to autonomous driving mode.

[0414] Step 5:

[0415] Proposal Notification

[0416] Server → Device: Sends generated suggestions and warnings to the device.

[0417] Device: Notify the driver of the suggestion through a display or audio feedback, for example, "Your stress level is high. We recommend switching to autonomous driving mode."

[0418] Input: Proposal to transition to autonomous driving mode.

[0419] Output: Display of suggestion notification and audio feedback.

[0420] Step 6:

[0421] Processing Feedback

[0422] User: The driver provides verbal feedback to the suggestion or notification, for example, by replying "Yes, please."

[0423] Terminal: Verbal feedback is converted into text data using speech recognition technology and sent to the server.

[0424] Input: User's verbal feedback.

[0425] Output: Feedback converted to text data.

[0426] Step 7:

[0427] Feedback analysis and system improvement

[0428] Server: Analyzes the received verbal feedback and retrains the generative model and emotion engine to improve the system.

[0429] Input: Feedback converted to text data.

[0430] Output: Improved generative model and emotion engine.

[0431] Step 8:

[0432] Communication support in urban areas

[0433] Terminal → Server: Collects data to recognize speech and gestures during autonomous driving in urban areas and sends it to the server.

[0434] Server: The generative model generates an appropriate response based on the received data. For example, if a pedestrian raises their hand, the server understands their intention and determines the appropriate action. The server then sends the generated response and instructions to the device.

[0435] Terminal: The terminal notifies people around it of the received instructions through voice or display. For example, it may announce the instruction "Please go ahead" through a speaker.

[0436] Input: Speech and gesture data.

[0437] Output: Audio and display notifications for people around you.

[0438] (Application example 2)

[0439] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0440] Conventional driving support systems have difficulty accurately analyzing the driver's fatigue level and emotional state in real time, resulting in an inability to suggest transitioning to autonomous driving mode at the appropriate time. Effectively collecting feedback from the driver and utilizing it for continuous system improvement has also been a challenge. Furthermore, smooth communication with surrounding people in urban areas is also required.

[0441] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for collecting the driver's facial expressions, voice tone, and driving behavior data in real time using a smartphone; means for transmitting the collected data to the server and analyzing it using an AI generation model and an emotion engine; means for notifying the driver via audio or visual means on the smartphone based on the analysis results; means for converting the driver's verbal feedback into text data using voice recognition technology and transmitting it to the server; and means for supporting communication with surrounding people while driving. This makes it possible to accurately grasp the driver's fatigue level and emotional state and suggest switching to autonomous driving mode at the appropriate time. Furthermore, it is possible to effectively collect driver feedback and achieve continuous system improvement. Furthermore, it also enables smooth communication in urban areas.

[0442] A "driving skill assessment" is an assessment of how safely and efficiently a driver can drive.

[0443] "Sensor data" refers to information collected from various sensors, including camera footage, audio, acceleration, and location information.

[0444] A "generative model" is an algorithm that uses AI technology to analyze data for a specific purpose and predict an outcome.

[0445] "Fatigue assessment" is the process of measuring how tired a driver is.

[0446] "Autonomous driving mode" refers to a mode in which the vehicle performs driving operations autonomously.

[0447] "Notification" refers to the act of the system informing the driver of information, and includes audio and visual notifications.

[0448] "Feedback" refers to the opinions and reactions provided by the driver to the system.

[0449] "System improvement" means improving the performance and accuracy of the system based on collected feedback and data.

[0450] A "smartphone" is a device that combines advanced computing power and connectivity with a mobile phone.

[0451] "Facial expression" refers to the facial state that is revealed by the movement of the driver's facial muscles.

[0452] "Voice tone" refers to the pitch, strength, and emotional characteristics of a driver's voice.

[0453] "Driving behavior data" refers to data related to the driver's driving operations, including acceleration, deceleration, braking, etc.

[0454] The "emotion engine" is an algorithm that analyzes voice tone and facial expressions to detect the driver's emotional state.

[0455] "Speech recognition technology" refers to the technology that converts voice into text data.

[0456] "Communication in urban areas" refers to smoothly exchanging information with surrounding pedestrians and other vehicles while driving.

[0457] The safe driving support system of the present invention evaluates the driver's driving skill, fatigue level, and emotional state, and suggests transitioning to autonomous driving mode at an appropriate time. This system uses a smartphone, a server, and various sensors.

[0458] Hardware Configuration

[0459] 1. Smartphone: A smartphone equipped with a camera, microphone, accelerometer, and GPS is used to collect the driver's facial expressions, voice tone, and driving behavior data in real time.

[0460] 2. Server: A server equipped with a high-performance GPU is used. The server receives data sent from the smartphone and analyzes it using a generative AI model and emotion engine.

[0461] 3. Communication network: Data is sent and received using communication networks such as Wi-Fi and 4G / 5G.

[0462] Software Configuration

[0463] 1. Smartphone app: Install the app that collects driving data, sends the data to a server, notifies the driver of the analysis results, and collects driver feedback. This app runs on either ANDROID (registered trademark) or iOS.

[0464] 2. Speech Recognition System: Use speech recognition technology, such as Google Cloud Speech-to-Text, to convert the driver's verbal feedback into text data.

[0465] 3. Generative AI model and emotion engine: This model will be equipped with algorithms to analyze driving data and evaluate driving skill, fatigue level, and emotional state. These algorithms will be implemented using machine learning libraries such as PyTorch and TensorFlow.

[0466] System Operation

[0467] The server receives driving data sent from the smartphone. The received data is analyzed using a generative AI model and an emotion engine. This analysis evaluates the driver's driving skill, fatigue level, and emotional state. For example, if the server determines that the driver is very tired, it generates a notification suggesting, "Do you want to switch to autonomous driving mode?" This notification is communicated to the driver via audio or visual means via the smartphone.

[0468] Driver feedback is also important. When the driver responds verbally to suggestions and notifications, the smartphone's voice recognition system converts the speech into text data and sends it to the server. The server analyzes the feedback and uses the results to retrain the generative AI model and emotion engine, ensuring continuous improvement of the system.

[0469] To facilitate smooth communication in urban areas while driving, the system also sends voice messages to enable drivers to safely and smoothly exchange information with those around them, and this function also supports communication with pedestrians and other vehicles.

[0470] Examples and prompts

[0471] As a concrete example, if a driver continues driving for a long period of time, their facial expressions are recorded by a camera mounted on their smartphone. The generative AI model determines that the driver is highly fatigued from their facial expression and issues a message saying, "You seem tired. Would you like to switch to autonomous driving mode?" If the driver replies "yes," the voice recognition system converts the speech into text data and sends it to a server. Based on this feedback, the system switches to autonomous driving mode.

[0472] Example prompt sentence:

[0473] "The system analyzes the driver's emotional state from their facial expressions and vocal tone to assess their driving skill and fatigue level. If it determines that their fatigue level is high, it notifies the driver, 'Do you want to switch to autonomous driving mode?' and receives verbal feedback. It communicates with the server and uses the driver's feedback to improve the system."

[0474] This will improve driver safety and comfort, and also increase social acceptance of autonomous driving technology.

[0475] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0476] Step 1:

[0477] The device (smartphone) uses a camera, microphone, acceleration sensor, and GPS to collect the driver's facial expressions, voice tone, and driving behavior data in real time while driving. Inputs include the driver's facial image, voice, acceleration data, and location information. This data is collected from various sensors and stored on the device.

[0478] Step 2:

[0479] The terminal periodically transmits the collected data to the server. The driving data collected by the terminal is used as input, and the data is transmitted to the server as output. This transmission is performed using a communication network (Wi-Fi, 4G / 5G).

[0480] Step 3:

[0481] The server receives the data sent from the device and inputs it into the generative AI model and emotion engine. The input includes driving data (facial images, audio, driving behavior data, and location information). The server passes the received data to the AI ​​model and begins analysis.

[0482] Step 4:

[0483] The generative AI model analyzes driving data and evaluates driving skill and fatigue level in real time. Facial images and driving behavior data are used as input, and the output is an evaluation result of driving skill and fatigue level. For example, if a driver blinks frequently, the model will determine that the driver is highly fatigued.

[0484] Step 5:

[0485] The emotion engine analyzes the driver's emotional state from their voice tone and facial expression. It uses voice data and facial images as input, and generates an evaluation result of the driver's emotional state as output. For example, if the voice tone is unstable, the server determines that the driver is stressed.

[0486] Step 6:

[0487] The server evaluates the driver's state based on the analysis results of the generative AI model and emotion engine. The input includes the evaluation results of driving skill, fatigue level, and emotional state. The output generates a proposal to switch to autonomous driving mode if necessary. For example, a proposal such as "Do you want to switch to autonomous driving mode?" is generated.

[0488] Step 7:

[0489] The server sends the generated proposal to the terminal. The proposal content is used as input, and the proposal is sent to the terminal as output. The terminal notifies the driver of this proposal by voice or visual means. Specifically, the smartphone speaker will say, "Do you want to switch to autonomous driving mode?"

[0490] Step 8:

[0491] The user (driver) provides verbal feedback in response to the notification from the terminal. The driver's voice is used as input, and the voice data is collected by the terminal as output. For example, the driver replies "Yes."

[0492] Step 9:

[0493] The terminal converts the collected voice data into text data using a voice recognition system. The driver's voice data is used as input, and text data is generated as output. This converts the voice "yes" into the text "yes."

[0494] Step 10:

[0495] The device sends textual feedback to the server. Text data is used as input, and feedback is sent to the server as output. The server analyzes the received feedback and retrains the AI ​​model and emotion engine based on the results to improve the system. Specifically, data on how the driver responded to each notification is accumulated.

[0496] Step 11:

[0497] During autonomous driving in urban areas, the server and terminal collect data to communicate with surrounding pedestrians and other vehicles and send it to the server. Surrounding audio and image data are used as input, and sent to the server as output. The server generates an appropriate response based on the received data and sends it to the terminal. The terminal notifies surrounding people of the generated response by voice or display. For example, an instruction such as "Please go ahead" may be issued through the speaker.

[0498] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0499] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0500] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0501] [Second embodiment]

[0502] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0503] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0504] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0505] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0506] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0507] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0508] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0509] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0510] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0511] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0512] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0513] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0514] The safe driving support system of this invention collects driving data in real time from sensors installed in the vehicle and analyzes it using an AI model to evaluate the driver's driving skill and fatigue level. Based on the results of this evaluation, it proposes an appropriate transition to autonomous driving mode and notifies the driver. It also collects verbal feedback from the driver and uses it to retrain the AI ​​model, thereby ensuring continuous improvement of the system. Furthermore, it supports communication with surrounding people when driving in urban areas.

[0515] System Overview

[0516] Device:

[0517] The vehicle is equipped with multiple sensors, including cameras, lidar, GPS, and fatigue monitoring sensors. The device collects data from these sensors and transmits it to a server. For example, a sensor that monitors the driver's eye movements provides important information for assessing the driver's fatigue level.

[0518] server:

[0519] The server receives data sent from the device and inputs it into the generative model. The generative model uses AI algorithms to analyze driving skill and fatigue level in real time. Based on the analysis results, it determines whether switching to autonomous driving mode is necessary and generates a recommendation. The server also receives feedback from the driver to help improve the system.

[0520] User:

[0521] The user is the driver and receives suggestions and warnings from the device. They also provide verbal feedback to contribute to system improvement. For example, after driving for a long time on a highway, the user may receive a notification from the device saying, "We recommend switching to autonomous driving mode." By responding "yes," the user can switch to autonomous driving mode.

[0522] Program processing

[0523] The processing contents of the program will be specifically explained below.

[0524] Sensor data collection:

[0525] The device collects real-time data from various sensors in the vehicle. For example, camera data provides visual information, lidar data provides distance information to surrounding objects, and fatigue monitoring sensors monitor the driver's facial expressions and eye movements to assess fatigue level.

[0526] Data transmission and analysis:

[0527] The device sends the collected data to a server. The server inputs the received data into a generative model for analysis. This analysis evaluates the driver's driving skill and fatigue level. For example, if the driver blinks frequently, the generative model will determine that the driver is highly fatigued.

[0528] Proposal generation and notification:

[0529] The server generates suggestions and warnings and sends them to the device, which then notifies the driver via audio or visual notification. For example, a notification might say, "Your fatigue level is increasing. We recommend switching to autonomous driving mode."

[0530] Processing verbal feedback:

[0531] The device uses voice recognition technology to convert verbal feedback from the driver into text data and sends it to the server. The server analyzes this feedback and uses it to retrain the generative model. For example, if the driver verbally commands, "Please switch to autonomous driving mode," the system converts this into text data and sends it to the server.

[0532] Communication in urban areas:

[0533] During autonomous driving in cities, the system recognizes speech and gestures and provides appropriate responses through generative models, such as an audio message to pedestrians about to cross the street saying, "Please go ahead."

[0534] In this way, by using the system of the present invention, it is possible to increase social acceptance of autonomous driving technology while ensuring driver safety.

[0535] The processing flow will be explained below.

[0536] Program processing steps

[0537] Step 1: Initialize the sensor

[0538] Device:

[0539] 1. Launch an integrated system for autonomous vehicles.

[0540] 2. Initialize the vehicle's camera, lidar, GPS, and fatigue monitoring sensors.

[0541] 3. Start collecting data from the sensors.

[0542] Step 2: Collect sensor data

[0543] Device:

[0544] 1. Collect data in real time from each sensor on the vehicle.

[0545] Camera: Captures visual information about the surroundings.

[0546] Lidar: Obtains distance information to obstacles.

[0547] GPS: Get your current location.

[0548] Fatigue monitoring sensor: Monitors the driver's eye movements and facial expressions.

[0549] Step 3: Sending data

[0550] Device → Server:

[0551] 1. Collected sensor data is periodically sent to a server via batch processing or real-time streaming.

[0552] 2. Data includes speed, frequency of braking, steering angle, eye movements, etc.

[0553] Step 4: Data analysis

[0554] server:

[0555] 1. Input the received sensor data into the generative model.

[0556] 2. A generative model analyzes the data and assesses driving skill and fatigue in real time.

[0557] The analysis results include detecting driving behavior patterns and signs of fatigue.

[0558] Step 5: Generate proposals

[0559] server:

[0560] 1. Evaluate the driver's condition based on the analysis results.

[0561] For example, if the fatigue level is high, the system will determine that the fatigue level is high.

[0562] 2. Generate a proposal to transition to autonomous driving mode.

[0563] For example, it generates a message saying, "We recommend switching to autonomous driving mode."

[0564] Step 6: Proposal Notification

[0565] Server → Device:

[0566] 1. Send generated suggestions and warnings to the device.

[0567] Device:

[0568] 1. Notify the driver of suggestions via display and audio feedback.

[0569] Step 7: Gathering human feedback

[0570] User:

[0571] 1. Providing verbal feedback on proposals and notifications.

[0572] For example, reply, "Yes, please."

[0573] Device:

[0574] 1. Convert verbal feedback into text data using speech recognition technology.

[0575] 2. Send the textual feedback to the server.

[0576] Step 8: Analyze feedback and improve the system

[0577] server:

[0578] 1. Analyze the verbal feedback received.

[0579] 2. Retrain the generative model based on the feedback results and improve the system.

[0580] Step 9: Communicating in the city

[0581] Device → Server:

[0582] 1. Collecting data to recognize speech and gestures while driving autonomously around town.

[0583] 2. Send the collected data to the server.

[0584] server:

[0585] 1. The generative model generates an appropriate response based on the received data.

[0586] For example, if a pedestrian raises their hand, the system understands their intention and determines the appropriate action to take.

[0587] 2. Send the generated responses and instructions to the terminal.

[0588] Device:

[0589] 1. Notify people around you of received instructions via voice or display.

[0590] For example, instructions such as "Please go ahead" are announced through the speaker.

[0591] Through these detailed steps, the system of the present invention can ensure driver safety and increase social acceptance of autonomous driving technology.

[0592] Example 1

[0593] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0594] In modern society, long-distance driving and driving in urban areas place a heavy burden on drivers, and the risk of accidents increases as fatigue accumulates. To solve these problems and achieve safe and efficient driving, a system is needed that monitors the driver's condition in real time and suggests switching to autonomous driving mode as necessary. It is also important to be able to communicate smoothly with people around you in the city.

[0595] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0596] In this invention, the server includes a means for collecting sensor data from the vehicle to evaluate driving skills, a means for analyzing the sensor data and using a generative model to evaluate the driver's fatigue level, a means for proposing switching to autonomous driving mode based on the analysis results, a means for collecting user feedback to help improve the system, and a means for recognizing speech and gestures and providing a response based on the surrounding situation. This makes it possible to accurately grasp the driver's fatigue level and propose switching to autonomous driving mode when necessary. In addition, appropriate responses can be provided even in urban areas, facilitating communication with surrounding people.

[0597] "Sensor data" refers to information collected by various sensors to understand the situation inside and around the vehicle.

[0598] "Driving ability assessment" refers to the measurement and analysis of a driver's driving ability and skills.

[0599] "Fatigue assessment" is the process of monitoring and measuring the driver's fatigue state.

[0600] A "generative model" is an AI algorithm that learns from large amounts of data and analyzes and evaluates driving skills and fatigue levels.

[0601] "Autonomous driving mode" refers to a mode in which the vehicle automatically performs driving operations, minimizing driver involvement.

[0602] "Suggestion" refers to notifications to the driver to change their driving mode or to issue warnings based on the analysis results.

[0603] "Notification" is the act of conveying information to the driver through audio or visual means.

[0604] "Feedback" refers to the evaluations and opinions that the driver provides to the system, which are used to improve the system.

[0605] "Speech recognition" refers to the technology that uses voice recognition technology to understand and process the driver's verbal instructions and feedback.

[0606] "Gesture recognition" is a technology that detects gestures and hand movements made by drivers and pedestrians, and enables the system to provide an appropriate response based on those gestures.

[0607] The safe driving support system of this invention is a system that evaluates the driver's driving skill and fatigue level by collecting driving data in real time from various sensors installed in the vehicle and analyzing it based on an AI-generated model. The system consists of the following components.

[0608] Hardware Configuration

[0609] The device collects data from multiple sensors installed in the vehicle and transmits it to a server. These include a camera, lidar, GPS, and a fatigue monitoring sensor. For example, the camera monitors the driver's face and road conditions, while the lidar measures the distance to surrounding objects. The fatigue monitoring sensor monitors the driver's facial expressions and eye movements and evaluates their fatigue level based on the data.

[0610] Software Configuration

[0611] The server receives the data sent from the device and analyzes it in real time using an AI generative model. A generative model is an AI algorithm that has been trained on a large amount of driving data in advance. Specifically, the server analyzes camera data using an image recognition algorithm (for example, face detection using deep learning) to evaluate the driver's fatigue level. Lidar data and GPS data are also integrated to evaluate the driving environment and road conditions.

[0612] System operation example

[0613] For example, if camera data shows that a driver is blinking frequently after driving for a long period of time, the server's generative model will assess the driver's fatigue level as high. Based on this result, the server will generate a recommendation such as "Your fatigue level is increasing. We recommend switching to autonomous driving mode," and send it to the device. The device will then notify the driver of this recommendation via voice and display.

[0614] The driver provides verbal feedback, such as "Please switch to autonomous driving mode." This verbal feedback is captured by a microphone installed on the device and converted into text data using voice recognition technology. The server receives this text data and uses it to retrain the system. In this way, the system is continuously improved.

[0615] During autonomous driving in urban areas, the device understands the surrounding situation based on data from lidar and cameras. For example, if a pedestrian is detected approaching a crosswalk, the generative model will suggest an appropriate voice message such as "Please go ahead," and the device will notify the driver by voice.

[0616] Prompt Sentence Examples

[0617] "Please explain the process of the program that generates a notification suggesting switching to autonomous driving mode based on fatigue assessment after a long period of driving."

[0618] "Please tell me about the function of the system that allows vehicles traveling in urban areas to provide appropriate audio notifications to pedestrians."

[0619] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0620] Step 1: Collect sensor data

[0621] The device collects data in real time from multiple sensors installed on the vehicle. The inputs are raw data from cameras, lidar, GPS, and fatigue monitoring sensors. These data include cameras capturing images of the driver's face and road conditions, lidar measuring the distance to surrounding objects, GPS acquiring location information, and fatigue monitoring sensors monitoring the driver's facial expressions and eye movements. As an output, these data are shaped into a single integrated data stream, ready to be sent to a server.

[0622] Step 2: Send and analyze data

[0623] The device sends the collected sensor data to the server. The input is the integrated data stream collected and shaped in step 1. The server receives this data stream and inputs it into the generative AI model. The generative AI model uses image recognition algorithms and pattern recognition algorithms to analyze the driver's fatigue level from the camera data and evaluate the driving environment and road conditions from other sensor data. The output is an evaluation of the driver's fatigue level and driving skills.

[0624] Step 3: Proposal generation and notification

[0625] The server generates suggestions and warnings based on the analysis results of step 2. The input is data on the driver's fatigue level assessment and driving skill assessment. The server generates suggestions and warning messages such as "Your fatigue level is increasing. We recommend switching to autonomous driving mode." As output, these messages are sent to the terminal. The terminal notifies the driver of this suggestion via voice or display. Specifically, a warning message is displayed on the display and voice guidance is played from the speaker.

[0626] Step 4: Processing verbal feedback

[0627] The user provides verbal feedback in response to notifications and suggestions from the device. The input is the driver's verbal instruction (e.g., "Please switch to autonomous driving mode"). The device captures this voice feedback and converts it into text data using voice recognition technology. As output, this text data is sent to the server and used for retraining the system. Specifically, the microphone captures the voice, and the internal voice recognition module converts it into text.

[0628] Step 5: Communicating in urban areas

[0629] During autonomous driving in urban areas, the device understands the surrounding situation based on data from the lidar and camera. The input is real-time data from the lidar and camera. For example, if a pedestrian is detected when approaching a crosswalk, the generative AI model will suggest an appropriate voice message such as "Please go ahead." The device will notify the driver of this suggestion by voice. The output is appropriate communication with the driver and pedestrian. In concrete terms, the camera and lidar scan the surrounding situation, and the speaker plays the voice message.

[0630] (Application example 1)

[0631] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0632] Autonomous vehicles are required to accurately assess the driver's fatigue level and driving skills in real time and make appropriate suggestions based on the assessment. However, conventional safe driving support systems are unable to monitor the driver's eye movements and facial expressions in detail, making it difficult to accurately grasp the driver's condition. Furthermore, the lack of effective means to collect and analyze feedback from the driver makes continuous system improvement difficult. Furthermore, there is an issue of being unable to effectively support communication with humans in urban areas during autonomous driving.

[0633] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0634] In this invention, the server includes means for collecting sensor data from the vehicle to evaluate driving skills, means for analyzing the sensor data and using a generative model to evaluate the driver's fatigue level, means for monitoring the driver's eye movements and facial expressions and evaluating them in real time, means for proposing switching to autonomous driving mode based on the analysis results, means for notifying the driver of the suggestion, means for collecting verbal feedback from the driver and converting it into text data using speech recognition technology, means for analyzing the converted text data and using it to improve the system, and means for recognizing speech and gestures during autonomous driving in urban areas and providing appropriate responses. This enables the system to accurately grasp the driver's condition, make appropriate suggestions, and effectively collect and analyze feedback from the driver, enabling continuous system improvement and communication support in urban areas.

[0635] "Driving skill assessment" is the process of measuring and assessing a driver's level of driving skill and ability.

[0636] "Sensor data" refers to information obtained from various sensors installed on the vehicle, including camera images, distance data from LIDAR, and GPS information.

[0637] A "generative model" is an artificial intelligence (AI) algorithm used to analyze collected data and assess the driver's state.

[0638] "Fatigue assessment" is the measurement and evaluation of a driver's level of fatigue, usually by analyzing eye movements and facial expressions.

[0639] "Autonomous driving mode" refers to a mode in which a vehicle drives autonomously, responding to road conditions and the surrounding environment without driver intervention.

[0640] "Notification Means" means the method used to communicate a suggestion or warning to the driver, including a visual display or audio notification.

[0641] "Feedback collection" is the process of collecting opinions and reactions from drivers, which allows for improvements to the system.

[0642] "Voice recognition technology" is a technology that converts voice data into text data, which allows the driver's verbal feedback to be analyzed.

[0643] "Communication in urban areas" refers to an automated vehicle's ability to communicate appropriately with other vehicles and pedestrians in urban areas, including through speech and gesture interaction.

[0644] overview

[0645] The safe driving support system of this invention aims to evaluate driving skills, monitor fatigue levels, collect feedback, appropriately transition to autonomous driving mode, and provide communication support in urban areas. The system collects data from multiple sensors installed in the vehicle and analyzes it with a generative AI model. Based on the analysis results, it makes appropriate suggestions to the driver and collects verbal feedback from the driver for use in retraining the system.

[0646] System Configuration

[0647] 1. Hardware Configuration

[0648] Smart glasses: Monitor the driver's eye movements and facial expressions and provide visual and audio notifications.

[0649] In-vehicle sensors: camera, lidar, GPS, fatigue monitoring sensors.

[0650] Server: A server for running generative AI models for data analysis.

[0651] 2. Software Configuration

[0652] AI Model: A generative AI model built using TensorFlow or PyTorch.

[0653] Speech recognition technology: We use Google Speech-to-Text API or equivalent technology.

[0654] Data communication: Data is transmitted between the smart glasses and the server via Wi-Fi or Bluetooth.

[0655] Process Overview

[0656] 1. Data Collection

[0657] The smart glasses and in-vehicle sensors collect real-time data, including the driver's eye movements, facial expressions, camera footage of the surrounding environment, distance information from LiDAR, and location information from GPS.

[0658] 2. Data Transmission

[0659] The collected data is sent to a server via Wi-Fi or Bluetooth.

[0660] 3. Analysis using AI models

[0661] The server inputs the received data into a generative AI model to evaluate the driver's fatigue level and driving skill. For example, if frequent blinking is observed, it is determined that the driver is highly fatigued.

[0662] 4. Proposals and Notifications

[0663] Based on the analysis, it determines whether a transition to autonomous driving mode is necessary and generates recommendations, which are communicated to the driver via visual and audio notifications via the smart glasses.

[0664] 5. Feedback Collection

[0665] Verbal feedback from the driver is collected by the microphone in the smart glasses and converted into text data using voice recognition technology, which is then sent to a server and used to retrain the system.

[0666] 6. Communication in urban areas

[0667] During autonomous driving, the system supports communication with pedestrians and other vehicles in urban areas. For example, it has a function to tell pedestrians "please go ahead."

[0668] Specific examples

[0669] When a driver is driving for a long period of time, the smart glasses monitor the driver's eye movements, and if frequent blinking is detected, a notification will be displayed saying, "Your level of fatigue is increasing. We recommend switching to autonomous driving mode."

[0670] When the driver issues a voice command to "switch to autonomous driving," the system recognizes the voice and switches to autonomous driving mode.

[0671] Prompt Sentence Examples

[0672] Please enter your driving data and fatigue evaluation data in the following format.

[0673] {

[0674] "Driving data": {

[0675] "Speed": "80",

[0676] "camera": "visual information data",

[0677] "lidar": "distance information data",

[0678] "GPS": "Location data"

[0679] },

[0680] "Fatigue Level": "High"

[0681] }

[0682] Generate appropriate suggestions.

[0683] {

[0684] "Suggestion": "We recommend switching to Autopilot mode",

[0685] "Notification Method": "Visual and Audio"

[0686] }

[0687] In this way, the invention can improve driver safety and comfort and provide reliable support during long drives and automated driving in urban areas.

[0688] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0689] Step 1: Data collection

[0690] The terminal collects data in real time from various sensors installed inside the vehicle. Specifically, the camera captures visual information about the vehicle's surroundings, the lidar captures distance information from objects, the GPS captures location information, and the smart glasses capture the driver's eye movements. This collected data is the initial input.

[0691] Step 2: Send data

[0692] The data collected by the device is sent to the server via Wi-Fi or Bluetooth. The input here is the sensor data sent from the device, and the output is the data received by the server.

[0693] Step 3: Data analysis

[0694] The server inputs the received sensor data into the generative AI model to evaluate the driver's driving skill and fatigue level. The input is sensor data, which is calculated and analyzed by the generative AI model. The output is an evaluation of the driver's driving skill and fatigue level.

[0695] Step 4: Proposal Generation

[0696] The server determines whether switching to autonomous driving mode is necessary based on the analysis results and generates a proposal. The input is the result of the data analysis, and the output is the proposal. This proposal may include content such as "We recommend switching to autonomous driving mode."

[0697] Step 5: Notification

[0698] The device communicates the suggestions generated by the server to the driver through the smart glasses with visual and audio notifications. The input is the suggestion from the server, and the output is the notification to the driver. Specifically, the display will say "We recommend switching to autonomous driving mode," and the same message will be conveyed via audio.

[0699] Step 6: Gather feedback

[0700] The user (driver) provides verbal feedback through the microphone in the smart glasses. For example, they give a voice command such as "switch to automatic driving." This becomes the input data. The device converts this voice feedback into text using speech recognition technology and sends it to the server. The output is a string of feedback data.

[0701] Step 7: Feedback analysis

[0702] The server analyzes the character string feedback data sent from the device and uses it to retrain the generative AI model. The input is the feedback data, and the output is the updated results of the model. Specifically, the parameters of the generative AI model are adjusted based on the driver's feedback.

[0703] Step 8: Communicating in urban areas

[0704] The device uses data from sensors to support communication with pedestrians and other vehicles while the vehicle is driving autonomously in urban areas. The input is data from lidar and cameras, which the server analyzes and generates appropriate responses such as "please go ahead." The output is a notification to the driver or pedestrian. For example, if a pedestrian is standing at a crosswalk, the device will notify them by voice, "please go ahead."

[0705] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0706] The safe driving support system of this invention uses in-vehicle sensors to collect driving data in real time and analyzes it using an AI generation model and emotion engine to evaluate the driver's driving skill, fatigue level, and emotional state. Based on the results of this evaluation, the system proposes an appropriate transition to autonomous driving mode and notifies the driver. Furthermore, verbal feedback from the driver is collected and used to retrain the AI ​​generation model, thereby ensuring continuous improvement of the system. Furthermore, when driving in urban areas, the system supports communication with surrounding people.

[0707] System Overview

[0708] Device:

[0709] The vehicle is equipped with multiple sensors, including cameras, lidar, GPS, fatigue monitoring sensors, and an emotion engine that detects the driver's emotional state. The device collects data from these sensors and transmits it to a server. For example, a sensor monitors the driver's eye movements to assess the driver's fatigue level, and facial expressions and voice tone to assess emotions.

[0710] server:

[0711] The server receives data sent from the device and inputs it into the generative AI model and emotion engine. The generative model analyzes the data and evaluates driving skill and fatigue level in real time. The emotion engine analyzes the driver's emotional state and detects stress and anxiety. Based on the results of this analysis, it determines whether switching to autonomous driving mode is necessary and generates a recommendation to do so. The server also receives feedback from the driver to help improve the system.

[0712] User:

[0713] The user is the driver and receives suggestions and warnings from the device. They can also provide verbal feedback to contribute to system improvements. For example, if a user receives a notification from the device while driving on a highway saying, "We recommend switching to autonomous driving mode," they can respond "yes" to switch to autonomous driving mode. If the user feels stressed while driving, the emotion engine can also return advice such as "relax."

[0714] Program processing

[0715] Sensor data collection

[0716] Device:

[0717] 1. Collect data in real time from various sensors on the vehicle.

[0718] Camera: Captures visual information about the surroundings.

[0719] Lidar: Obtains distance information to obstacles.

[0720] GPS: Get your current location.

[0721] Fatigue monitoring sensor: Detects the driver's eye movements and facial expressions.

[0722] Emotion Engine: Identifies the driver's emotional state from their tone of voice and facial expressions.

[0723] Data transmission and analysis

[0724] Device → Server:

[0725] 1. Collected sensor data is periodically sent to the server.

[0726] Data includes speed, frequency of brake use, steering angle, eye movements and voice tones.

[0727] server:

[0728] 1. Input the received data into the generative model and emotion engine.

[0729] 2. The generative model analyzes driving skill and fatigue level in real time.

[0730] For example, if a driver blinks frequently, the generative model will determine that the driver is highly fatigued.

[0731] 3. The emotion engine analyzes the driver's emotional state from their tone of voice and facial expressions.

[0732] For example, if a person's voice is trembling or their facial expression shows signs of tension, it is determined that they are feeling stressed.

[0733] Proposal generation and notification

[0734] server:

[0735] 1. Evaluate the driver's condition based on the analysis results.

[0736] For example, if the driver is highly fatigued or stressed, it is recommended to switch to autonomous driving mode.

[0737] 2. Generate a suggestion: "We recommend transitioning to autonomous driving mode."

[0738] Server → Device:

[0739] 1. Sending generated suggestions and warnings to the device.

[0740] Device:

[0741] 1. Notify the driver of suggestions via display and audio feedback.

[0742] For example, it may notify you, "Your stress level is high. We recommend switching to autonomous driving mode."

[0743] Processing verbal feedback

[0744] User:

[0745] 1. Providing verbal feedback on proposals and notifications.

[0746] For example, reply, "Yes, please."

[0747] Device:

[0748] 1. Convert verbal feedback into text data using speech recognition technology.

[0749] 2. Send the textual feedback to the server.

[0750] Feedback analysis and system improvement

[0751] server:

[0752] 1. Analyze the verbal feedback received.

[0753] 2. Based on the feedback results, the generative model and emotion engine are retrained to improve the system.

[0754] Communication in urban areas

[0755] Device → Server:

[0756] 1. Collecting data for speech and gesture recognition during autonomous driving in urban areas.

[0757] 2. Send the collected data to the server.

[0758] server:

[0759] 1. The generative model generates an appropriate response based on the received data.

[0760] For example, if a pedestrian raises their hand, the system understands their intention and determines the appropriate action to take.

[0761] 2. Send the generated responses and instructions to the terminal.

[0762] Device:

[0763] 1. Notify people around you of received instructions via voice or display.

[0764] For example, instructions such as "Please go ahead" are announced through the speaker.

[0765] In this way, by using the system of the present invention, it is possible to increase social acceptance of autonomous driving technology while ensuring the safety and comfort of drivers.

[0766] The processing flow will be explained below.

[0767] Program processing steps

[0768] Step 1: Initialize the sensor

[0769] Device:

[0770] 1. Launch an integrated system for autonomous vehicles.

[0771] 2. Initialize the vehicle's cameras, lidar, GPS, fatigue monitoring sensors, and emotion engine.

[0772] 3. Start collecting data from the sensors.

[0773] Step 2: Collect sensor data

[0774] Device:

[0775] 1. Collect data in real time from various sensors on the vehicle.

[0776] Camera: Captures visual information about the surroundings.

[0777] Lidar: Obtains distance information to obstacles and other vehicles.

[0778] GPS: Get your current location.

[0779] Fatigue monitoring sensor: Detects the driver's eye movements and facial expressions.

[0780] Emotion Engine: Identifies the driver's emotional state from their tone of voice and facial expressions.

[0781] Step 3: Sending data

[0782] Device → Server:

[0783] 1. Collected sensor data is sent to the server at regular intervals.

[0784] Data includes speed, frequency of brake use, steering angle, eye movements and voice tones.

[0785] Step 4: Data analysis

[0786] server:

[0787] 1. Input the received sensor data into the generative model and emotion engine.

[0788] 2. The generative model analyzes driving skill and fatigue level in real time.

[0789] Driving behavior patterns are analyzed and compared with normal driving behavior.

[0790] Detect signs of fatigue (e.g., frequent blinking).

[0791] 3. The emotion engine analyzes the driver's emotional state from their tone of voice and facial expressions.

[0792] Stress is detected when the voice tone is high and the voice is trembling.

[0793] If the facial muscles are tense, assess anxiety or tension.

[0794] Step 5: Generate proposals

[0795] server:

[0796] 1. Evaluate the driver's condition based on the analysis results.

[0797] For example, if the level of fatigue is high, it is determined that "the driver is fatigued."

[0798] Emotional states such as stress and tension are also assessed.

[0799] 2. Generate a proposal to transition to autonomous driving mode.

[0800] For example, create a message that reads, "Your fatigue and stress levels are high, so we recommend switching to autonomous driving mode."

[0801] Step 6: Proposal Notification

[0802] Server → Device:

[0803] 1. Send generated suggestions and warnings to the device.

[0804] Device:

[0805] 1. Notify the driver of suggestions via display and audio feedback.

[0806] For example, a voice notification may say, "You are highly fatigued, so we recommend switching to autonomous driving mode."

[0807] Step 7: Gathering human feedback

[0808] User:

[0809] 1. Providing verbal feedback on proposals and notifications.

[0810] For example, reply, "Yes, please."

[0811] Device:

[0812] 1. Convert verbal feedback into text data using speech recognition technology.

[0813] 2. Send the textual feedback to the server.

[0814] Step 8: Analyze feedback and improve the system

[0815] server:

[0816] 1. Analyze the verbal feedback received.

[0817] 2. Based on the feedback results, the generative model and emotion engine are retrained to improve the system.

[0818] For example, if the driver instructs the system to "switch to autonomous driving mode," the generative model is updated accordingly.

[0819] Step 9: Communicating in the city

[0820] Device → Server:

[0821] 1. Collecting data for speech and gesture recognition during autonomous driving in urban areas.

[0822] 2. Send the collected data to the server.

[0823] server:

[0824] 1. The generative model generates an appropriate response based on the received data.

[0825] For example, if a pedestrian raises their hand, the system understands their intention and determines the appropriate action to take.

[0826] 2. Send the generated responses and instructions to the terminal.

[0827] Device:

[0828] 1. Notify people around you of received instructions via voice or display.

[0829] For example, instructions such as "Please go ahead" are announced through the speaker.

[0830] Through these detailed steps, the system of the present invention can ensure driver safety and comfort and increase social acceptance of autonomous driving technology.

[0831] Example 2

[0832] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0833] Conventional driving assistance systems have struggled to assess the driver's driving skill, fatigue level, and emotional state in real time and provide appropriate driving assistance based on that assessment. Furthermore, there was a lack of a way to incorporate driver feedback into system improvements, making continuous system improvement difficult. Furthermore, they lacked a function to support communication with surrounding people when the vehicle is stopped in urban areas.

[0834] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0835] In this invention, the server includes means for collecting sensor data from the vehicle to evaluate driving skill, means for analyzing the sensor data and using a generative model to evaluate the driver's fatigue level, means for using an emotion engine to analyze the driver's emotional state, means for proposing transition to autonomous driving mode based on the analysis results, means for notifying the driver of the suggestion, means for converting verbal feedback from the driver into text data using speech recognition technology, means for collecting driver feedback and using it to improve the system, and means for supporting communication with surrounding people when the vehicle is stopped. This enables the driver's driving skill, fatigue level, and emotional state to be evaluated in real time, and appropriate driving assistance based on the evaluation. Furthermore, collecting feedback and using it to improve the system promotes continuous improvement. Furthermore, smooth communication with surrounding people when the vehicle is stopped in urban areas is possible.

[0836] "Sensor data" refers to data obtained from multiple sensors installed in a vehicle and used to evaluate the driver's driving skill, fatigue level, and emotional state.

[0837] A "generative model" is a machine learning algorithm that analyzes collected sensor data to evaluate a driver's driving skill and fatigue level.

[0838] The "emotion engine" is an algorithm that analyzes the driver's emotional state from their tone of voice and facial expressions, and identifies emotions such as stress and anxiety.

[0839] "Autonomous driving mode" is a mode in which the system controls the vehicle on behalf of the driver, and switches automatically depending on the driver's level of fatigue and emotional state.

[0840] "Suggestions" refer to instructions or advice that the system notifies the driver based on the analysis results, including, for example, a notification recommending switching to autonomous driving mode.

[0841] "Feedback" is the verbal or other responses or opinions from the driver that are collected and used to improve the system.

[0842] "Voice recognition technology" is a technology that converts the driver's voice into a digital signal and analyzes it as text data.

[0843] "System improvement" is the process of retraining the generative model and emotion engine based on collected feedback to improve the overall performance of the safe driving support system.

[0844] "Communication Support" is an assistance function that facilitates smooth interactions with people around the vehicle when stopped, and provides appropriate responses and instructions via voice and display.

[0845] This invention relates to a safe driving support system that uses in-vehicle sensors to collect driving data and analyzes it using a generative AI model and emotion engine to evaluate the driver's driving skill, fatigue level, and emotional state. Based on the analysis results, the system proposes an appropriate transition to an autonomous driving mode and notifies the driver. Furthermore, verbal feedback from the driver is collected and used to retrain the system, ensuring continuous improvement. Furthermore, when driving in urban areas, the system supports communication with surrounding people.

[0846] Hardware and software used

[0847] Hardware:

[0848] Camera: Captures visual information about the vehicle's surroundings.

[0849] Lidar: Obtains distance information to obstacles.

[0850] GPS: Get your current location.

[0851] Fatigue monitoring sensor: Detects eye movements and facial expressions.

[0852] Emotion Engine: Identify emotional states from voice tone and facial expressions.

[0853] software:

[0854] Generative AI model: Uses collected data to analyze driving skills and fatigue levels in real time.

[0855] Speech recognition technology: Analyzes verbal feedback and converts it into text data.

[0856] Explanation of program processing

[0857] Device:

[0858] Various sensors installed inside the vehicle collect data in real time and send it to a server. Specifically, the camera captures visual information about the surroundings, the lidar measures the distance to obstacles, and the GPS tracks location information. In addition, a fatigue monitoring sensor monitors the driver's eye movements and facial expressions, and an emotion engine analyzes the driver's emotional state from voice tone and facial expressions. For example, if the driver blinks frequently, it is determined that the driver is highly fatigued.

[0859] server:

[0860] The server receives sensor data sent from the device and inputs it into the generative AI model and emotion engine for analysis. The generative AI model evaluates driving skill and fatigue level, and the emotion engine analyzes the driver's emotional state. Based on the analysis results, it determines whether switching to autonomous driving mode is necessary and generates a proposal to do so. The proposal or warning is sent to the device and notified to the driver. If the driver answers "yes," the system switches to autonomous driving mode. In addition, verbal feedback from the driver is converted into text data using voice recognition technology and used to improve the system. For example, the system can switch to autonomous driving by replying "yes" to a notification that says, "We recommend switching to autonomous driving mode."

[0861] User:

[0862] The driver responds to the suggestions and warnings provided and provides verbal feedback. For example, if the driver receives a notification saying, "Your fatigue level is increasing. We recommend switching to autonomous driving mode," and responds, "Yes, please," the system will switch to autonomous driving mode. Additionally, when the vehicle is stopped in urban areas, the system will provide appropriate instructions to nearby pedestrians via voice, such as, "Please go ahead and let the vehicle pass."

[0863] Examples of concrete examples and prompts

[0864] Examples:

[0865] If the driver feels fatigued and blinks more frequently while driving on a highway, the system will issue a voice notification saying, "Your fatigue level is high, so we recommend switching to autonomous driving mode." If the driver replies "yes," the system will switch to autonomous driving mode.

[0866] Example prompt sentence:

[0867] "Analyze data from vehicle sensors as follows and suggest transitioning to autonomous driving mode: When the driver is highly fatigued and stressed."

[0868] As described above, the safe driving support system of the present invention can provide advanced support for driver safety and comfort by combining a wide range of sensors with an advanced AI model and emotion engine.

[0869] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0870] Step 1:

[0871] Sensor data collection

[0872] Device: Collects data in real time from various sensors installed in the vehicle. Specifically, the camera captures visual information of the surroundings, the lidar measures the distance to obstacles, and the GPS acquires the current location information. The fatigue monitoring sensor monitors the driver's eye movements and facial expressions, and the emotion engine analyzes the driver's emotional state from voice tone and facial expressions.

[0873] Inputs: Data from camera, lidar, GPS, fatigue monitoring sensors, emotion engine.

[0874] Output: Camera visual information, rider distance information, GPS location information, fatigue level information, emotional state information.

[0875] Step 2:

[0876] Sending collected data

[0877] Device → Server: Collected sensor data is periodically sent to the server, including speed information, braking frequency, steering angle, eye movements, and audio tones.

[0878] Input: Data collected from various sensors.

[0879] Output: Sensor data sent to the server.

[0880] Step 3:

[0881] Data analysis

[0882] Server: The received data is input into the generative AI model and emotion engine to analyze driving skill and fatigue level. The generative AI model evaluates driving skill and determines fatigue level based on factors such as frequent blinking. The emotion engine analyzes the driver's emotional state based on voice tone and facial expressions. For example, if the voice is trembling, it is interpreted as being stressed.

[0883] Input: Sensor data sent to the server.

[0884] Output: Driving skill evaluation results, fatigue evaluation results, emotional state evaluation results.

[0885] Step 4:

[0886] Proposal Generation

[0887] Server: Based on the analysis results, evaluates the driver's condition and determines whether switching to autonomous driving mode is necessary. For example, if the driver's fatigue level or stress level is high, a recommendation is generated to "recommend switching to autonomous driving mode."

[0888] Input: Driving skill evaluation results, fatigue evaluation results, emotional state evaluation results.

[0889] Output: Proposal to transition to autonomous driving mode.

[0890] Step 5:

[0891] Proposal Notification

[0892] Server → Device: Sends generated suggestions and warnings to the device.

[0893] Device: Notify the driver of the suggestion through a display or audio feedback, for example, "Your stress level is high. We recommend switching to autonomous driving mode."

[0894] Input: Proposal to transition to autonomous driving mode.

[0895] Output: Display of suggestion notification and audio feedback.

[0896] Step 6:

[0897] Processing Feedback

[0898] User: The driver provides verbal feedback to the suggestion or notification, for example, by replying "Yes, please."

[0899] Terminal: Verbal feedback is converted into text data using speech recognition technology and sent to the server.

[0900] Input: User's verbal feedback.

[0901] Output: Feedback converted to text data.

[0902] Step 7:

[0903] Feedback analysis and system improvement

[0904] Server: Analyzes the received verbal feedback and retrains the generative model and emotion engine to improve the system.

[0905] Input: Feedback converted to text data.

[0906] Output: Improved generative model and emotion engine.

[0907] Step 8:

[0908] Communication support in urban areas

[0909] Terminal → Server: Collects data to recognize speech and gestures during autonomous driving in urban areas and sends it to the server.

[0910] Server: The generative model generates an appropriate response based on the received data. For example, if a pedestrian raises their hand, the server understands their intention and determines the appropriate action. The server then sends the generated response and instructions to the device.

[0911] Terminal: The terminal notifies people around it of the received instructions through voice or display. For example, it may announce the instruction "Please go ahead" through a speaker.

[0912] Input: Speech and gesture data.

[0913] Output: Audio and display notifications for people around you.

[0914] (Application example 2)

[0915] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0916] Conventional driving support systems have difficulty accurately analyzing the driver's fatigue level and emotional state in real time, resulting in an inability to suggest transitioning to autonomous driving mode at the appropriate time. Effectively collecting feedback from the driver and utilizing it for continuous system improvement has also been a challenge. Furthermore, smooth communication with surrounding people in urban areas is also required.

[0917] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for collecting the driver's facial expressions, voice tone, and driving behavior data in real time using a smartphone; means for transmitting the collected data to the server and analyzing it using an AI generation model and an emotion engine; means for notifying the driver via audio or visual means on the smartphone based on the analysis results; means for converting the driver's verbal feedback into text data using voice recognition technology and transmitting it to the server; and means for supporting communication with surrounding people while driving. This makes it possible to accurately grasp the driver's fatigue level and emotional state and suggest switching to autonomous driving mode at the appropriate time. Furthermore, it is possible to effectively collect driver feedback and achieve continuous system improvement. Furthermore, it also enables smooth communication in urban areas.

[0918] A "driving skill assessment" is an assessment of how safely and efficiently a driver can drive.

[0919] "Sensor data" refers to information collected from various sensors, including camera footage, audio, acceleration, and location information.

[0920] A "generative model" is an algorithm that uses AI technology to analyze data for a specific purpose and predict an outcome.

[0921] "Fatigue assessment" is the process of measuring how tired a driver is.

[0922] "Autonomous driving mode" refers to a mode in which the vehicle performs driving operations autonomously.

[0923] "Notification" refers to the act of the system informing the driver of information, and includes audio and visual notifications.

[0924] "Feedback" refers to the opinions and reactions provided by the driver to the system.

[0925] "System improvement" means improving the performance and accuracy of the system based on collected feedback and data.

[0926] A "smartphone" is a device that combines advanced computing power and connectivity with a mobile phone.

[0927] "Facial expression" refers to the facial state that is revealed by the movement of the driver's facial muscles.

[0928] "Voice tone" refers to the pitch, strength, and emotional characteristics of a driver's voice.

[0929] "Driving behavior data" refers to data related to the driver's driving operations, including acceleration, deceleration, braking, etc.

[0930] The "emotion engine" is an algorithm that analyzes voice tone and facial expressions to detect the driver's emotional state.

[0931] "Speech recognition technology" refers to the technology that converts voice into text data.

[0932] "Communication in urban areas" refers to smoothly exchanging information with surrounding pedestrians and other vehicles while driving.

[0933] The safe driving support system of the present invention evaluates the driver's driving skill, fatigue level, and emotional state, and suggests transitioning to autonomous driving mode at an appropriate time. This system uses a smartphone, a server, and various sensors.

[0934] Hardware Configuration

[0935] 1. Smartphone: A smartphone equipped with a camera, microphone, accelerometer, and GPS is used to collect the driver's facial expressions, voice tone, and driving behavior data in real time.

[0936] 2. Server: A server equipped with a high-performance GPU is used. The server receives data sent from the smartphone and analyzes it using a generative AI model and emotion engine.

[0937] 3. Communication network: Data is sent and received using communication networks such as Wi-Fi and 4G / 5G.

[0938] Software Configuration

[0939] 1. Smartphone app: Install the app that collects driving data, sends the data to a server, notifies the driver of the analysis results, and collects driver feedback. This app works on Android or iOS.

[0940] 2. Speech Recognition System: Use speech recognition technology, such as Google Cloud Speech-to-Text, to convert the driver's verbal feedback into text data.

[0941] 3. Generative AI model and emotion engine: This model will be equipped with algorithms to analyze driving data and evaluate driving skill, fatigue level, and emotional state. These algorithms will be implemented using machine learning libraries such as PyTorch and TensorFlow.

[0942] System Operation

[0943] The server receives driving data sent from the smartphone. The received data is analyzed using a generative AI model and an emotion engine. This analysis evaluates the driver's driving skill, fatigue level, and emotional state. For example, if the server determines that the driver is very tired, it generates a notification suggesting, "Do you want to switch to autonomous driving mode?" This notification is communicated to the driver via audio or visual means via the smartphone.

[0944] Driver feedback is also important. When the driver responds verbally to suggestions and notifications, the smartphone's voice recognition system converts the speech into text data and sends it to the server. The server analyzes the feedback and uses the results to retrain the generative AI model and emotion engine, ensuring continuous improvement of the system.

[0945] To facilitate smooth communication in urban areas while driving, the system also sends voice messages to enable drivers to safely and smoothly exchange information with those around them, and this function also supports communication with pedestrians and other vehicles.

[0946] Examples and prompts

[0947] As a concrete example, if a driver continues driving for a long period of time, their facial expressions are recorded by a camera mounted on their smartphone. The generative AI model determines that the driver is highly fatigued from their facial expression and issues a message saying, "You seem tired. Would you like to switch to autonomous driving mode?" If the driver replies "yes," the voice recognition system converts the speech into text data and sends it to a server. Based on this feedback, the system switches to autonomous driving mode.

[0948] Example prompt sentence:

[0949] "The system analyzes the driver's emotional state from their facial expressions and vocal tone to assess their driving skill and fatigue level. If it determines that their fatigue level is high, it notifies the driver, 'Do you want to switch to autonomous driving mode?' and receives verbal feedback. It communicates with the server and uses the driver's feedback to improve the system."

[0950] This will improve driver safety and comfort, and also increase social acceptance of autonomous driving technology.

[0951] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0952] Step 1:

[0953] The device (smartphone) uses a camera, microphone, acceleration sensor, and GPS to collect the driver's facial expressions, voice tone, and driving behavior data in real time while driving. Inputs include the driver's facial image, voice, acceleration data, and location information. This data is collected from various sensors and stored on the device.

[0954] Step 2:

[0955] The terminal periodically transmits the collected data to the server. The driving data collected by the terminal is used as input, and the data is transmitted to the server as output. This transmission is performed using a communication network (Wi-Fi, 4G / 5G).

[0956] Step 3:

[0957] The server receives the data sent from the device and inputs it into the generative AI model and emotion engine. The input includes driving data (facial images, audio, driving behavior data, and location information). The server passes the received data to the AI ​​model and begins analysis.

[0958] Step 4:

[0959] The generative AI model analyzes driving data and evaluates driving skill and fatigue level in real time. Facial images and driving behavior data are used as input, and the output is an evaluation result of driving skill and fatigue level. For example, if a driver blinks frequently, the model will determine that the driver is highly fatigued.

[0960] Step 5:

[0961] The emotion engine analyzes the driver's emotional state from their voice tone and facial expression. It uses voice data and facial images as input, and generates an evaluation result of the driver's emotional state as output. For example, if the voice tone is unstable, the server determines that the driver is stressed.

[0962] Step 6:

[0963] The server evaluates the driver's state based on the analysis results of the generative AI model and emotion engine. The input includes the evaluation results of driving skill, fatigue level, and emotional state. The output generates a proposal to switch to autonomous driving mode if necessary. For example, a proposal such as "Do you want to switch to autonomous driving mode?" is generated.

[0964] Step 7:

[0965] The server sends the generated proposal to the terminal. The proposal content is used as input, and the proposal is sent to the terminal as output. The terminal notifies the driver of this proposal by voice or visual means. Specifically, the smartphone speaker will say, "Do you want to switch to autonomous driving mode?"

[0966] Step 8:

[0967] The user (driver) provides verbal feedback in response to the notification from the terminal. The driver's voice is used as input, and the voice data is collected by the terminal as output. For example, the driver replies "Yes."

[0968] Step 9:

[0969] The terminal converts the collected voice data into text data using a voice recognition system. The driver's voice data is used as input, and text data is generated as output. This converts the voice "yes" into the text "yes."

[0970] Step 10:

[0971] The device sends textual feedback to the server. Text data is used as input, and feedback is sent to the server as output. The server analyzes the received feedback and retrains the AI ​​model and emotion engine based on the results to improve the system. Specifically, data on how the driver responded to each notification is accumulated.

[0972] Step 11:

[0973] During autonomous driving in urban areas, the server and terminal collect data to communicate with surrounding pedestrians and other vehicles and send it to the server. Surrounding audio and image data are used as input, and sent to the server as output. The server generates an appropriate response based on the received data and sends it to the terminal. The terminal notifies surrounding people of the generated response by voice or display. For example, an instruction such as "Please go ahead" may be issued through the speaker.

[0974] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0975] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0976] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0977] [Third embodiment]

[0978] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0979] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0980] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0981] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0982] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0983] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0984] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0985] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0986] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0987] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0988] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0989] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0990] The safe driving support system of this invention collects driving data in real time from sensors installed in the vehicle and analyzes it using an AI model to evaluate the driver's driving skill and fatigue level. Based on the results of this evaluation, it proposes an appropriate transition to autonomous driving mode and notifies the driver. It also collects verbal feedback from the driver and uses it to retrain the AI ​​model, thereby ensuring continuous improvement of the system. Furthermore, it supports communication with surrounding people when driving in urban areas.

[0991] System Overview

[0992] Device:

[0993] The vehicle is equipped with multiple sensors, including cameras, lidar, GPS, and fatigue monitoring sensors. The device collects data from these sensors and transmits it to a server. For example, a sensor that monitors the driver's eye movements provides important information for assessing the driver's fatigue level.

[0994] server:

[0995] The server receives data sent from the device and inputs it into the generative model. The generative model uses AI algorithms to analyze driving skill and fatigue level in real time. Based on the analysis results, it determines whether switching to autonomous driving mode is necessary and generates a recommendation. The server also receives feedback from the driver to help improve the system.

[0996] User:

[0997] The user is the driver and receives suggestions and warnings from the device. They also provide verbal feedback to contribute to system improvement. For example, after driving for a long time on a highway, the user may receive a notification from the device saying, "We recommend switching to autonomous driving mode." By responding "yes," the user can switch to autonomous driving mode.

[0998] Program processing

[0999] The processing contents of the program will be specifically explained below.

[1000] Sensor data collection:

[1001] The device collects real-time data from various sensors in the vehicle. For example, camera data provides visual information, lidar data provides distance information to surrounding objects, and fatigue monitoring sensors monitor the driver's facial expressions and eye movements to assess fatigue level.

[1002] Data transmission and analysis:

[1003] The device sends the collected data to a server. The server inputs the received data into a generative model for analysis. This analysis evaluates the driver's driving skill and fatigue level. For example, if the driver blinks frequently, the generative model will determine that the driver is highly fatigued.

[1004] Proposal generation and notification:

[1005] The server generates suggestions and warnings and sends them to the device, which then notifies the driver via audio or visual notification. For example, a notification might say, "Your fatigue level is increasing. We recommend switching to autonomous driving mode."

[1006] Processing verbal feedback:

[1007] The device uses voice recognition technology to convert verbal feedback from the driver into text data and sends it to the server. The server analyzes this feedback and uses it to retrain the generative model. For example, if the driver verbally commands, "Please switch to autonomous driving mode," the system converts this into text data and sends it to the server.

[1008] Communication in urban areas:

[1009] During autonomous driving in cities, the system recognizes speech and gestures and provides appropriate responses through generative models, such as an audio message to pedestrians about to cross the street saying, "Please go ahead."

[1010] In this way, by using the system of the present invention, it is possible to increase social acceptance of autonomous driving technology while ensuring driver safety.

[1011] The processing flow will be explained below.

[1012] Program processing steps

[1013] Step 1: Initialize the sensor

[1014] Device:

[1015] 1. Launch an integrated system for autonomous vehicles.

[1016] 2. Initialize the vehicle's camera, lidar, GPS, and fatigue monitoring sensors.

[1017] 3. Start collecting data from the sensors.

[1018] Step 2: Collect sensor data

[1019] Device:

[1020] 1. Collect data in real time from each sensor on the vehicle.

[1021] Camera: Captures visual information about the surroundings.

[1022] Lidar: Obtains distance information to obstacles.

[1023] GPS: Get your current location.

[1024] Fatigue monitoring sensor: Monitors the driver's eye movements and facial expressions.

[1025] Step 3: Sending data

[1026] Device → Server:

[1027] 1. Collected sensor data is periodically sent to a server via batch processing or real-time streaming.

[1028] 2. Data includes speed, frequency of braking, steering angle, eye movements, etc.

[1029] Step 4: Data analysis

[1030] server:

[1031] 1. Input the received sensor data into the generative model.

[1032] 2. A generative model analyzes the data and assesses driving skill and fatigue in real time.

[1033] The analysis results include detecting driving behavior patterns and signs of fatigue.

[1034] Step 5: Generate proposals

[1035] server:

[1036] 1. Evaluate the driver's condition based on the analysis results.

[1037] For example, if the fatigue level is high, the system will determine that the fatigue level is high.

[1038] 2. Generate a proposal to transition to autonomous driving mode.

[1039] For example, it generates a message saying, "We recommend switching to autonomous driving mode."

[1040] Step 6: Proposal Notification

[1041] Server → Device:

[1042] 1. Send generated suggestions and warnings to the device.

[1043] Device:

[1044] 1. Notify the driver of suggestions via display and audio feedback.

[1045] Step 7: Gathering human feedback

[1046] User:

[1047] 1. Providing verbal feedback on proposals and notifications.

[1048] For example, reply, "Yes, please."

[1049] Device:

[1050] 1. Convert verbal feedback into text data using speech recognition technology.

[1051] 2. Send the textual feedback to the server.

[1052] Step 8: Analyze feedback and improve the system

[1053] server:

[1054] 1. Analyze the verbal feedback received.

[1055] 2. Retrain the generative model based on the feedback results and improve the system.

[1056] Step 9: Communicating in the city

[1057] Device → Server:

[1058] 1. Collecting data to recognize speech and gestures while driving autonomously around town.

[1059] 2. Send the collected data to the server.

[1060] server:

[1061] 1. The generative model generates an appropriate response based on the received data.

[1062] For example, if a pedestrian raises their hand, the system understands their intention and determines the appropriate action to take.

[1063] 2. Send the generated responses and instructions to the terminal.

[1064] Device:

[1065] 1. Notify people around you of received instructions via voice or display.

[1066] For example, instructions such as "Please go ahead" are announced through the speaker.

[1067] Through these detailed steps, the system of the present invention can ensure driver safety and increase social acceptance of autonomous driving technology.

[1068] Example 1

[1069] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1070] In modern society, long-distance driving and driving in urban areas place a heavy burden on drivers, and the risk of accidents increases as fatigue accumulates. To solve these problems and achieve safe and efficient driving, a system is needed that monitors the driver's condition in real time and suggests switching to autonomous driving mode as necessary. It is also important to be able to communicate smoothly with people around you in the city.

[1071] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1072] In this invention, the server includes a means for collecting sensor data from the vehicle to evaluate driving skills, a means for analyzing the sensor data and using a generative model to evaluate the driver's fatigue level, a means for proposing switching to autonomous driving mode based on the analysis results, a means for collecting user feedback to help improve the system, and a means for recognizing speech and gestures and providing a response based on the surrounding situation. This makes it possible to accurately grasp the driver's fatigue level and propose switching to autonomous driving mode when necessary. In addition, appropriate responses can be provided even in urban areas, facilitating communication with surrounding people.

[1073] "Sensor data" refers to information collected by various sensors to understand the situation inside and around the vehicle.

[1074] "Driving ability assessment" refers to the measurement and analysis of a driver's driving ability and skills.

[1075] "Fatigue assessment" is the process of monitoring and measuring the driver's fatigue state.

[1076] A "generative model" is an AI algorithm that learns from large amounts of data and analyzes and evaluates driving skills and fatigue levels.

[1077] "Autonomous driving mode" refers to a mode in which the vehicle automatically performs driving operations, minimizing driver involvement.

[1078] "Suggestion" refers to notifications to the driver to change their driving mode or to issue warnings based on the analysis results.

[1079] "Notification" is the act of conveying information to the driver through audio or visual means.

[1080] "Feedback" refers to the evaluations and opinions that the driver provides to the system, which are used to improve the system.

[1081] "Speech recognition" refers to the technology that uses voice recognition technology to understand and process the driver's verbal instructions and feedback.

[1082] "Gesture recognition" is a technology that detects gestures and hand movements made by drivers and pedestrians, and enables the system to provide an appropriate response based on those gestures.

[1083] The safe driving support system of this invention is a system that evaluates the driver's driving skill and fatigue level by collecting driving data in real time from various sensors installed in the vehicle and analyzing it based on an AI-generated model. The system consists of the following components.

[1084] Hardware Configuration

[1085] The device collects data from multiple sensors installed in the vehicle and transmits it to a server. These include a camera, lidar, GPS, and a fatigue monitoring sensor. For example, the camera monitors the driver's face and road conditions, while the lidar measures the distance to surrounding objects. The fatigue monitoring sensor monitors the driver's facial expressions and eye movements and evaluates their fatigue level based on the data.

[1086] Software Configuration

[1087] The server receives the data sent from the device and analyzes it in real time using an AI generative model. A generative model is an AI algorithm that has been trained on a large amount of driving data in advance. Specifically, the server analyzes camera data using an image recognition algorithm (for example, face detection using deep learning) to evaluate the driver's fatigue level. Lidar data and GPS data are also integrated to evaluate the driving environment and road conditions.

[1088] System operation example

[1089] For example, if camera data shows that a driver is blinking frequently after driving for a long period of time, the server's generative model will assess the driver's fatigue level as high. Based on this result, the server will generate a recommendation such as "Your fatigue level is increasing. We recommend switching to autonomous driving mode," and send it to the device. The device will then notify the driver of this recommendation via voice and display.

[1090] The driver provides verbal feedback, such as "Please switch to autonomous driving mode." This verbal feedback is captured by a microphone installed on the device and converted into text data using voice recognition technology. The server receives this text data and uses it to retrain the system. In this way, the system is continuously improved.

[1091] During autonomous driving in urban areas, the device understands the surrounding situation based on data from lidar and cameras. For example, if a pedestrian is detected approaching a crosswalk, the generative model will suggest an appropriate voice message such as "Please go ahead," and the device will notify the driver by voice.

[1092] Prompt Sentence Examples

[1093] "Please explain the process of the program that generates a notification suggesting switching to autonomous driving mode based on fatigue assessment after a long period of driving."

[1094] "Please tell me about the function of the system that allows vehicles traveling in urban areas to provide appropriate audio notifications to pedestrians."

[1095] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1096] Step 1: Collect sensor data

[1097] The device collects data in real time from multiple sensors installed on the vehicle. The inputs are raw data from cameras, lidar, GPS, and fatigue monitoring sensors. These data include cameras capturing images of the driver's face and road conditions, lidar measuring the distance to surrounding objects, GPS acquiring location information, and fatigue monitoring sensors monitoring the driver's facial expressions and eye movements. As an output, these data are shaped into a single integrated data stream, ready to be sent to a server.

[1098] Step 2: Send and analyze data

[1099] The device sends the collected sensor data to the server. The input is the integrated data stream collected and shaped in step 1. The server receives this data stream and inputs it into the generative AI model. The generative AI model uses image recognition algorithms and pattern recognition algorithms to analyze the driver's fatigue level from the camera data and evaluate the driving environment and road conditions from other sensor data. The output is an evaluation of the driver's fatigue level and driving skills.

[1100] Step 3: Proposal generation and notification

[1101] The server generates suggestions and warnings based on the analysis results of step 2. The input is data on the driver's fatigue level assessment and driving skill assessment. The server generates suggestions and warning messages such as "Your fatigue level is increasing. We recommend switching to autonomous driving mode." As output, these messages are sent to the terminal. The terminal notifies the driver of this suggestion via voice or display. Specifically, a warning message is displayed on the display and voice guidance is played from the speaker.

[1102] Step 4: Processing verbal feedback

[1103] The user provides verbal feedback in response to notifications and suggestions from the device. The input is the driver's verbal instruction (e.g., "Please switch to autonomous driving mode"). The device captures this voice feedback and converts it into text data using voice recognition technology. As output, this text data is sent to the server and used for retraining the system. Specifically, the microphone captures the voice, and the internal voice recognition module converts it into text.

[1104] Step 5: Communicating in urban areas

[1105] During autonomous driving in urban areas, the device understands the surrounding situation based on data from the lidar and camera. The input is real-time data from the lidar and camera. For example, if a pedestrian is detected when approaching a crosswalk, the generative AI model will suggest an appropriate voice message such as "Please go ahead." The device will notify the driver of this suggestion by voice. The output is appropriate communication with the driver and pedestrian. In concrete terms, the camera and lidar scan the surrounding situation, and the speaker plays the voice message.

[1106] (Application example 1)

[1107] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1108] Autonomous vehicles are required to accurately assess the driver's fatigue level and driving skills in real time and make appropriate suggestions based on the assessment. However, conventional safe driving support systems are unable to monitor the driver's eye movements and facial expressions in detail, making it difficult to accurately grasp the driver's condition. Furthermore, the lack of effective means to collect and analyze feedback from the driver makes continuous system improvement difficult. Furthermore, there is an issue of being unable to effectively support communication with humans in urban areas during autonomous driving.

[1109] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1110] In this invention, the server includes means for collecting sensor data from the vehicle to evaluate driving skills, means for analyzing the sensor data and using a generative model to evaluate the driver's fatigue level, means for monitoring the driver's eye movements and facial expressions and evaluating them in real time, means for proposing switching to autonomous driving mode based on the analysis results, means for notifying the driver of the suggestion, means for collecting verbal feedback from the driver and converting it into text data using speech recognition technology, means for analyzing the converted text data and using it to improve the system, and means for recognizing speech and gestures during autonomous driving in urban areas and providing appropriate responses. This enables the system to accurately grasp the driver's condition, make appropriate suggestions, and effectively collect and analyze feedback from the driver, enabling continuous system improvement and communication support in urban areas.

[1111] "Driving skill assessment" is the process of measuring and assessing a driver's level of driving skill and ability.

[1112] "Sensor data" refers to information obtained from various sensors installed on the vehicle, including camera images, distance data from LIDAR, and GPS information.

[1113] A "generative model" is an artificial intelligence (AI) algorithm used to analyze collected data and assess the driver's state.

[1114] "Fatigue assessment" is the measurement and evaluation of a driver's level of fatigue, usually by analyzing eye movements and facial expressions.

[1115] "Autonomous driving mode" refers to a mode in which a vehicle drives autonomously, responding to road conditions and the surrounding environment without driver intervention.

[1116] "Notification Means" means the method used to communicate a suggestion or warning to the driver, including a visual display or audio notification.

[1117] "Feedback collection" is the process of collecting opinions and reactions from drivers, which allows for improvements to the system.

[1118] "Voice recognition technology" is a technology that converts voice data into text data, which allows the driver's verbal feedback to be analyzed.

[1119] "Communication in urban areas" refers to an automated vehicle's ability to communicate appropriately with other vehicles and pedestrians in urban areas, including through speech and gesture interaction.

[1120] overview

[1121] The safe driving support system of this invention aims to evaluate driving skills, monitor fatigue levels, collect feedback, appropriately transition to autonomous driving mode, and provide communication support in urban areas. The system collects data from multiple sensors installed in the vehicle and analyzes it with a generative AI model. Based on the analysis results, it makes appropriate suggestions to the driver and collects verbal feedback from the driver for use in retraining the system.

[1122] System Configuration

[1123] 1. Hardware Configuration

[1124] Smart glasses: Monitor the driver's eye movements and facial expressions and provide visual and audio notifications.

[1125] In-vehicle sensors: camera, lidar, GPS, fatigue monitoring sensors.

[1126] Server: A server for running generative AI models for data analysis.

[1127] 2. Software Configuration

[1128] AI Model: A generative AI model built using TensorFlow or PyTorch.

[1129] Speech recognition technology: We use Google Speech-to-Text API or equivalent technology.

[1130] Data communication: Data is transmitted between the smart glasses and the server via Wi-Fi or Bluetooth.

[1131] Process Overview

[1132] 1. Data Collection

[1133] The smart glasses and in-vehicle sensors collect real-time data, including the driver's eye movements, facial expressions, camera footage of the surrounding environment, distance information from LiDAR, and location information from GPS.

[1134] 2. Data Transmission

[1135] The collected data is sent to a server via Wi-Fi or Bluetooth.

[1136] 3. Analysis using AI models

[1137] The server inputs the received data into a generative AI model to evaluate the driver's fatigue level and driving skill. For example, if frequent blinking is observed, it is determined that the driver is highly fatigued.

[1138] 4. Proposals and Notifications

[1139] Based on the analysis, it determines whether a transition to autonomous driving mode is necessary and generates recommendations, which are communicated to the driver via visual and audio notifications via the smart glasses.

[1140] 5. Feedback Collection

[1141] Verbal feedback from the driver is collected by the microphone in the smart glasses and converted into text data using voice recognition technology, which is then sent to a server and used to retrain the system.

[1142] 6. Communication in urban areas

[1143] During autonomous driving, the system supports communication with pedestrians and other vehicles in urban areas. For example, it has a function to tell pedestrians "please go ahead."

[1144] Specific examples

[1145] When a driver is driving for a long period of time, the smart glasses monitor the driver's eye movements, and if frequent blinking is detected, a notification will be displayed saying, "Your level of fatigue is increasing. We recommend switching to autonomous driving mode."

[1146] When the driver issues a voice command to "switch to autonomous driving," the system recognizes the voice and switches to autonomous driving mode.

[1147] Prompt Sentence Examples

[1148] Please enter your driving data and fatigue evaluation data in the following format.

[1149] {

[1150] "Driving data": {

[1151] "Speed": "80",

[1152] "camera": "visual information data",

[1153] "lidar": "distance information data",

[1154] "GPS": "Location data"

[1155] },

[1156] "Fatigue Level": "High"

[1157] }

[1158] Generate appropriate suggestions.

[1159] {

[1160] "Suggestion": "We recommend switching to Autopilot mode",

[1161] "Notification Method": "Visual and Audio"

[1162] }

[1163] In this way, the invention can improve driver safety and comfort and provide reliable support during long drives and automated driving in urban areas.

[1164] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1165] Step 1: Data collection

[1166] The terminal collects data in real time from various sensors installed inside the vehicle. Specifically, the camera captures visual information about the vehicle's surroundings, the lidar captures distance information from objects, the GPS captures location information, and the smart glasses capture the driver's eye movements. This collected data is the initial input.

[1167] Step 2: Send data

[1168] The data collected by the device is sent to the server via Wi-Fi or Bluetooth. The input here is the sensor data sent from the device, and the output is the data received by the server.

[1169] Step 3: Data analysis

[1170] The server inputs the received sensor data into the generative AI model to evaluate the driver's driving skill and fatigue level. The input is sensor data, which is calculated and analyzed by the generative AI model. The output is an evaluation of the driver's driving skill and fatigue level.

[1171] Step 4: Proposal Generation

[1172] The server determines whether switching to autonomous driving mode is necessary based on the analysis results and generates a proposal. The input is the result of the data analysis, and the output is the proposal. This proposal may include content such as "We recommend switching to autonomous driving mode."

[1173] Step 5: Notification

[1174] The device communicates the suggestions generated by the server to the driver through the smart glasses with visual and audio notifications. The input is the suggestion from the server, and the output is the notification to the driver. Specifically, the display will say "We recommend switching to autonomous driving mode," and the same message will be conveyed via audio.

[1175] Step 6: Gather feedback

[1176] The user (driver) provides verbal feedback through the microphone in the smart glasses. For example, they give a voice command such as "switch to automatic driving." This becomes the input data. The device converts this voice feedback into text using speech recognition technology and sends it to the server. The output is a string of feedback data.

[1177] Step 7: Feedback analysis

[1178] The server analyzes the character string feedback data sent from the device and uses it to retrain the generative AI model. The input is the feedback data, and the output is the updated results of the model. Specifically, the parameters of the generative AI model are adjusted based on the driver's feedback.

[1179] Step 8: Communicating in urban areas

[1180] The device uses data from sensors to support communication with pedestrians and other vehicles while the vehicle is driving autonomously in urban areas. The input is data from lidar and cameras, which the server analyzes and generates appropriate responses such as "please go ahead." The output is a notification to the driver or pedestrian. For example, if a pedestrian is standing at a crosswalk, the device will notify them by voice, "please go ahead."

[1181] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1182] The safe driving support system of this invention uses in-vehicle sensors to collect driving data in real time and analyzes it using an AI generation model and emotion engine to evaluate the driver's driving skill, fatigue level, and emotional state. Based on the results of this evaluation, the system proposes an appropriate transition to autonomous driving mode and notifies the driver. Furthermore, verbal feedback from the driver is collected and used to retrain the AI ​​generation model, thereby ensuring continuous improvement of the system. Furthermore, when driving in urban areas, the system supports communication with surrounding people.

[1183] System Overview

[1184] Device:

[1185] The vehicle is equipped with multiple sensors, including cameras, lidar, GPS, fatigue monitoring sensors, and an emotion engine that detects the driver's emotional state. The device collects data from these sensors and transmits it to a server. For example, a sensor monitors the driver's eye movements to assess the driver's fatigue level, and facial expressions and voice tone to assess emotions.

[1186] server:

[1187] The server receives data sent from the device and inputs it into the generative AI model and emotion engine. The generative model analyzes the data and evaluates driving skill and fatigue level in real time. The emotion engine analyzes the driver's emotional state and detects stress and anxiety. Based on the results of this analysis, it determines whether switching to autonomous driving mode is necessary and generates a recommendation to do so. The server also receives feedback from the driver to help improve the system.

[1188] User:

[1189] The user is the driver and receives suggestions and warnings from the device. They can also provide verbal feedback to contribute to system improvements. For example, if a user receives a notification from the device while driving on a highway saying, "We recommend switching to autonomous driving mode," they can respond "yes" to switch to autonomous driving mode. If the user feels stressed while driving, the emotion engine can also return advice such as "relax."

[1190] Program processing

[1191] Sensor data collection

[1192] Device:

[1193] 1. Collect data in real time from various sensors on the vehicle.

[1194] Camera: Captures visual information about the surroundings.

[1195] Lidar: Obtains distance information to obstacles.

[1196] GPS: Get your current location.

[1197] Fatigue monitoring sensor: Detects the driver's eye movements and facial expressions.

[1198] Emotion Engine: Identifies the driver's emotional state from their tone of voice and facial expressions.

[1199] Data transmission and analysis

[1200] Device → Server:

[1201] 1. Collected sensor data is periodically sent to the server.

[1202] Data includes speed, frequency of brake use, steering angle, eye movements and voice tones.

[1203] server:

[1204] 1. Input the received data into the generative model and emotion engine.

[1205] 2. The generative model analyzes driving skill and fatigue level in real time.

[1206] For example, if a driver blinks frequently, the generative model will determine that the driver is highly fatigued.

[1207] 3. The emotion engine analyzes the driver's emotional state from their tone of voice and facial expressions.

[1208] For example, if a person's voice is trembling or their facial expression shows signs of tension, it is determined that they are feeling stressed.

[1209] Proposal generation and notification

[1210] server:

[1211] 1. Evaluate the driver's condition based on the analysis results.

[1212] For example, if the driver is highly fatigued or stressed, it is recommended to switch to autonomous driving mode.

[1213] 2. Generate a suggestion: "We recommend transitioning to autonomous driving mode."

[1214] Server → Device:

[1215] 1. Sending generated suggestions and warnings to the device.

[1216] Device:

[1217] 1. Notify the driver of suggestions via display and audio feedback.

[1218] For example, it may notify you, "Your stress level is high. We recommend switching to autonomous driving mode."

[1219] Processing verbal feedback

[1220] User:

[1221] 1. Providing verbal feedback on proposals and notifications.

[1222] For example, reply, "Yes, please."

[1223] Device:

[1224] 1. Convert verbal feedback into text data using speech recognition technology.

[1225] 2. Send the textual feedback to the server.

[1226] Feedback analysis and system improvement

[1227] server:

[1228] 1. Analyze the verbal feedback received.

[1229] 2. Based on the feedback results, the generative model and emotion engine are retrained to improve the system.

[1230] Communication in urban areas

[1231] Device → Server:

[1232] 1. Collecting data for speech and gesture recognition during autonomous driving in urban areas.

[1233] 2. Send the collected data to the server.

[1234] server:

[1235] 1. The generative model generates an appropriate response based on the received data.

[1236] For example, if a pedestrian raises their hand, the system understands their intention and determines the appropriate action to take.

[1237] 2. Send the generated responses and instructions to the terminal.

[1238] Device:

[1239] 1. Notify people around you of received instructions via voice or display.

[1240] For example, instructions such as "Please go ahead" are announced through the speaker.

[1241] In this way, by using the system of the present invention, it is possible to increase social acceptance of autonomous driving technology while ensuring the safety and comfort of drivers.

[1242] The processing flow will be explained below.

[1243] Program processing steps

[1244] Step 1: Initialize the sensor

[1245] Device:

[1246] 1. Launch an integrated system for autonomous vehicles.

[1247] 2. Initialize the vehicle's cameras, lidar, GPS, fatigue monitoring sensors, and emotion engine.

[1248] 3. Start collecting data from the sensors.

[1249] Step 2: Collect sensor data

[1250] Device:

[1251] 1. Collect data in real time from various sensors on the vehicle.

[1252] Camera: Captures visual information about the surroundings.

[1253] Lidar: Obtains distance information to obstacles and other vehicles.

[1254] GPS: Get your current location.

[1255] Fatigue monitoring sensor: Detects the driver's eye movements and facial expressions.

[1256] Emotion Engine: Identifies the driver's emotional state from their tone of voice and facial expressions.

[1257] Step 3: Sending data

[1258] Device → Server:

[1259] 1. Collected sensor data is sent to the server at regular intervals.

[1260] Data includes speed, frequency of brake use, steering angle, eye movements and voice tones.

[1261] Step 4: Data analysis

[1262] server:

[1263] 1. Input the received sensor data into the generative model and emotion engine.

[1264] 2. The generative model analyzes driving skill and fatigue level in real time.

[1265] Driving behavior patterns are analyzed and compared with normal driving behavior.

[1266] Detect signs of fatigue (e.g., frequent blinking).

[1267] 3. The emotion engine analyzes the driver's emotional state from their tone of voice and facial expressions.

[1268] Stress is detected when the voice tone is high and the voice is trembling.

[1269] If the facial muscles are tense, assess anxiety or tension.

[1270] Step 5: Generate proposals

[1271] server:

[1272] 1. Evaluate the driver's condition based on the analysis results.

[1273] For example, if the level of fatigue is high, it is determined that "the driver is fatigued."

[1274] Emotional states such as stress and tension are also assessed.

[1275] 2. Generate a proposal to transition to autonomous driving mode.

[1276] For example, create a message that reads, "Your fatigue and stress levels are high, so we recommend switching to autonomous driving mode."

[1277] Step 6: Proposal Notification

[1278] Server → Device:

[1279] 1. Send generated suggestions and warnings to the device.

[1280] Device:

[1281] 1. Notify the driver of suggestions via display and audio feedback.

[1282] For example, a voice notification may say, "You are highly fatigued, so we recommend switching to autonomous driving mode."

[1283] Step 7: Gathering human feedback

[1284] User:

[1285] 1. Providing verbal feedback on proposals and notifications.

[1286] For example, reply, "Yes, please."

[1287] Device:

[1288] 1. Convert verbal feedback into text data using speech recognition technology.

[1289] 2. Send the textual feedback to the server.

[1290] Step 8: Analyze feedback and improve the system

[1291] server:

[1292] 1. Analyze the verbal feedback received.

[1293] 2. Based on the feedback results, the generative model and emotion engine are retrained to improve the system.

[1294] For example, if the driver instructs the system to "switch to autonomous driving mode," the generative model is updated accordingly.

[1295] Step 9: Communicating in the city

[1296] Device → Server:

[1297] 1. Collecting data for speech and gesture recognition during autonomous driving in urban areas.

[1298] 2. Send the collected data to the server.

[1299] server:

[1300] 1. The generative model generates an appropriate response based on the received data.

[1301] For example, if a pedestrian raises their hand, the system understands their intention and determines the appropriate action to take.

[1302] 2. Send the generated responses and instructions to the terminal.

[1303] Device:

[1304] 1. Notify people around you of received instructions via voice or display.

[1305] For example, instructions such as "Please go ahead" are announced through the speaker.

[1306] Through these detailed steps, the system of the present invention can ensure driver safety and comfort and increase social acceptance of autonomous driving technology.

[1307] Example 2

[1308] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1309] Conventional driving assistance systems have struggled to assess the driver's driving skill, fatigue level, and emotional state in real time and provide appropriate driving assistance based on that assessment. Furthermore, there was a lack of a way to incorporate driver feedback into system improvements, making continuous system improvement difficult. Furthermore, they lacked a function to support communication with surrounding people when the vehicle is stopped in urban areas.

[1310] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1311] In this invention, the server includes means for collecting sensor data from the vehicle to evaluate driving skill, means for analyzing the sensor data and using a generative model to evaluate the driver's fatigue level, means for using an emotion engine to analyze the driver's emotional state, means for proposing transition to autonomous driving mode based on the analysis results, means for notifying the driver of the suggestion, means for converting verbal feedback from the driver into text data using speech recognition technology, means for collecting driver feedback and using it to improve the system, and means for supporting communication with surrounding people when the vehicle is stopped. This enables the driver's driving skill, fatigue level, and emotional state to be evaluated in real time, and appropriate driving assistance based on the evaluation. Furthermore, collecting feedback and using it to improve the system promotes continuous improvement. Furthermore, smooth communication with surrounding people when the vehicle is stopped in urban areas is possible.

[1312] "Sensor data" refers to data obtained from multiple sensors installed in a vehicle and used to evaluate the driver's driving skill, fatigue level, and emotional state.

[1313] A "generative model" is a machine learning algorithm that analyzes collected sensor data to evaluate a driver's driving skill and fatigue level.

[1314] The "emotion engine" is an algorithm that analyzes the driver's emotional state from their tone of voice and facial expressions, and identifies emotions such as stress and anxiety.

[1315] "Autonomous driving mode" is a mode in which the system controls the vehicle on behalf of the driver, and switches automatically depending on the driver's level of fatigue and emotional state.

[1316] "Suggestions" refer to instructions or advice that the system notifies the driver based on the analysis results, including, for example, a notification recommending switching to autonomous driving mode.

[1317] "Feedback" is the verbal or other responses or opinions from the driver that are collected and used to improve the system.

[1318] "Voice recognition technology" is a technology that converts the driver's voice into a digital signal and analyzes it as text data.

[1319] "System improvement" is the process of retraining the generative model and emotion engine based on collected feedback to improve the overall performance of the safe driving support system.

[1320] "Communication Support" is an assistance function that facilitates smooth interactions with people around the vehicle when stopped, and provides appropriate responses and instructions via voice and display.

[1321] This invention relates to a safe driving support system that uses in-vehicle sensors to collect driving data and analyzes it using a generative AI model and emotion engine to evaluate the driver's driving skill, fatigue level, and emotional state. Based on the analysis results, the system proposes an appropriate transition to an autonomous driving mode and notifies the driver. Furthermore, verbal feedback from the driver is collected and used to retrain the system, ensuring continuous improvement. Furthermore, when driving in urban areas, the system supports communication with surrounding people.

[1322] Hardware and software used

[1323] Hardware:

[1324] Camera: Captures visual information about the vehicle's surroundings.

[1325] Lidar: Obtains distance information to obstacles.

[1326] GPS: Get your current location.

[1327] Fatigue monitoring sensor: Detects eye movements and facial expressions.

[1328] Emotion Engine: Identify emotional states from voice tone and facial expressions.

[1329] software:

[1330] Generative AI model: Uses collected data to analyze driving skills and fatigue levels in real time.

[1331] Speech recognition technology: Analyzes verbal feedback and converts it into text data.

[1332] Explanation of program processing

[1333] Device:

[1334] Various sensors installed inside the vehicle collect data in real time and send it to a server. Specifically, the camera captures visual information about the surroundings, the lidar measures the distance to obstacles, and the GPS tracks location information. In addition, a fatigue monitoring sensor monitors the driver's eye movements and facial expressions, and an emotion engine analyzes the driver's emotional state from voice tone and facial expressions. For example, if the driver blinks frequently, it is determined that the driver is highly fatigued.

[1335] server:

[1336] The server receives sensor data sent from the device and inputs it into the generative AI model and emotion engine for analysis. The generative AI model evaluates driving skill and fatigue level, and the emotion engine analyzes the driver's emotional state. Based on the analysis results, it determines whether switching to autonomous driving mode is necessary and generates a proposal to do so. The proposal or warning is sent to the device and notified to the driver. If the driver answers "yes," the system switches to autonomous driving mode. In addition, verbal feedback from the driver is converted into text data using voice recognition technology and used to improve the system. For example, the system can switch to autonomous driving by replying "yes" to a notification that says, "We recommend switching to autonomous driving mode."

[1337] User:

[1338] The driver responds to the suggestions and warnings provided and provides verbal feedback. For example, if the driver receives a notification saying, "Your fatigue level is increasing. We recommend switching to autonomous driving mode," and responds, "Yes, please," the system will switch to autonomous driving mode. Additionally, when the vehicle is stopped in urban areas, the system will provide appropriate instructions to nearby pedestrians via voice, such as, "Please go ahead and let the vehicle pass."

[1339] Examples of concrete examples and prompts

[1340] Examples:

[1341] If the driver feels fatigued and blinks more frequently while driving on a highway, the system will issue a voice notification saying, "Your fatigue level is high, so we recommend switching to autonomous driving mode." If the driver replies "yes," the system will switch to autonomous driving mode.

[1342] Example prompt sentence:

[1343] "Analyze data from vehicle sensors as follows and suggest transitioning to autonomous driving mode: When the driver is highly fatigued and stressed."

[1344] As described above, the safe driving support system of the present invention can provide advanced support for driver safety and comfort by combining a wide range of sensors with an advanced AI model and emotion engine.

[1345] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1346] Step 1:

[1347] Sensor data collection

[1348] Device: Collects data in real time from various sensors installed in the vehicle. Specifically, the camera captures visual information of the surroundings, the lidar measures the distance to obstacles, and the GPS acquires the current location information. The fatigue monitoring sensor monitors the driver's eye movements and facial expressions, and the emotion engine analyzes the driver's emotional state from voice tone and facial expressions.

[1349] Inputs: Data from camera, lidar, GPS, fatigue monitoring sensors, emotion engine.

[1350] Output: Camera visual information, rider distance information, GPS location information, fatigue level information, emotional state information.

[1351] Step 2:

[1352] Sending collected data

[1353] Device → Server: Collected sensor data is periodically sent to the server, including speed information, braking frequency, steering angle, eye movements, and audio tones.

[1354] Input: Data collected from various sensors.

[1355] Output: Sensor data sent to the server.

[1356] Step 3:

[1357] Data analysis

[1358] Server: The received data is input into the generative AI model and emotion engine to analyze driving skill and fatigue level. The generative AI model evaluates driving skill and determines fatigue level based on factors such as frequent blinking. The emotion engine analyzes the driver's emotional state based on voice tone and facial expressions. For example, if the voice is trembling, it is interpreted as being stressed.

[1359] Input: Sensor data sent to the server.

[1360] Output: Driving skill evaluation results, fatigue evaluation results, emotional state evaluation results.

[1361] Step 4:

[1362] Proposal Generation

[1363] Server: Based on the analysis results, evaluates the driver's condition and determines whether switching to autonomous driving mode is necessary. For example, if the driver's fatigue level or stress level is high, a recommendation is generated to "recommend switching to autonomous driving mode."

[1364] Input: Driving skill evaluation results, fatigue evaluation results, emotional state evaluation results.

[1365] Output: Proposal to transition to autonomous driving mode.

[1366] Step 5:

[1367] Proposal Notification

[1368] Server → Device: Sends generated suggestions and warnings to the device.

[1369] Device: Notify the driver of the suggestion through a display or audio feedback, for example, "Your stress level is high. We recommend switching to autonomous driving mode."

[1370] Input: Proposal to transition to autonomous driving mode.

[1371] Output: Display of suggestion notification and audio feedback.

[1372] Step 6:

[1373] Processing Feedback

[1374] User: The driver provides verbal feedback to the suggestion or notification, for example, by replying "Yes, please."

[1375] Terminal: Verbal feedback is converted into text data using speech recognition technology and sent to the server.

[1376] Input: User's verbal feedback.

[1377] Output: Feedback converted to text data.

[1378] Step 7:

[1379] Feedback analysis and system improvement

[1380] Server: Analyzes the received verbal feedback and retrains the generative model and emotion engine to improve the system.

[1381] Input: Feedback converted to text data.

[1382] Output: Improved generative model and emotion engine.

[1383] Step 8:

[1384] Communication support in urban areas

[1385] Terminal → Server: Collects data to recognize speech and gestures during autonomous driving in urban areas and sends it to the server.

[1386] Server: The generative model generates an appropriate response based on the received data. For example, if a pedestrian raises their hand, the server understands their intention and determines the appropriate action. The server then sends the generated response and instructions to the device.

[1387] Terminal: The terminal notifies people around it of the received instructions through voice or display. For example, it may announce the instruction "Please go ahead" through a speaker.

[1388] Input: Speech and gesture data.

[1389] Output: Audio and display notifications for people around you.

[1390] (Application example 2)

[1391] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1392] Conventional driving support systems have difficulty accurately analyzing the driver's fatigue level and emotional state in real time, resulting in an inability to suggest transitioning to autonomous driving mode at the appropriate time. Effectively collecting feedback from the driver and utilizing it for continuous system improvement has also been a challenge. Furthermore, smooth communication with surrounding people in urban areas is also required.

[1393] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for collecting the driver's facial expressions, voice tone, and driving behavior data in real time using a smartphone; means for transmitting the collected data to the server and analyzing it using an AI generation model and an emotion engine; means for notifying the driver via audio or visual means on the smartphone based on the analysis results; means for converting the driver's verbal feedback into text data using voice recognition technology and transmitting it to the server; and means for supporting communication with surrounding people while driving. This makes it possible to accurately grasp the driver's fatigue level and emotional state and suggest switching to autonomous driving mode at the appropriate time. Furthermore, it is possible to effectively collect driver feedback and achieve continuous system improvement. Furthermore, it also enables smooth communication in urban areas.

[1394] A "driving skill assessment" is an assessment of how safely and efficiently a driver can drive.

[1395] "Sensor data" refers to information collected from various sensors, including camera footage, audio, acceleration, and location information.

[1396] A "generative model" is an algorithm that uses AI technology to analyze data for a specific purpose and predict an outcome.

[1397] "Fatigue assessment" is the process of measuring how tired a driver is.

[1398] "Autonomous driving mode" refers to a mode in which the vehicle performs driving operations autonomously.

[1399] "Notification" refers to the act of the system informing the driver of information, and includes audio and visual notifications.

[1400] "Feedback" refers to the opinions and reactions provided by the driver to the system.

[1401] "System improvement" means improving the performance and accuracy of the system based on collected feedback and data.

[1402] A "smartphone" is a device that combines advanced computing power and connectivity with a mobile phone.

[1403] "Facial expression" refers to the facial state that is revealed by the movement of the driver's facial muscles.

[1404] "Voice tone" refers to the pitch, strength, and emotional characteristics of a driver's voice.

[1405] "Driving behavior data" refers to data related to the driver's driving operations, including acceleration, deceleration, braking, etc.

[1406] The "emotion engine" is an algorithm that analyzes voice tone and facial expressions to detect the driver's emotional state.

[1407] "Speech recognition technology" refers to the technology that converts voice into text data.

[1408] "Communication in urban areas" refers to smoothly exchanging information with surrounding pedestrians and other vehicles while driving.

[1409] The safe driving support system of the present invention evaluates the driver's driving skill, fatigue level, and emotional state, and suggests transitioning to autonomous driving mode at an appropriate time. This system uses a smartphone, a server, and various sensors.

[1410] Hardware Configuration

[1411] 1. Smartphone: A smartphone equipped with a camera, microphone, accelerometer, and GPS is used to collect the driver's facial expressions, voice tone, and driving behavior data in real time.

[1412] 2. Server: A server equipped with a high-performance GPU is used. The server receives data sent from the smartphone and analyzes it using a generative AI model and emotion engine.

[1413] 3. Communication network: Data is sent and received using communication networks such as Wi-Fi and 4G / 5G.

[1414] Software Configuration

[1415] 1. Smartphone app: Install the app that collects driving data, sends the data to a server, notifies the driver of the analysis results, and collects driver feedback. This app works on Android or iOS.

[1416] 2. Speech Recognition System: Use speech recognition technology, such as Google Cloud Speech-to-Text, to convert the driver's verbal feedback into text data.

[1417] 3. Generative AI model and emotion engine: This model will be equipped with algorithms to analyze driving data and evaluate driving skill, fatigue level, and emotional state. These algorithms will be implemented using machine learning libraries such as PyTorch and TensorFlow.

[1418] System Operation

[1419] The server receives driving data sent from the smartphone. The received data is analyzed using a generative AI model and an emotion engine. This analysis evaluates the driver's driving skill, fatigue level, and emotional state. For example, if the server determines that the driver is very tired, it generates a notification suggesting, "Do you want to switch to autonomous driving mode?" This notification is communicated to the driver via audio or visual means via the smartphone.

[1420] Driver feedback is also important. When the driver responds verbally to suggestions and notifications, the smartphone's voice recognition system converts the speech into text data and sends it to the server. The server analyzes the feedback and uses the results to retrain the generative AI model and emotion engine, ensuring continuous improvement of the system.

[1421] To facilitate smooth communication in urban areas while driving, the system also sends voice messages to enable drivers to safely and smoothly exchange information with those around them, and this function also supports communication with pedestrians and other vehicles.

[1422] Examples and prompts

[1423] As a concrete example, if a driver continues driving for a long period of time, their facial expressions are recorded by a camera mounted on their smartphone. The generative AI model determines that the driver is highly fatigued from their facial expression and issues a message saying, "You seem tired. Would you like to switch to autonomous driving mode?" If the driver replies "yes," the voice recognition system converts the speech into text data and sends it to a server. Based on this feedback, the system switches to autonomous driving mode.

[1424] Example prompt sentence:

[1425] "The system analyzes the driver's emotional state from their facial expressions and vocal tone to assess their driving skill and fatigue level. If it determines that their fatigue level is high, it notifies the driver, 'Do you want to switch to autonomous driving mode?' and receives verbal feedback. It communicates with the server and uses the driver's feedback to improve the system."

[1426] This will improve driver safety and comfort, and also increase social acceptance of autonomous driving technology.

[1427] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1428] Step 1:

[1429] The device (smartphone) uses a camera, microphone, acceleration sensor, and GPS to collect the driver's facial expressions, voice tone, and driving behavior data in real time while driving. Inputs include the driver's facial image, voice, acceleration data, and location information. This data is collected from various sensors and stored on the device.

[1430] Step 2:

[1431] The terminal periodically transmits the collected data to the server. The driving data collected by the terminal is used as input, and the data is transmitted to the server as output. This transmission is performed using a communication network (Wi-Fi, 4G / 5G).

[1432] Step 3:

[1433] The server receives the data sent from the device and inputs it into the generative AI model and emotion engine. The input includes driving data (facial images, audio, driving behavior data, and location information). The server passes the received data to the AI ​​model and begins analysis.

[1434] Step 4:

[1435] The generative AI model analyzes driving data and evaluates driving skill and fatigue level in real time. Facial images and driving behavior data are used as input, and the output is an evaluation result of driving skill and fatigue level. For example, if a driver blinks frequently, the model will determine that the driver is highly fatigued.

[1436] Step 5:

[1437] The emotion engine analyzes the driver's emotional state from their voice tone and facial expression. It uses voice data and facial images as input, and generates an evaluation result of the driver's emotional state as output. For example, if the voice tone is unstable, the server determines that the driver is stressed.

[1438] Step 6:

[1439] The server evaluates the driver's state based on the analysis results of the generative AI model and emotion engine. The input includes the evaluation results of driving skill, fatigue level, and emotional state. The output generates a proposal to switch to autonomous driving mode if necessary. For example, a proposal such as "Do you want to switch to autonomous driving mode?" is generated.

[1440] Step 7:

[1441] The server sends the generated proposal to the terminal. The proposal content is used as input, and the proposal is sent to the terminal as output. The terminal notifies the driver of this proposal by voice or visual means. Specifically, the smartphone speaker will say, "Do you want to switch to autonomous driving mode?"

[1442] Step 8:

[1443] The user (driver) provides verbal feedback in response to the notification from the terminal. The driver's voice is used as input, and the voice data is collected by the terminal as output. For example, the driver replies "Yes."

[1444] Step 9:

[1445] The terminal converts the collected voice data into text data using a voice recognition system. The driver's voice data is used as input, and text data is generated as output. This converts the voice "yes" into the text "yes."

[1446] Step 10:

[1447] The device sends textual feedback to the server. Text data is used as input, and feedback is sent to the server as output. The server analyzes the received feedback and retrains the AI ​​model and emotion engine based on the results to improve the system. Specifically, data on how the driver responded to each notification is accumulated.

[1448] Step 11:

[1449] During autonomous driving in urban areas, the server and terminal collect data to communicate with surrounding pedestrians and other vehicles and send it to the server. Surrounding audio and image data are used as input, and sent to the server as output. The server generates an appropriate response based on the received data and sends it to the terminal. The terminal notifies surrounding people of the generated response by voice or display. For example, an instruction such as "Please go ahead" may be issued through the speaker.

[1450] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1451] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1452] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1453] [Fourth embodiment]

[1454] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1455] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1456] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1457] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1458] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1459] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1460] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1461] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1462] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1463] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1464] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1465] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1466] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1467] The safe driving support system of this invention collects driving data in real time from sensors installed in the vehicle and analyzes it using an AI model to evaluate the driver's driving skill and fatigue level. Based on the results of this evaluation, it proposes an appropriate transition to autonomous driving mode and notifies the driver. It also collects verbal feedback from the driver and uses it to retrain the AI ​​model, thereby ensuring continuous improvement of the system. Furthermore, it supports communication with surrounding people when driving in urban areas.

[1468] System Overview

[1469] Device:

[1470] The vehicle is equipped with multiple sensors, including cameras, lidar, GPS, and fatigue monitoring sensors. The device collects data from these sensors and transmits it to a server. For example, a sensor that monitors the driver's eye movements provides important information for assessing the driver's fatigue level.

[1471] server:

[1472] The server receives data sent from the device and inputs it into the generative model. The generative model uses AI algorithms to analyze driving skill and fatigue level in real time. Based on the analysis results, it determines whether switching to autonomous driving mode is necessary and generates a recommendation. The server also receives feedback from the driver to help improve the system.

[1473] User:

[1474] The user is the driver and receives suggestions and warnings from the device. They also provide verbal feedback to contribute to system improvement. For example, after driving for a long time on a highway, the user may receive a notification from the device saying, "We recommend switching to autonomous driving mode." By responding "yes," the user can switch to autonomous driving mode.

[1475] Program processing

[1476] The processing contents of the program will be specifically explained below.

[1477] Sensor data collection:

[1478] The device collects real-time data from various sensors in the vehicle. For example, camera data provides visual information, lidar data provides distance information to surrounding objects, and fatigue monitoring sensors monitor the driver's facial expressions and eye movements to assess fatigue level.

[1479] Data transmission and analysis:

[1480] The device sends the collected data to a server. The server inputs the received data into a generative model for analysis. This analysis evaluates the driver's driving skill and fatigue level. For example, if the driver blinks frequently, the generative model will determine that the driver is highly fatigued.

[1481] Proposal generation and notification:

[1482] The server generates suggestions and warnings and sends them to the device, which then notifies the driver via audio or visual notification. For example, a notification might say, "Your fatigue level is increasing. We recommend switching to autonomous driving mode."

[1483] Processing verbal feedback:

[1484] The device uses voice recognition technology to convert verbal feedback from the driver into text data and sends it to the server. The server analyzes this feedback and uses it to retrain the generative model. For example, if the driver verbally commands, "Please switch to autonomous driving mode," the system converts this into text data and sends it to the server.

[1485] Communication in urban areas:

[1486] During autonomous driving in cities, the system recognizes speech and gestures and provides appropriate responses through generative models, such as an audio message to pedestrians about to cross the street saying, "Please go ahead."

[1487] In this way, by using the system of the present invention, it is possible to increase social acceptance of autonomous driving technology while ensuring driver safety.

[1488] The processing flow will be explained below.

[1489] Program processing steps

[1490] Step 1: Initialize the sensor

[1491] Device:

[1492] 1. Launch an integrated system for autonomous vehicles.

[1493] 2. Initialize the vehicle's camera, lidar, GPS, and fatigue monitoring sensors.

[1494] 3. Start collecting data from the sensors.

[1495] Step 2: Collect sensor data

[1496] Device:

[1497] 1. Collect data in real time from each sensor on the vehicle.

[1498] Camera: Captures visual information about the surroundings.

[1499] Lidar: Obtains distance information to obstacles.

[1500] GPS: Get your current location.

[1501] Fatigue monitoring sensor: Monitors the driver's eye movements and facial expressions.

[1502] Step 3: Sending data

[1503] Device → Server:

[1504] 1. Collected sensor data is periodically sent to a server via batch processing or real-time streaming.

[1505] 2. Data includes speed, frequency of braking, steering angle, eye movements, etc.

[1506] Step 4: Data analysis

[1507] server:

[1508] 1. Input the received sensor data into the generative model.

[1509] 2. A generative model analyzes the data and assesses driving skill and fatigue in real time.

[1510] The analysis results include detecting driving behavior patterns and signs of fatigue.

[1511] Step 5: Generate proposals

[1512] server:

[1513] 1. Evaluate the driver's condition based on the analysis results.

[1514] For example, if the fatigue level is high, the system will determine that the fatigue level is high.

[1515] 2. Generate a proposal to transition to autonomous driving mode.

[1516] For example, it generates a message saying, "We recommend switching to autonomous driving mode."

[1517] Step 6: Proposal Notification

[1518] Server → Device:

[1519] 1. Send generated suggestions and warnings to the device.

[1520] Device:

[1521] 1. Notify the driver of suggestions via display and audio feedback.

[1522] Step 7: Gathering human feedback

[1523] User:

[1524] 1. Providing verbal feedback on proposals and notifications.

[1525] For example, reply, "Yes, please."

[1526] Device:

[1527] 1. Convert verbal feedback into text data using speech recognition technology.

[1528] 2. Send the textual feedback to the server.

[1529] Step 8: Analyze feedback and improve the system

[1530] server:

[1531] 1. Analyze the verbal feedback received.

[1532] 2. Retrain the generative model based on the feedback results and improve the system.

[1533] Step 9: Communicating in the city

[1534] Device → Server:

[1535] 1. Collecting data to recognize speech and gestures while driving autonomously around town.

[1536] 2. Send the collected data to the server.

[1537] server:

[1538] 1. The generative model generates an appropriate response based on the received data.

[1539] For example, if a pedestrian raises their hand, the system understands their intention and determines the appropriate action to take.

[1540] 2. Send the generated responses and instructions to the terminal.

[1541] Device:

[1542] 1. Notify people around you of received instructions via voice or display.

[1543] For example, instructions such as "Please go ahead" are announced through the speaker.

[1544] Through these detailed steps, the system of the present invention can ensure driver safety and increase social acceptance of autonomous driving technology.

[1545] Example 1

[1546] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1547] In modern society, long-distance driving and driving in urban areas place a heavy burden on drivers, and the risk of accidents increases as fatigue accumulates. To solve these problems and achieve safe and efficient driving, a system is needed that monitors the driver's condition in real time and suggests switching to autonomous driving mode as necessary. It is also important to be able to communicate smoothly with people around you in the city.

[1548] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1549] In this invention, the server includes a means for collecting sensor data from the vehicle to evaluate driving skills, a means for analyzing the sensor data and using a generative model to evaluate the driver's fatigue level, a means for proposing switching to autonomous driving mode based on the analysis results, a means for collecting user feedback to help improve the system, and a means for recognizing speech and gestures and providing a response based on the surrounding situation. This makes it possible to accurately grasp the driver's fatigue level and propose switching to autonomous driving mode when necessary. In addition, appropriate responses can be provided even in urban areas, facilitating communication with surrounding people.

[1550] "Sensor data" refers to information collected by various sensors to understand the situation inside and around the vehicle.

[1551] "Driving ability assessment" refers to the measurement and analysis of a driver's driving ability and skills.

[1552] "Fatigue assessment" is the process of monitoring and measuring the driver's fatigue state.

[1553] A "generative model" is an AI algorithm that learns from large amounts of data and analyzes and evaluates driving skills and fatigue levels.

[1554] "Autonomous driving mode" refers to a mode in which the vehicle automatically performs driving operations, minimizing driver involvement.

[1555] "Suggestion" refers to notifications to the driver to change their driving mode or to issue warnings based on the analysis results.

[1556] "Notification" is the act of conveying information to the driver through audio or visual means.

[1557] "Feedback" refers to the evaluations and opinions that the driver provides to the system, which are used to improve the system.

[1558] "Speech recognition" refers to the technology that uses voice recognition technology to understand and process the driver's verbal instructions and feedback.

[1559] "Gesture recognition" is a technology that detects gestures and hand movements made by drivers and pedestrians, and enables the system to provide an appropriate response based on those gestures.

[1560] The safe driving support system of this invention is a system that evaluates the driver's driving skill and fatigue level by collecting driving data in real time from various sensors installed in the vehicle and analyzing it based on an AI-generated model. The system consists of the following components.

[1561] Hardware Configuration

[1562] The device collects data from multiple sensors installed in the vehicle and transmits it to a server. These include a camera, lidar, GPS, and a fatigue monitoring sensor. For example, the camera monitors the driver's face and road conditions, while the lidar measures the distance to surrounding objects. The fatigue monitoring sensor monitors the driver's facial expressions and eye movements and evaluates their fatigue level based on the data.

[1563] Software Configuration

[1564] The server receives the data sent from the device and analyzes it in real time using an AI generative model. A generative model is an AI algorithm that has been trained on a large amount of driving data in advance. Specifically, the server analyzes camera data using an image recognition algorithm (for example, face detection using deep learning) to evaluate the driver's fatigue level. Lidar data and GPS data are also integrated to evaluate the driving environment and road conditions.

[1565] System operation example

[1566] For example, if camera data shows that a driver is blinking frequently after driving for a long period of time, the server's generative model will assess the driver's fatigue level as high. Based on this result, the server will generate a recommendation such as "Your fatigue level is increasing. We recommend switching to autonomous driving mode," and send it to the device. The device will then notify the driver of this recommendation via voice and display.

[1567] The driver provides verbal feedback, such as "Please switch to autonomous driving mode." This verbal feedback is captured by a microphone installed on the device and converted into text data using voice recognition technology. The server receives this text data and uses it to retrain the system. In this way, the system is continuously improved.

[1568] During autonomous driving in urban areas, the device understands the surrounding situation based on data from lidar and cameras. For example, if a pedestrian is detected approaching a crosswalk, the generative model will suggest an appropriate voice message such as "Please go ahead," and the device will notify the driver by voice.

[1569] Prompt Sentence Examples

[1570] "Please explain the process of the program that generates a notification suggesting switching to autonomous driving mode based on fatigue assessment after a long period of driving."

[1571] "Please tell me about the function of the system that allows vehicles traveling in urban areas to provide appropriate audio notifications to pedestrians."

[1572] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1573] Step 1: Collect sensor data

[1574] The device collects data in real time from multiple sensors installed on the vehicle. The inputs are raw data from cameras, lidar, GPS, and fatigue monitoring sensors. These data include cameras capturing images of the driver's face and road conditions, lidar measuring the distance to surrounding objects, GPS acquiring location information, and fatigue monitoring sensors monitoring the driver's facial expressions and eye movements. As an output, these data are shaped into a single integrated data stream, ready to be sent to a server.

[1575] Step 2: Send and analyze data

[1576] The device sends the collected sensor data to the server. The input is the integrated data stream collected and shaped in step 1. The server receives this data stream and inputs it into the generative AI model. The generative AI model uses image recognition algorithms and pattern recognition algorithms to analyze the driver's fatigue level from the camera data and evaluate the driving environment and road conditions from other sensor data. The output is an evaluation of the driver's fatigue level and driving skills.

[1577] Step 3: Proposal generation and notification

[1578] The server generates suggestions and warnings based on the analysis results of step 2. The input is data on the driver's fatigue level assessment and driving skill assessment. The server generates suggestions and warning messages such as "Your fatigue level is increasing. We recommend switching to autonomous driving mode." As output, these messages are sent to the terminal. The terminal notifies the driver of this suggestion via voice or display. Specifically, a warning message is displayed on the display and voice guidance is played from the speaker.

[1579] Step 4: Processing verbal feedback

[1580] The user provides verbal feedback in response to notifications and suggestions from the device. The input is the driver's verbal instruction (e.g., "Please switch to autonomous driving mode"). The device captures this voice feedback and converts it into text data using voice recognition technology. As output, this text data is sent to the server and used for retraining the system. Specifically, the microphone captures the voice, and the internal voice recognition module converts it into text.

[1581] Step 5: Communicating in urban areas

[1582] During autonomous driving in urban areas, the device understands the surrounding situation based on data from the lidar and camera. The input is real-time data from the lidar and camera. For example, if a pedestrian is detected when approaching a crosswalk, the generative AI model will suggest an appropriate voice message such as "Please go ahead." The device will notify the driver of this suggestion by voice. The output is appropriate communication with the driver and pedestrian. In concrete terms, the camera and lidar scan the surrounding situation, and the speaker plays the voice message.

[1583] (Application example 1)

[1584] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1585] Autonomous vehicles are required to accurately assess the driver's fatigue level and driving skills in real time and make appropriate suggestions based on the assessment. However, conventional safe driving support systems are unable to monitor the driver's eye movements and facial expressions in detail, making it difficult to accurately grasp the driver's condition. Furthermore, the lack of effective means to collect and analyze feedback from the driver makes continuous system improvement difficult. Furthermore, there is an issue of being unable to effectively support communication with humans in urban areas during autonomous driving.

[1586] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1587] In this invention, the server includes means for collecting sensor data from the vehicle to evaluate driving skills, means for analyzing the sensor data and using a generative model to evaluate the driver's fatigue level, means for monitoring the driver's eye movements and facial expressions and evaluating them in real time, means for proposing switching to autonomous driving mode based on the analysis results, means for notifying the driver of the suggestion, means for collecting verbal feedback from the driver and converting it into text data using speech recognition technology, means for analyzing the converted text data and using it to improve the system, and means for recognizing speech and gestures during autonomous driving in urban areas and providing appropriate responses. This enables the system to accurately grasp the driver's condition, make appropriate suggestions, and effectively collect and analyze feedback from the driver, enabling continuous system improvement and communication support in urban areas.

[1588] "Driving skill assessment" is the process of measuring and assessing a driver's level of driving skill and ability.

[1589] "Sensor data" refers to information obtained from various sensors installed on the vehicle, including camera images, distance data from LIDAR, and GPS information.

[1590] A "generative model" is an artificial intelligence (AI) algorithm used to analyze collected data and assess the driver's state.

[1591] "Fatigue assessment" is the measurement and evaluation of a driver's level of fatigue, usually by analyzing eye movements and facial expressions.

[1592] "Autonomous driving mode" refers to a mode in which a vehicle drives autonomously, responding to road conditions and the surrounding environment without driver intervention.

[1593] "Notification Means" means the method used to communicate a suggestion or warning to the driver, including a visual display or audio notification.

[1594] "Feedback collection" is the process of collecting opinions and reactions from drivers, which allows for improvements to the system.

[1595] "Voice recognition technology" is a technology that converts voice data into text data, which allows the driver's verbal feedback to be analyzed.

[1596] "Communication in urban areas" refers to an automated vehicle's ability to communicate appropriately with other vehicles and pedestrians in urban areas, including through speech and gesture interaction.

[1597] overview

[1598] The safe driving support system of this invention aims to evaluate driving skills, monitor fatigue levels, collect feedback, appropriately transition to autonomous driving mode, and provide communication support in urban areas. The system collects data from multiple sensors installed in the vehicle and analyzes it with a generative AI model. Based on the analysis results, it makes appropriate suggestions to the driver and collects verbal feedback from the driver for use in retraining the system.

[1599] System Configuration

[1600] 1. Hardware Configuration

[1601] Smart glasses: Monitor the driver's eye movements and facial expressions and provide visual and audio notifications.

[1602] In-vehicle sensors: camera, lidar, GPS, fatigue monitoring sensors.

[1603] Server: A server for running generative AI models for data analysis.

[1604] 2. Software Configuration

[1605] AI Model: A generative AI model built using TensorFlow or PyTorch.

[1606] Speech recognition technology: We use Google Speech-to-Text API or equivalent technology.

[1607] Data communication: Data is transmitted between the smart glasses and the server via Wi-Fi or Bluetooth.

[1608] Process Overview

[1609] 1. Data Collection

[1610] The smart glasses and in-vehicle sensors collect real-time data, including the driver's eye movements, facial expressions, camera footage of the surrounding environment, distance information from LiDAR, and location information from GPS.

[1611] 2. Data Transmission

[1612] The collected data is sent to a server via Wi-Fi or Bluetooth.

[1613] 3. Analysis using AI models

[1614] The server inputs the received data into a generative AI model to evaluate the driver's fatigue level and driving skill. For example, if frequent blinking is observed, it is determined that the driver is highly fatigued.

[1615] 4. Proposals and Notifications

[1616] Based on the analysis, it determines whether a transition to autonomous driving mode is necessary and generates recommendations, which are communicated to the driver via visual and audio notifications via the smart glasses.

[1617] 5. Feedback Collection

[1618] Verbal feedback from the driver is collected by the microphone in the smart glasses and converted into text data using voice recognition technology, which is then sent to a server and used to retrain the system.

[1619] 6. Communication in urban areas

[1620] During autonomous driving, the system supports communication with pedestrians and other vehicles in urban areas. For example, it has a function to tell pedestrians "please go ahead."

[1621] Specific examples

[1622] When a driver is driving for a long period of time, the smart glasses monitor the driver's eye movements, and if frequent blinking is detected, a notification will be displayed saying, "Your level of fatigue is increasing. We recommend switching to autonomous driving mode."

[1623] When the driver issues a voice command to "switch to autonomous driving," the system recognizes the voice and switches to autonomous driving mode.

[1624] Prompt Sentence Examples

[1625] Please enter your driving data and fatigue evaluation data in the following format.

[1626] {

[1627] "Driving data": {

[1628] "Speed": "80",

[1629] "camera": "visual information data",

[1630] "lidar": "distance information data",

[1631] "GPS": "Location data"

[1632] },

[1633] "Fatigue Level": "High"

[1634] }

[1635] Generate appropriate suggestions.

[1636] {

[1637] "Suggestion": "We recommend switching to Autopilot mode",

[1638] "Notification Method": "Visual and Audio"

[1639] }

[1640] In this way, the invention can improve driver safety and comfort and provide reliable support during long drives and automated driving in urban areas.

[1641] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1642] Step 1: Data collection

[1643] The terminal collects data in real time from various sensors installed inside the vehicle. Specifically, the camera captures visual information about the vehicle's surroundings, the lidar captures distance information from objects, the GPS captures location information, and the smart glasses capture the driver's eye movements. This collected data is the initial input.

[1644] Step 2: Send data

[1645] The data collected by the device is sent to the server via Wi-Fi or Bluetooth. The input here is the sensor data sent from the device, and the output is the data received by the server.

[1646] Step 3: Data analysis

[1647] The server inputs the received sensor data into the generative AI model to evaluate the driver's driving skill and fatigue level. The input is sensor data, which is calculated and analyzed by the generative AI model. The output is an evaluation of the driver's driving skill and fatigue level.

[1648] Step 4: Proposal Generation

[1649] The server determines whether switching to autonomous driving mode is necessary based on the analysis results and generates a proposal. The input is the result of the data analysis, and the output is the proposal. This proposal may include content such as "We recommend switching to autonomous driving mode."

[1650] Step 5: Notification

[1651] The device communicates the suggestions generated by the server to the driver through the smart glasses with visual and audio notifications. The input is the suggestion from the server, and the output is the notification to the driver. Specifically, the display will say "We recommend switching to autonomous driving mode," and the same message will be conveyed via audio.

[1652] Step 6: Gather feedback

[1653] The user (driver) provides verbal feedback through the microphone in the smart glasses. For example, they give a voice command such as "switch to automatic driving." This becomes the input data. The device converts this voice feedback into text using speech recognition technology and sends it to the server. The output is a string of feedback data.

[1654] Step 7: Feedback analysis

[1655] The server analyzes the character string feedback data sent from the device and uses it to retrain the generative AI model. The input is the feedback data, and the output is the updated results of the model. Specifically, the parameters of the generative AI model are adjusted based on the driver's feedback.

[1656] Step 8: Communicating in urban areas

[1657] The device uses data from sensors to support communication with pedestrians and other vehicles while the vehicle is driving autonomously in urban areas. The input is data from lidar and cameras, which the server analyzes and generates appropriate responses such as "please go ahead." The output is a notification to the driver or pedestrian. For example, if a pedestrian is standing at a crosswalk, the device will notify them by voice, "please go ahead."

[1658] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1659] The safe driving support system of this invention uses in-vehicle sensors to collect driving data in real time and analyzes it using an AI generation model and emotion engine to evaluate the driver's driving skill, fatigue level, and emotional state. Based on the results of this evaluation, the system proposes an appropriate transition to autonomous driving mode and notifies the driver. Furthermore, verbal feedback from the driver is collected and used to retrain the AI ​​generation model, thereby ensuring continuous improvement of the system. Furthermore, when driving in urban areas, the system supports communication with surrounding people.

[1660] System Overview

[1661] Device:

[1662] The vehicle is equipped with multiple sensors, including cameras, lidar, GPS, fatigue monitoring sensors, and an emotion engine that detects the driver's emotional state. The device collects data from these sensors and transmits it to a server. For example, a sensor monitors the driver's eye movements to assess the driver's fatigue level, and facial expressions and voice tone to assess emotions.

[1663] server:

[1664] The server receives data sent from the device and inputs it into the generative AI model and emotion engine. The generative model analyzes the data and evaluates driving skill and fatigue level in real time. The emotion engine analyzes the driver's emotional state and detects stress and anxiety. Based on the results of this analysis, it determines whether switching to autonomous driving mode is necessary and generates a recommendation to do so. The server also receives feedback from the driver to help improve the system.

[1665] User:

[1666] The user is the driver and receives suggestions and warnings from the device. They can also provide verbal feedback to contribute to system improvements. For example, if a user receives a notification from the device while driving on a highway saying, "We recommend switching to autonomous driving mode," they can respond "yes" to switch to autonomous driving mode. If the user feels stressed while driving, the emotion engine can also return advice such as "relax."

[1667] Program processing

[1668] Sensor data collection

[1669] Device:

[1670] 1. Collect data in real time from various sensors on the vehicle.

[1671] Camera: Captures visual information about the surroundings.

[1672] Lidar: Obtains distance information to obstacles.

[1673] GPS: Get your current location.

[1674] Fatigue monitoring sensor: Detects the driver's eye movements and facial expressions.

[1675] Emotion Engine: Identifies the driver's emotional state from their tone of voice and facial expressions.

[1676] Data transmission and analysis

[1677] Device → Server:

[1678] 1. Collected sensor data is periodically sent to the server.

[1679] Data includes speed, frequency of brake use, steering angle, eye movements and voice tones.

[1680] server:

[1681] 1. Input the received data into the generative model and emotion engine.

[1682] 2. The generative model analyzes driving skill and fatigue level in real time.

[1683] For example, if a driver blinks frequently, the generative model will determine that the driver is highly fatigued.

[1684] 3. The emotion engine analyzes the driver's emotional state from their tone of voice and facial expressions.

[1685] For example, if a person's voice is trembling or their facial expression shows signs of tension, it is determined that they are feeling stressed.

[1686] Proposal generation and notification

[1687] server:

[1688] 1. Evaluate the driver's condition based on the analysis results.

[1689] For example, if the driver is highly fatigued or stressed, it is recommended to switch to autonomous driving mode.

[1690] 2. Generate a suggestion: "We recommend transitioning to autonomous driving mode."

[1691] Server → Device:

[1692] 1. Sending generated suggestions and warnings to the device.

[1693] Device:

[1694] 1. Notify the driver of suggestions via display and audio feedback.

[1695] For example, it may notify you, "Your stress level is high. We recommend switching to autonomous driving mode."

[1696] Processing verbal feedback

[1697] User:

[1698] 1. Providing verbal feedback on proposals and notifications.

[1699] For example, reply, "Yes, please."

[1700] Device:

[1701] 1. Convert verbal feedback into text data using speech recognition technology.

[1702] 2. Send the textual feedback to the server.

[1703] Feedback analysis and system improvement

[1704] server:

[1705] 1. Analyze the verbal feedback received.

[1706] 2. Based on the feedback results, the generative model and emotion engine are retrained to improve the system.

[1707] Communication in urban areas

[1708] Device → Server:

[1709] 1. Collecting data for speech and gesture recognition during autonomous driving in urban areas.

[1710] 2. Send the collected data to the server.

[1711] server:

[1712] 1. The generative model generates an appropriate response based on the received data.

[1713] For example, if a pedestrian raises their hand, the system understands their intention and determines the appropriate action to take.

[1714] 2. Send the generated responses and instructions to the terminal.

[1715] Device:

[1716] 1. Notify people around you of received instructions via voice or display.

[1717] For example, instructions such as "Please go ahead" are announced through the speaker.

[1718] In this way, by using the system of the present invention, it is possible to increase social acceptance of autonomous driving technology while ensuring the safety and comfort of drivers.

[1719] The processing flow will be explained below.

[1720] Program processing steps

[1721] Step 1: Initialize the sensor

[1722] Device:

[1723] 1. Launch an integrated system for autonomous vehicles.

[1724] 2. Initialize the vehicle's cameras, lidar, GPS, fatigue monitoring sensors, and emotion engine.

[1725] 3. Start collecting data from the sensors.

[1726] Step 2: Collect sensor data

[1727] Device:

[1728] 1. Collect data in real time from various sensors on the vehicle.

[1729] Camera: Captures visual information about the surroundings.

[1730] Lidar: Obtains distance information to obstacles and other vehicles.

[1731] GPS: Get your current location.

[1732] Fatigue monitoring sensor: Detects the driver's eye movements and facial expressions.

[1733] Emotion Engine: Identifies the driver's emotional state from their tone of voice and facial expressions.

[1734] Step 3: Sending data

[1735] Device → Server:

[1736] 1. Collected sensor data is sent to the server at regular intervals.

[1737] Data includes speed, frequency of brake use, steering angle, eye movements and voice tones.

[1738] Step 4: Data analysis

[1739] server:

[1740] 1. Input the received sensor data into the generative model and emotion engine.

[1741] 2. The generative model analyzes driving skill and fatigue level in real time.

[1742] Driving behavior patterns are analyzed and compared with normal driving behavior.

[1743] Detect signs of fatigue (e.g., frequent blinking).

[1744] 3. The emotion engine analyzes the driver's emotional state from their tone of voice and facial expressions.

[1745] Stress is detected when the voice tone is high and the voice is trembling.

[1746] If the facial muscles are tense, assess anxiety or tension.

[1747] Step 5: Generate proposals

[1748] server:

[1749] 1. Evaluate the driver's condition based on the analysis results.

[1750] For example, if the level of fatigue is high, it is determined that "the driver is fatigued."

[1751] Emotional states such as stress and tension are also assessed.

[1752] 2. Generate a proposal to transition to autonomous driving mode.

[1753] For example, create a message that reads, "Your fatigue and stress levels are high, so we recommend switching to autonomous driving mode."

[1754] Step 6: Proposal Notification

[1755] Server → Device:

[1756] 1. Send generated suggestions and warnings to the device.

[1757] Device:

[1758] 1. Notify the driver of suggestions via display and audio feedback.

[1759] For example, a voice notification may say, "You are highly fatigued, so we recommend switching to autonomous driving mode."

[1760] Step 7: Gathering human feedback

[1761] User:

[1762] 1. Providing verbal feedback on proposals and notifications.

[1763] For example, reply, "Yes, please."

[1764] Device:

[1765] 1. Convert verbal feedback into text data using speech recognition technology.

[1766] 2. Send the textual feedback to the server.

[1767] Step 8: Analyze feedback and improve the system

[1768] server:

[1769] 1. Analyze the verbal feedback received.

[1770] 2. Based on the feedback results, the generative model and emotion engine are retrained to improve the system.

[1771] For example, if the driver instructs the system to "switch to autonomous driving mode," the generative model is updated accordingly.

[1772] Step 9: Communicating in the city

[1773] Device → Server:

[1774] 1. Collecting data for speech and gesture recognition during autonomous driving in urban areas.

[1775] 2. Send the collected data to the server.

[1776] server:

[1777] 1. The generative model generates an appropriate response based on the received data.

[1778] For example, if a pedestrian raises their hand, the system understands their intention and determines the appropriate action to take.

[1779] 2. Send the generated responses and instructions to the terminal.

[1780] Device:

[1781] 1. Notify people around you of received instructions via voice or display.

[1782] For example, instructions such as "Please go ahead" are announced through the speaker.

[1783] Through these detailed steps, the system of the present invention can ensure driver safety and comfort and increase social acceptance of autonomous driving technology.

[1784] Example 2

[1785] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1786] Conventional driving assistance systems have struggled to assess the driver's driving skill, fatigue level, and emotional state in real time and provide appropriate driving assistance based on that assessment. Furthermore, there was a lack of a way to incorporate driver feedback into system improvements, making continuous system improvement difficult. Furthermore, they lacked a function to support communication with surrounding people when the vehicle is stopped in urban areas.

[1787] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1788] In this invention, the server includes means for collecting sensor data from the vehicle to evaluate driving skill, means for analyzing the sensor data and using a generative model to evaluate the driver's fatigue level, means for using an emotion engine to analyze the driver's emotional state, means for proposing transition to autonomous driving mode based on the analysis results, means for notifying the driver of the suggestion, means for converting verbal feedback from the driver into text data using speech recognition technology, means for collecting driver feedback and using it to improve the system, and means for supporting communication with surrounding people when the vehicle is stopped. This enables the driver's driving skill, fatigue level, and emotional state to be evaluated in real time, and appropriate driving assistance based on the evaluation. Furthermore, collecting feedback and using it to improve the system promotes continuous improvement. Furthermore, smooth communication with surrounding people when the vehicle is stopped in urban areas is possible.

[1789] "Sensor data" refers to data obtained from multiple sensors installed in a vehicle and used to evaluate the driver's driving skill, fatigue level, and emotional state.

[1790] A "generative model" is a machine learning algorithm that analyzes collected sensor data to evaluate a driver's driving skill and fatigue level.

[1791] The "emotion engine" is an algorithm that analyzes the driver's emotional state from their tone of voice and facial expressions, and identifies emotions such as stress and anxiety.

[1792] "Autonomous driving mode" is a mode in which the system controls the vehicle on behalf of the driver, and switches automatically depending on the driver's level of fatigue and emotional state.

[1793] "Suggestions" refer to instructions or advice that the system notifies the driver based on the analysis results, including, for example, a notification recommending switching to autonomous driving mode.

[1794] "Feedback" is the verbal or other responses or opinions from the driver that are collected and used to improve the system.

[1795] "Voice recognition technology" is a technology that converts the driver's voice into a digital signal and analyzes it as text data.

[1796] "System improvement" is the process of retraining the generative model and emotion engine based on collected feedback to improve the overall performance of the safe driving support system.

[1797] "Communication Support" is an assistance function that facilitates smooth interactions with people around the vehicle when stopped, and provides appropriate responses and instructions via voice and display.

[1798] This invention relates to a safe driving support system that uses in-vehicle sensors to collect driving data and analyzes it using a generative AI model and emotion engine to evaluate the driver's driving skill, fatigue level, and emotional state. Based on the analysis results, the system proposes an appropriate transition to an autonomous driving mode and notifies the driver. Furthermore, verbal feedback from the driver is collected and used to retrain the system, ensuring continuous improvement. Furthermore, when driving in urban areas, the system supports communication with surrounding people.

[1799] Hardware and software used

[1800] Hardware:

[1801] Camera: Captures visual information about the vehicle's surroundings.

[1802] Lidar: Obtains distance information to obstacles.

[1803] GPS: Get your current location.

[1804] Fatigue monitoring sensor: Detects eye movements and facial expressions.

[1805] Emotion Engine: Identify emotional states from voice tone and facial expressions.

[1806] software:

[1807] Generative AI model: Uses collected data to analyze driving skills and fatigue levels in real time.

[1808] Speech recognition technology: Analyzes verbal feedback and converts it into text data.

[1809] Explanation of program processing

[1810] Device:

[1811] Various sensors installed inside the vehicle collect data in real time and send it to a server. Specifically, the camera captures visual information about the surroundings, the lidar measures the distance to obstacles, and the GPS tracks location information. In addition, a fatigue monitoring sensor monitors the driver's eye movements and facial expressions, and an emotion engine analyzes the driver's emotional state from voice tone and facial expressions. For example, if the driver blinks frequently, it is determined that the driver is highly fatigued.

[1812] server:

[1813] The server receives sensor data sent from the device and inputs it into the generative AI model and emotion engine for analysis. The generative AI model evaluates driving skill and fatigue level, and the emotion engine analyzes the driver's emotional state. Based on the analysis results, it determines whether switching to autonomous driving mode is necessary and generates a proposal to do so. The proposal or warning is sent to the device and notified to the driver. If the driver answers "yes," the system switches to autonomous driving mode. In addition, verbal feedback from the driver is converted into text data using voice recognition technology and used to improve the system. For example, the system can switch to autonomous driving by replying "yes" to a notification that says, "We recommend switching to autonomous driving mode."

[1814] User:

[1815] The driver responds to the suggestions and warnings provided and provides verbal feedback. For example, if the driver receives a notification saying, "Your fatigue level is increasing. We recommend switching to autonomous driving mode," and responds, "Yes, please," the system will switch to autonomous driving mode. Additionally, when the vehicle is stopped in urban areas, the system will provide appropriate instructions to nearby pedestrians via voice, such as, "Please go ahead and let the vehicle pass."

[1816] Examples of concrete examples and prompts

[1817] Examples:

[1818] If the driver feels fatigued and blinks more frequently while driving on a highway, the system will issue a voice notification saying, "Your fatigue level is high, so we recommend switching to autonomous driving mode." If the driver replies "yes," the system will switch to autonomous driving mode.

[1819] Example prompt sentence:

[1820] "Analyze data from vehicle sensors as follows and suggest transitioning to autonomous driving mode: When the driver is highly fatigued and stressed."

[1821] As described above, the safe driving support system of the present invention can provide advanced support for driver safety and comfort by combining a wide range of sensors with an advanced AI model and emotion engine.

[1822] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1823] Step 1:

[1824] Sensor data collection

[1825] Device: Collects data in real time from various sensors installed in the vehicle. Specifically, the camera captures visual information of the surroundings, the lidar measures the distance to obstacles, and the GPS acquires the current location information. The fatigue monitoring sensor monitors the driver's eye movements and facial expressions, and the emotion engine analyzes the driver's emotional state from voice tone and facial expressions.

[1826] Inputs: Data from camera, lidar, GPS, fatigue monitoring sensors, emotion engine.

[1827] Output: Camera visual information, rider distance information, GPS location information, fatigue level information, emotional state information.

[1828] Step 2:

[1829] Sending collected data

[1830] Device → Server: Collected sensor data is periodically sent to the server, including speed information, braking frequency, steering angle, eye movements, and audio tones.

[1831] Input: Data collected from various sensors.

[1832] Output: Sensor data sent to the server.

[1833] Step 3:

[1834] Data analysis

[1835] Server: The received data is input into the generative AI model and emotion engine to analyze driving skill and fatigue level. The generative AI model evaluates driving skill and determines fatigue level based on factors such as frequent blinking. The emotion engine analyzes the driver's emotional state based on voice tone and facial expressions. For example, if the voice is trembling, it is interpreted as being stressed.

[1836] Input: Sensor data sent to the server.

[1837] Output: Driving skill evaluation results, fatigue evaluation results, emotional state evaluation results.

[1838] Step 4:

[1839] Proposal Generation

[1840] Server: Based on the analysis results, evaluates the driver's condition and determines whether switching to autonomous driving mode is necessary. For example, if the driver's fatigue level or stress level is high, a recommendation is generated to "recommend switching to autonomous driving mode."

[1841] Input: Driving skill evaluation results, fatigue evaluation results, emotional state evaluation results.

[1842] Output: Proposal to transition to autonomous driving mode.

[1843] Step 5:

[1844] Proposal Notification

[1845] Server → Device: Sends generated suggestions and warnings to the device.

[1846] Device: Notify the driver of the suggestion through a display or audio feedback, for example, "Your stress level is high. We recommend switching to autonomous driving mode."

[1847] Input: Proposal to transition to autonomous driving mode.

[1848] Output: Display of suggestion notification and audio feedback.

[1849] Step 6:

[1850] Processing Feedback

[1851] User: The driver provides verbal feedback to the suggestion or notification, for example, by replying "Yes, please."

[1852] Terminal: Verbal feedback is converted into text data using speech recognition technology and sent to the server.

[1853] Input: User's verbal feedback.

[1854] Output: Feedback converted to text data.

[1855] Step 7:

[1856] Feedback analysis and system improvement

[1857] Server: Analyzes the received verbal feedback and retrains the generative model and emotion engine to improve the system.

[1858] Input: Feedback converted to text data.

[1859] Output: Improved generative model and emotion engine.

[1860] Step 8:

[1861] Communication support in urban areas

[1862] Terminal → Server: Collects data to recognize speech and gestures during autonomous driving in urban areas and sends it to the server.

[1863] Server: The generative model generates an appropriate response based on the received data. For example, if a pedestrian raises their hand, the server understands their intention and determines the appropriate action. The server then sends the generated response and instructions to the device.

[1864] Terminal: The terminal notifies people around it of the received instructions through voice or display. For example, it may announce the instruction "Please go ahead" through a speaker.

[1865] Input: Speech and gesture data.

[1866] Output: Audio and display notifications for people around you.

[1867] (Application example 2)

[1868] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1869] Conventional driving support systems have difficulty accurately analyzing the driver's fatigue level and emotional state in real time, resulting in an inability to suggest transitioning to autonomous driving mode at the appropriate time. Effectively collecting feedback from the driver and utilizing it for continuous system improvement has also been a challenge. Furthermore, smooth communication with surrounding people in urban areas is also required.

[1870] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for collecting the driver's facial expressions, voice tone, and driving behavior data in real time using a smartphone; means for transmitting the collected data to the server and analyzing it using an AI generation model and an emotion engine; means for notifying the driver via audio or visual means on the smartphone based on the analysis results; means for converting the driver's verbal feedback into text data using voice recognition technology and transmitting it to the server; and means for supporting communication with surrounding people while driving. This makes it possible to accurately grasp the driver's fatigue level and emotional state and suggest switching to autonomous driving mode at the appropriate time. Furthermore, it is possible to effectively collect driver feedback and achieve continuous system improvement. Furthermore, it also enables smooth communication in urban areas.

[1871] A "driving skill assessment" is an assessment of how safely and efficiently a driver can drive.

[1872] "Sensor data" refers to information collected from various sensors, including camera footage, audio, acceleration, and location information.

[1873] A "generative model" is an algorithm that uses AI technology to analyze data for a specific purpose and predict an outcome.

[1874] "Fatigue assessment" is the process of measuring how tired a driver is.

[1875] "Autonomous driving mode" refers to a mode in which the vehicle performs driving operations autonomously.

[1876] "Notification" refers to the act of the system informing the driver of information, and includes audio and visual notifications.

[1877] "Feedback" refers to the opinions and reactions provided by the driver to the system.

[1878] "System improvement" means improving the performance and accuracy of the system based on collected feedback and data.

[1879] A "smartphone" is a device that combines advanced computing power and connectivity with a mobile phone.

[1880] "Facial expression" refers to the facial state that is revealed by the movement of the driver's facial muscles.

[1881] "Voice tone" refers to the pitch, strength, and emotional characteristics of a driver's voice.

[1882] "Driving behavior data" refers to data related to the driver's driving operations, including acceleration, deceleration, braking, etc.

[1883] The "emotion engine" is an algorithm that analyzes voice tone and facial expressions to detect the driver's emotional state.

[1884] "Speech recognition technology" refers to the technology that converts voice into text data.

[1885] "Communication in urban areas" refers to smoothly exchanging information with surrounding pedestrians and other vehicles while driving.

[1886] The safe driving support system of the present invention evaluates the driver's driving skill, fatigue level, and emotional state, and suggests transitioning to autonomous driving mode at an appropriate time. This system uses a smartphone, a server, and various sensors.

[1887] Hardware Configuration

[1888] 1. Smartphone: A smartphone equipped with a camera, microphone, accelerometer, and GPS is used to collect the driver's facial expressions, voice tone, and driving behavior data in real time.

[1889] 2. Server: A server equipped with a high-performance GPU is used. The server receives data sent from the smartphone and analyzes it using a generative AI model and emotion engine.

[1890] 3. Communication network: Data is sent and received using communication networks such as Wi-Fi and 4G / 5G.

[1891] Software Configuration

[1892] 1. Smartphone app: Install the app that collects driving data, sends the data to a server, notifies the driver of the analysis results, and collects driver feedback. This app works on Android or iOS.

[1893] 2. Speech Recognition System: Use speech recognition technology, such as Google Cloud Speech-to-Text, to convert the driver's verbal feedback into text data.

[1894] 3. Generative AI model and emotion engine: This model will be equipped with algorithms to analyze driving data and evaluate driving skill, fatigue level, and emotional state. These algorithms will be implemented using machine learning libraries such as PyTorch and TensorFlow.

[1895] System Operation

[1896] The server receives driving data sent from the smartphone. The received data is analyzed using a generative AI model and an emotion engine. This analysis evaluates the driver's driving skill, fatigue level, and emotional state. For example, if the server determines that the driver is very tired, it generates a notification suggesting, "Do you want to switch to autonomous driving mode?" This notification is communicated to the driver via audio or visual means via the smartphone.

[1897] Driver feedback is also important. When the driver responds verbally to suggestions and notifications, the smartphone's voice recognition system converts the speech into text data and sends it to the server. The server analyzes the feedback and uses the results to retrain the generative AI model and emotion engine, ensuring continuous improvement of the system.

[1898] To facilitate smooth communication in urban areas while driving, the system also sends voice messages to enable drivers to safely and smoothly exchange information with those around them, and this function also supports communication with pedestrians and other vehicles.

[1899] Examples and prompts

[1900] As a concrete example, if a driver continues driving for a long period of time, their facial expressions are recorded by a camera mounted on their smartphone. The generative AI model determines that the driver is highly fatigued from their facial expression and issues a message saying, "You seem tired. Would you like to switch to autonomous driving mode?" If the driver replies "yes," the voice recognition system converts the speech into text data and sends it to a server. Based on this feedback, the system switches to autonomous driving mode.

[1901] Example prompt sentence:

[1902] "The system analyzes the driver's emotional state from their facial expressions and vocal tone to assess their driving skill and fatigue level. If it determines that their fatigue level is high, it notifies the driver, 'Do you want to switch to autonomous driving mode?' and receives verbal feedback. It communicates with the server and uses the driver's feedback to improve the system."

[1903] This will improve driver safety and comfort, and also increase social acceptance of autonomous driving technology.

[1904] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1905] Step 1:

[1906] The device (smartphone) uses a camera, microphone, acceleration sensor, and GPS to collect the driver's facial expressions, voice tone, and driving behavior data in real time while driving. Inputs include the driver's facial image, voice, acceleration data, and location information. This data is collected from various sensors and stored on the device.

[1907] Step 2:

[1908] The terminal periodically transmits the collected data to the server. The driving data collected by the terminal is used as input, and the data is transmitted to the server as output. This transmission is performed using a communication network (Wi-Fi, 4G / 5G).

[1909] Step 3:

[1910] The server receives the data sent from the device and inputs it into the generative AI model and emotion engine. The input includes driving data (facial images, audio, driving behavior data, and location information). The server passes the received data to the AI ​​model and begins analysis.

[1911] Step 4:

[1912] The generative AI model analyzes driving data and evaluates driving skill and fatigue level in real time. Facial images and driving behavior data are used as input, and the output is an evaluation result of driving skill and fatigue level. For example, if a driver blinks frequently, the model will determine that the driver is highly fatigued.

[1913] Step 5:

[1914] The emotion engine analyzes the driver's emotional state from their voice tone and facial expression. It uses voice data and facial images as input, and generates an evaluation result of the driver's emotional state as output. For example, if the voice tone is unstable, the server determines that the driver is stressed.

[1915] Step 6:

[1916] The server evaluates the driver's state based on the analysis results of the generative AI model and emotion engine. The input includes the evaluation results of driving skill, fatigue level, and emotional state. The output generates a proposal to switch to autonomous driving mode if necessary. For example, a proposal such as "Do you want to switch to autonomous driving mode?" is generated.

[1917] Step 7:

[1918] The server sends the generated proposal to the terminal. The proposal content is used as input, and the proposal is sent to the terminal as output. The terminal notifies the driver of this proposal by voice or visual means. Specifically, the smartphone speaker will say, "Do you want to switch to autonomous driving mode?"

[1919] Step 8:

[1920] The user (driver) provides verbal feedback in response to the notification from the terminal. The driver's voice is used as input, and the voice data is collected by the terminal as output. For example, the driver replies "Yes."

[1921] Step 9:

[1922] The terminal converts the collected voice data into text data using a voice recognition system. The driver's voice data is used as input, and text data is generated as output. This converts the voice "yes" into the text "yes."

[1923] Step 10:

[1924] The device sends textual feedback to the server. Text data is used as input, and feedback is sent to the server as output. The server analyzes the received feedback and retrains the AI ​​model and emotion engine based on the results to improve the system. Specifically, data on how the driver responded to each notification is accumulated.

[1925] Step 11:

[1926] During autonomous driving in urban areas, the server and terminal collect data to communicate with surrounding pedestrians and other vehicles and send it to the server. Surrounding audio and image data are used as input, and sent to the server as output. The server generates an appropriate response based on the received data and sends it to the terminal. The terminal notifies surrounding people of the generated response by voice or display. For example, an instruction such as "Please go ahead" may be issued through the speaker.

[1927] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1928] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1929] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1930] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1931] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1932] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1933] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1934] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1935] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1936] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1937] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1938] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1939] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1940] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1941] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1942] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1943] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1944] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1945] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1946] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1947] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1948] The following is further disclosed regarding the above embodiment.

[1949] (Claim 1)

[1950] means for collecting sensor data from the vehicle for driving skill assessment;

[1951] a means for analyzing sensor data and using a generative model to perform driver fatigue assessment;

[1952] A means for proposing a transition to an autonomous driving mode based on the analysis results; and

[1953] means for notifying the driver of the proposal;

[1954] A means of collecting human feedback to help improve the system; and

[1955] Safe driving support system including.

[1956] (Claim 2)

[1957] 10. The system of claim 1, wherein the system provides audio or visual notifications of suggestions depending on the driver's condition.

[1958] (Claim 3)

[1959] The system according to claim 1, which collects sensor data in real time and outputs analytical results at high speed using a generative model.

[1960] (Claim 4)

[1961] The system of claim 1, wherein the generative model analyzes the driver's state and determines the driver's driving skill and fatigue level.

[1962] (Claim 5)

[1963] 2. The system of claim 1, wherein the driver's feedback is converted into text data using voice recognition technology and analyzed.

[1964] (Claim 6)

[1965] 6. The system of claim 5, wherein the generative model is retrained based on the collected feedback to improve the system.

[1966] (Claim 7)

[1967] 10. The system of claim 1, further comprising means for recognizing speech and gestures during autonomous driving in city streets and providing appropriate responses through a generative model.

[1968] (Claim 8)

[1969] The system according to claim 7, which notifies people in the vicinity of reactions and instructions via the generative model via audio or visual means.

[1970] (Claim 9)

[1971] The system of claim 1, wherein the driver's real-time data is constantly transmitted to a server, and the server uses the generative model to provide safe driving information.

[1972] "Example 1"

[1973] (Claim 1)

[1974] means for collecting sensor data from the vehicle for driving skill assessment;

[1975] a means for analyzing sensor data and using a generative model to perform driver fatigue assessment;

[1976] A means for proposing a transition to an autonomous driving mode based on the analysis results; and

[1977] means for notifying the driver of the proposal;

[1978] A means of collecting user feedback to help improve the system; and

[1979] A means of recognizing speech and gestures and providing a response based on the surrounding context;

[1980] A system including:

[1981] (Claim 2)

[1982] 10. The system of claim 1, wherein the system provides audio or visual notifications of suggestions depending on the driver's condition.

[1983] (Claim 3)

[1984] The system according to claim 1, which collects sensor data in real time and outputs analytical results at high speed using a generative model.

[1985] "Application Example 1"

[1986] (Claim 1)

[1987] means for collecting sensor data from the vehicle for driving skill assessment;

[1988] a means for analyzing sensor data and using a generative model to perform driver fatigue assessment;

[1989] A means of monitoring and evaluating the driver's eye movements and facial expressions in real time;

[1990] A means for proposing a transition to an autonomous driving mode based on the analysis results; and

[1991] means for notifying the driver of the proposal;

[1992] a means for collecting verbal feedback from the driver and converting it into text data using speech recognition technology;

[1993] A means for analyzing the converted text data and using it to improve the system;

[1994] A means of recognizing speech and gestures and providing appropriate responses during autonomous driving in urban areas;

[1995] A system including:

[1996] ...

Claims

1. means for collecting sensor data from the vehicle for driving skill assessment; a means for analyzing sensor data and using a generative model to perform driver fatigue assessment; A means for proposing a transition to an autonomous driving mode based on the analysis results; and means for notifying the driver of the proposal; A means of collecting human feedback to help improve the system; and Safe driving support system including.

2. 10. The system of claim 1, wherein the system notifies the driver of the suggestion by voice or visually depending on the driver's condition.

3. The system according to claim 1 , wherein sensor data is collected in real time and analysis results are output at high speed using a generative model.

4. The system according to claim 1 , wherein the generative model analyzes the driver's state and determines the driver's driving skill and fatigue level.

5. 2. The system according to claim 1, wherein the driver's feedback is converted into text data using voice recognition technology and analyzed.

6. The system of claim 5, wherein the generative model is retrained based on the collected feedback to improve the system.

7. The system of claim 1 , further comprising means for recognizing speech and gestures during autonomous driving in a city and providing appropriate responses through a generative model.

8. The system according to claim 7, wherein reactions and instructions via the generative model are notified to people in the vicinity via voice or visual.

9. The system of claim 1, wherein the driver's real-time data is constantly transmitted to a server, and the server uses the generative model to provide safe driving information.

Images