System

Abstract

To provide a system capable of improving a system based on feedback of a user while providing safe driving support according to a state of a driver and a surrounding traffic situation.SOLUTION: An autonomous driving assistance system including: means for evaluating a degree of fatigue of a driver; means for collecting and analyzing sensor data; means for evaluating a driving skill of the driver; means for proposing a transition to an autonomous driving mode; and means for collecting feedback from a user and improving a system.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] With the recent spread and development of autonomous driving technology, there is a need for appropriate responses to variations in driver skill, fatigue level, and surrounding traffic conditions. Furthermore, to increase the reliability and social acceptance of autonomous driving systems, it is necessary to continuously collect feedback from users and improve the system. However, collecting this information extensively and efficiently and appropriately analyzing it to enhance the safety of autonomous driving systems is a technically difficult challenge. [Means for solving the problem]

[0005] In order to solve the above-mentioned problems, the present invention provides the following means: An automated driving assistance system including means for assessing the driver's fatigue level, means for collecting and analyzing sensor data, means for assessing the driver's driving skill, means for proposing transition to automated driving mode, and means for collecting user feedback and improving the system. This makes it possible to provide safe driving support according to the driver's state and surrounding traffic conditions while improving the system based on user feedback, thereby improving the reliability and acceptability of the automated driving system.

[0006] "Driver fatigue" refers to the degree of fatigue and reduced attention a driver feels while driving.

[0007] "Sensor data" refers to data acquired from various sensors installed in a vehicle, and includes information such as speed, lane position, and distance to the vehicle ahead.

[0008] "Driving ability" evaluates a driver's driving skills and abilities, and indicates the driver's ability to make appropriate decisions and operate with precision while driving.

[0009] "Autonomous driving mode" refers to a mode in which the vehicle drives autonomously without driver operation.

[0010] "Feedback" refers to opinions and evaluations provided by users to the system, including information that can be used to improve the system.

[0011] An "autonomous driving assistance system" is a system designed to reduce the burden on the driver and assist in safe and efficient driving.

[0012] "Safe driving support" includes information, warnings, and assistance functions to help avoid dangers that occur while driving and enable safe driving. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] This invention provides a specific method for evaluating driver fatigue and driving skills in an automated driving assistance system and proposing a transition to automated driving mode at an appropriate time. It also aims to enhance user trust and social acceptance by collecting user feedback and continuously improving the system.

[0035] Program processing

[0036] 1. System initialization

[0037] The server initializes the system. At the time of instantiation, the driver's current fatigue level, driving skill, collected sensor data, and past feedback are initialized.

[0038] 2. Sensor data collection

[0039] The device collects data from various sensors installed in the vehicle, including driving information such as the vehicle's speed, lane position, and distance to the vehicle ahead.

[0040] 3. Driver condition assessment

[0041] The device assesses the driver's level of fatigue, which is quantified based on certain criteria and indicates a decline in attention and reaction time while driving.

[0042] 4. Proposal for autonomous driving mode

[0043] The device will determine whether to switch to autonomous driving mode based on the collected data and the evaluation results, and will suggest switching to autonomous driving mode if the driver is highly fatigued or the driving conditions are difficult.

[0044] 5. Safe driving support

[0045] It analyzes sensor data and provides warnings and advice for safe driving as needed, for example, if the distance to the vehicle ahead is too close.

[0046] 6. Gathering Feedback

[0047] The user provides feedback about the driving experience, for example, by providing specific feedback such as whether the system is "too aggressive" or "too slow."

[0048] 7. System Improvements

[0049] The server improves the system based on the collected feedback. It analyzes the feedback, reevaluates the driver's driving skills based on the feedback, and adjusts the system's operating parameters.

[0050] Specific examples

[0051] Next, a specific example of the automated driving assistance system of the present invention will be shown. For example, if a driver has been driving for a long time, the system will evaluate the driver's fatigue level in real time. If the fatigue level exceeds a certain level, the device will suggest switching to automated driving mode. At this time, the driver will be notified by voice, "Do you want to switch the vehicle to automated driving mode?", ensuring a smooth transition to automated driving mode.

[0052] Additionally, if the distance to the vehicle ahead is significantly short while driving, the sensor will detect this information and the device will issue a warning saying, "You are getting too close to the vehicle ahead." This allows the driver to immediately recognize the situation and maintain safe driving.

[0053] Furthermore, suppose the user provides feedback after the drive, such as "My lane change was a bit extreme." The server analyzes this feedback and adjusts the system parameters for the next drive. In this way, the system is continuously improved based on the user's experience and opinions.

[0054] By combining these functions, the automated driving assistance system of the present invention can improve the safety and comfort of the driver.

[0055] The processing flow will be explained below.

[0056] Step 1:

[0057] The server initializes the system by creating an instance of UrbanCohesionDriveAssist, setting the driver's current fatigue level to 0, and the driving skill level to intermediate (5). It also initializes the sensor data and feedback lists as empty.

[0058] Step 2:

[0059] The terminal calls the collect_sensor_data method to collect data from various sensors installed in the vehicle. The collected data includes speed, lane position, and distance to the vehicle ahead. A random number is generated for each piece of data and recorded as sensor data.

[0060] Step 3:

[0061] The device calls the evaluate_driver_state method to evaluate the driver's fatigue level. The fatigue level increases by 1 and is adjusted downward according to the driver's driving skill. As a result, the fatigue level is set so that it will not fall below 0.

[0062] Step 4:

[0063] The device calls the evaluate_and_propose_autonomous_mode method to propose switching to autonomous driving mode if the driver's fatigue level exceeds a certain threshold (e.g., 10). If the proposal is accepted, the switch_to_autonomous_mode method is called to switch to autonomous driving mode.

[0064] Step 5:

[0065] The device calls the provide_safety_support method to provide safe driving support based on the collected sensor data. For example, if the distance to the vehicle ahead is less than 20 meters, the device warns the driver that the vehicle is "too close to the vehicle ahead."

[0066] Step 6:

[0067] The user provides feedback using the collect_feedback method. Feedback includes their opinion and evaluation of the driving experience, such as "the system is too aggressive" or "it's too slow."

[0068] Step 7:

[0069] The server calls the improve_system_based_on_feedback method to analyze the collected feedback and improve the system. It evaluates the feedback and adjusts the driver's driving skills and the system's operating parameters accordingly. The improved settings are then applied the next time the vehicle is driven.

[0070] Example 1

[0071] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0072] Conventional automated driving assistance systems have problems such as insufficient assessment of driver fatigue and driving skills, making it impossible to suggest switching to automated driving mode at the appropriate time. They also lack the ability to provide real-time safe driving support based on sensor data and improve the system based on user feedback. As a result, they have been unable to fully ensure driver safety and comfort.

[0073] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0074] In this invention, the server includes a means for assessing the driver's fatigue level, a means for collecting sensor data from various sensors and analyzing it in real time, a means for assessing the driver's driving skill, a means for determining whether or not switching to autonomous driving mode is necessary and making a recommendation, a means for providing warnings and advice for safe driving as necessary based on the collected sensor data, and a means for collecting feedback from users and improving the system based on that feedback. This makes it possible to accurately assess the driver's fatigue level and driving skill and to suggest switching to autonomous driving mode at an appropriate time. It also makes it possible to provide safe driving support in real time and improve the system based on that feedback.

[0075] The "means for assessing the driver's level of fatigue" refers to a device or system that analyzes the driver's eye opening and closing and facial movements to quantify the level of fatigue.

[0076] "Various sensors" refers to a group of sensors that collect driving information such as vehicle speed, lane position, and distance to the vehicle ahead.

[0077] "Real-time analysis means" refers to devices or software for processing and analyzing collected sensor data in real time.

[0078] "Means for evaluating a driver's driving skills" refers to a system that analyzes the vehicle's driving patterns and the driver's operations, and quantifies or evaluates the skills.

[0079] "Means for determining whether or not switching to autonomous driving mode is necessary and proposing this" refers to a device or system that determines whether or not switching to autonomous driving mode is necessary based on collected data and evaluation results and notifies the driver.

[0080] "Means for providing warnings and advice for safe driving" refers to devices or software that provide instructions or warnings to drivers for safe driving based on data analyzed in real time.

[0081] "Means for collecting user feedback" refers to an interface through which drivers can input their opinions and thoughts about their driving experience, and a system for collecting that data.

[0082] A "means for improving the system" is a device or software that analyzes the collected feedback and adjusts or refines the operating parameters of the system.

[0083] MODE FOR CARRYING OUT THE INVENTION

[0084] This invention provides a specific method for an automated driving assistance system that evaluates the driver's fatigue level and driving skill and suggests transitioning to automated driving mode at an appropriate time. It also aims to enhance driver safety and comfort by collecting user feedback and continuously improving the system.

[0085] System configuration

[0086] The main components of the automated driving assistance system are a server, a terminal, and a user interface. The role and operation of each component are explained below.

[0087] Server Roles

[0088] The server plays a central role in the entire system, specifically implementing the following functions:

[0089] System initialization:

[0090] The server initializes the system, including the driver's current fatigue level, driving skill, collected sensor data, and past feedback.

[0091] Data Processing:

[0092] The server analyzes the collected sensor data and evaluates the driver's level of fatigue and driving skills.

[0093] System Improvements:

[0094] The server analyzes user feedback and adjusts the system's operating parameters.

[0095] Device Role

[0096] The terminal collects and analyzes data in real time from various sensors installed in the vehicle. Specific functions are as follows:

[0097] Sensor data collection:

[0098] The device collects data from sensors installed in the vehicle, including the vehicle's speed, lane position, distance to the vehicle ahead, and a driver fatigue index based on facial recognition.

[0099] Driver Condition Rating:

[0100] The device evaluates the driver's level of fatigue and driving skills based on the data collected.

[0101] Autonomous driving mode suggestions:

[0102] The device determines whether it is necessary to switch to autonomous driving mode and notifies the driver.

[0103] Safe driving support:

[0104] The device analyzes the data in real time and issues warnings to the driver as needed.

[0105] User Interface

[0106] The user interface is primarily responsible for:

[0107] Providing feedback:

[0108] The system provides an interface that allows users to easily provide feedback, such as specific opinions like "too extreme" or "too slow."

[0109] Notification function:

[0110] When suggesting transition to autonomous driving mode, audio and visual notifications will be provided.

[0111] Hardware and software used

[0112] Hardware:

[0113] Various sensors (speed sensors, lane position sensors, distance sensors, face recognition cameras, etc.)

[0114] On-board computer (terminal)

[0115] server

[0116] software:

[0117] Data Analysis Software

[0118] Feedback Analysis Algorithm

[0119] Notification and warning interface program

[0120] Specific examples

[0121] Driver fatigue assessment and autonomous driving mode proposal

[0122] As a specific example, consider a driver who continues to drive for long periods of time. The device evaluates the driver's fatigue level in real time, and if it exceeds a certain level, a voice notification will be issued asking, "Do you want to switch the vehicle to autonomous driving mode?" At this point, the driver can choose to switch to autonomous driving mode via the voice notification.

[0123] Safe driving support function

[0124] If the distance to the vehicle ahead becomes too short while driving, the device will issue a warning saying, "You are getting too close to the vehicle ahead," prompting the driver to take immediate action to increase their distance.

[0125] Gathering feedback and improving the system

[0126] After completing the drive, the user provides feedback such as "My lane change was a bit extreme." The server analyzes this feedback and adjusts the system's operating parameters for the next drive, improving the user experience.

[0127] Prompt Sentence Examples

[0128] Examples of prompts to enter into a generative AI model include:

[0129] Prompt: Describe the specific steps for fatigue assessment and feedback processing in an automated driving assistance system.

[0130] Objective: Learn more about how to assess driver fatigue in real time and suggest automated driving modes at the right time.

[0131] With such settings, the automatic driving assistance system of the present invention can improve the safety and comfort of the driver.

[0132] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0133] The flow of this system's program processing

[0134] Step 1:

[0135] System initialization

[0136] The server initializes the system. As input, it obtains the driver's current fatigue level, driving skill, collected sensor data, and past feedback data from the database. Data processing generates a driver profile and initializes driving tendencies. As output, it obtains an initialized system and an initial driver profile. Specifically, the server reads driving history data from the database and prepares for a new driving session.

[0137] Step 2:

[0138] Sensor data collection

[0139] The terminal collects data in real time from various sensors attached to the vehicle. The input includes data obtained from speed sensors, lane position sensors, distance sensors, face recognition cameras, etc. Data processing involves storing each sensor data in a buffer and converting it into the required data format. As an output, real-time data from the sensors is stored in the buffer. Specifically, the terminal collects the latest driving information every second and sends that information to the server in real time.

[0140] Step 3:

[0141] Driver condition assessment

[0142] The driver's condition is evaluated based on the sensor data collected by the device. The input includes real-time sensor data stored in a buffer. For data calculation, facial recognition technology is used to analyze the opening and closing of the driver's eyes and facial movements to quantify the level of fatigue. Driving patterns are also analyzed to evaluate driving skills. The output is a driver fatigue score and driving skill score. Specifically, an AI model is used to evaluate real-time data, and the driver's fatigue level is displayed on the screen.

[0143] Step 4:

[0144] Proposal for autonomous driving mode

[0145] The device will suggest switching to autonomous driving mode based on the evaluation results. Inputs include the driver's fatigue score and driving skill score. Data calculations compare the evaluation score with preset reference values ​​to determine whether it is necessary. As an output, a notification suggesting switching to autonomous driving mode is generated for the driver. Specifically, the device will issue a voice notification asking, "Do you want to switch the vehicle to autonomous driving mode?"

[0146] Step 5:

[0147] Safe driving support

[0148] The device analyzes sensor data in real time and provides warnings and advice for safe driving as needed. The input includes continuously collected sensor data. Data processing monitors the distance to the vehicle ahead and lane departure, generating a warning if certain criteria are exceeded. The output generates audio and visual warnings for the driver. Specifically, the device displays a warning message such as "You are getting too close to the vehicle ahead."

[0149] Step 6:

[0150] Gathering feedback

[0151] The user provides feedback on the driving experience. The input includes feedback data provided through the user interface. Data processing classifies the feedback and stores it as analysis data for system improvement. The output is the collected feedback stored on the server. Specifically, the user enters "The lane change was a bit extreme" into the smartphone app.

[0152] Step 7:

[0153] System Improvements

[0154] The server improves the system based on the collected feedback. Inputs include user feedback and past driving data. Data calculations involve analyzing the feedback, reassessing the driver's driving skills, and adjusting the system's operating parameters. The output is improved system settings. Specifically, the server analyzes the feedback and configures the system to operate with the new parameter set the next time the vehicle is driven.

[0155] In this way, automated driving assistance systems can increase driver safety and comfort.

[0156] (Application example 1)

[0157] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0158] Current autonomous driving technology does not adequately assess driver fatigue and driving skills, making it difficult to fully guarantee driver safety and comfort. Furthermore, there is a lack of mechanisms for detecting driver fatigue and loss of concentration in real time during long periods of driving and prompting the vehicle to transition to autonomous driving mode at the appropriate time. Furthermore, existing systems lack sufficient processes for collecting user feedback and improving the system, resulting in low reliability and social acceptance. Therefore, there is a need for an autonomous driving assistance system that can be continuously improved while achieving both safety and comfort.

[0159] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0160] In this invention, the server includes a means for assessing the driver's fatigue level, a means for collecting and analyzing sensor data, a means for assessing the driver's driving skill, a means for proposing transition to autonomous driving mode, a means for collecting user feedback and improving the system, and a means for providing visual and audio warnings to support the driver's safe driving. This makes it possible to evaluate the driver's condition in real time and suggest transitioning to autonomous driving mode at an appropriate time. It also supports safe driving by notifying the driver of dangerous situations while driving with visual and audio warnings. Furthermore, by collecting user feedback and continuously improving the system based on that feedback, it is possible to increase reliability and social acceptance.

[0161] The "means for assessing driver fatigue" refers to a method and device for quantifying and assessing the driver's fatigue level in real time based on facial recognition, eye movements, etc.

[0162] "Means for collecting and analyzing sensor data" refers to the technology and equipment that collects data such as speed, lane position, and distance to the vehicle ahead from various sensors attached to the vehicle and analyzes it.

[0163] The "means for assessing the driver's driving skills" refers to algorithms and devices for assessing the driving skills based on collected sensor data and the driver's driving history.

[0164] "Means for proposing transition to autonomous driving mode" refers to technologies and devices that notify and suggest transition to autonomous driving mode to the driver at an appropriate time based on the evaluation results.

[0165] "Means for collecting user feedback and improving the system" refers to technologies and devices that collect feedback provided by drivers based on their driving experience, analyze it, adjust system parameters, and continuously improve the system.

[0166] "Means for providing visual and audio warnings to support safe driving" refers to technologies and devices that provide visual and audio warnings to drivers based on collected sensor data and evaluation results, and encourage appropriate safe driving behavior.

[0167] The following configurations are included as embodiments of the present invention.

[0168] The server initializes the system and provides a means to evaluate the driver's fatigue level and driving skills. It uses data from various sensors attached to the vehicle (speed, lane position, distance to the vehicle ahead, etc.) to collect and analyze sensor data. The analyzed data is sent to the server, which evaluates the driver's condition in real time.

[0169] Based on the evaluation results, the device provides a means to suggest switching to autonomous driving mode. This suggestion is made using visual and audio output devices such as smart glasses. For example, if the driver's fatigue level exceeds a certain level, the device will notify the driver via audio and visual means, saying, "Your fatigue level is high. Would you like to suggest switching to autonomous driving mode?" If the driver answers "Yes" via audio, the device will smoothly switch to autonomous driving mode.

[0170] In addition, the device provides visual and audio warnings to help drivers drive safely. For example, if the distance to the vehicle ahead is significantly short, the device will warn the driver that they are driving too close to the vehicle ahead, allowing them to immediately recognize the situation and take appropriate action.

[0171] After completing a drive, users can provide feedback to the system. The feedback is sent to the server and used to improve the system. For example, specific opinions such as "Your lane changes were a bit extreme" can be reflected and the system's operating parameters will be adjusted for the next drive. This allows the system to continuously improve, increasing its reliability and social acceptance.

[0172] Description of the hardware and software used

[0173] Hardware: Smart glasses (e.g., Microsoft® HoloLens®, Google® Glass®), various in-vehicle sensors

[0174] Software: Driver state assessment algorithm, smart glasses SDK (e.g. GoggleSDK)

[0175] Data processing and calculation

[0176] The data is processed as follows:

[0177] 1. Sensor data collection:

[0178] Data is collected from various sensors installed in the vehicle.

[0179] 2. Driver Condition Assessment:

[0180] Based on the collected sensor data, algorithms are applied to assess fatigue levels and driving skills.

[0181] 3. Autonomous driving mode suggestions and warnings:

[0182] -Based on the evaluation results, the device will suggest transitioning to autonomous driving mode and provide visual and audio warnings.

[0183] Examples of concrete examples and prompts

[0184] Specific examples

[0185] When a driver is driving for a long period of time, the smart glasses monitor the driver's eye movements and facial expressions in real time to detect an increase in fatigue. As a result, a notification will appear on the smart glasses' HUD saying, "Fatigue level is high. Would you like to suggest switching to autonomous driving mode?" If the driver answers "Yes" aloud, the smart glasses will automatically switch to autonomous driving mode.

[0186] Prompt Sentence Examples

[0187] Consider a scenario in which smart glasses monitor the eye movements and facial expressions of a driver in real time while driving for long periods of time to detect increased fatigue, and explain the appropriate message display and voice response procedures.

[0188] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0189] Step 1:

[0190] System initialization

[0191] The server initializes the system. The inputs are the driver's current fatigue level, driving skill, collected sensor data, and past feedback. Based on this, all initial parameters are set. The output is the initialized system state. Specifically, the server resets all parameters and performs the initial settings.

[0192] Step 2:

[0193] Sensor data collection

[0194] The terminal collects data from various sensors attached to the vehicle. The input is real-time data such as the vehicle's speed, lane position, and distance to the vehicle ahead. This data is collected by the sensors and sent to the terminal. The output is a set of collected sensor data. Specifically, the terminal collects sensor data at regular intervals and stores it for analysis.

[0195] Step 3:

[0196] Driver condition assessment

[0197] The device evaluates the driver's fatigue level and driving skills based on the sensor data collected. The sensor data collected in step 2 is used as input. A data analysis algorithm quantifies the fatigue level and evaluates the driving skills. The output is the quantified fatigue level and driving skill evaluation results. Specifically, the device executes the data analysis algorithm and calculates the evaluation results.

[0198] Step 4:

[0199] Proposal for autonomous driving mode

[0200] Based on the evaluation results, the device will suggest switching to autonomous driving mode. The inputs are the fatigue level and driving skill evaluation results obtained in step 3. When the fatigue level exceeds a certain value, the device will output a suggestion notification. Specifically, it will display a message on the smart glasses' HUD and notify the driver by voice, "Your fatigue level is high. Would you like to suggest switching to autonomous driving mode?"

[0201] Step 5:

[0202] Safe driving support

[0203] The device analyzes sensor data in real time and provides warnings and advice for safe driving. The input is sensor data and the driver's state evaluation results. The output is visual and audio warning and advice messages. Specifically, the device monitors the distance to the vehicle ahead and warns the driver that "you are too close to the vehicle ahead" if the distance is too close.

[0204] Step 6:

[0205] Gathering feedback

[0206] The user provides feedback about the driving experience. The input is the specific feedback provided by the driver, including their opinions and thoughts about the system's behavior. The output is the collected feedback data. The specific operation is that the terminal receives the feedback from the user and sends it to the server.

[0207] Step 7:

[0208] System Improvements

[0209] The server adjusts and improves the system's operating parameters based on the collected feedback. The input is the feedback data collected in step 6. The feedback is analyzed and necessary adjustments are made. The output is the improved system's operating parameters. Specifically, the server runs the feedback analysis algorithm and updates the system settings.

[0210] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0211] This invention combines an emotion engine with an automated driving assistance system to improve the effectiveness of driver fatigue assessment and feedback collection, and to improve the accuracy of suggestions for switching to automated driving mode. The emotion engine recognizes emotions from the user's tone of voice, facial expressions, body movements, etc., and reflects them in the system.

[0212] Program processing

[0213] 1. System initialization

[0214] The server initializes the system. When creating an instance of UrbanCohesionDriveAssist, it initializes the driver's fatigue level, driving skill, sensor data, emotional data, and feedback.

[0215] 2. Collecting sensor data and emotion data

[0216] The device calls the collect_sensor_data method to collect data from the vehicle's sensors and the emotion engine that recognizes the user's emotions. The collected data includes speed, lane position, distance to the vehicle ahead, and the user's emotional state (e.g., stress level, distraction level, etc.).

[0217] 3. Driver condition assessment

[0218] The device calls the evaluate_driver_state method to comprehensively evaluate the driver's fatigue level based on emotional data and driving data, taking into account, for example, whether the driver is not only tired but also in a high-stress state.

[0219] 4. Proposal for autonomous driving mode

[0220] The device calls the evaluate_and_propose_autonomous_mode method and suggests switching to autonomous driving mode if the driver's state or emotional data exceeds a certain threshold. Specifically, the device encourages switching to autonomous driving mode even if the driver is feeling highly stressed based on emotional data.

[0221] 5. Safe driving support

[0222] The device analyzes sensor data and emotional data to provide safe driving support. For example, it not only issues a warning when the distance to the vehicle ahead is short, but also considers the causes of driver distraction based on emotional data and provides warnings and advice based on that.

[0223] 6. Gathering Feedback

[0224] When a user provides feedback using the collect_feedback method, we also collect emotional data, such as tone of voice and facial expressions, to assess the user's true perception of the quality of the experience.

[0225] 7. System Improvements

[0226] The server calls the improve_system_based_on_feedback method to analyze the feedback and emotional data. Based on this, the system can be improved. For example, if the user perceives the system as "extreme," and the emotional data indicates a stress response, the server can adjust the driver's skill level or the system's response speed.

[0227] Specific examples

[0228] Next, a specific example of the present invention will be shown. For example, when a driver has been driving for a long time, the system evaluates the driver's fatigue level in real time. This evaluation includes emotional data such as the driver's pale complexion or nervous expression. If the fatigue level exceeds a certain threshold, the device will issue a voice notification saying, "You seem tired. Would you like to switch to autonomous driving mode?" and suggest a smooth transition to autonomous driving mode.

[0229] Additionally, if the distance to the vehicle ahead is significantly short while driving, the sensor data will detect this information, and the emotion engine will sense anxiety from the driver's facial expression. Based on this information, the device will issue a warning to the driver saying, "You are getting very close to the vehicle ahead. Please be careful."

[0230] Furthermore, when a user provides feedback after completing a drive, such as "My lane change was a bit extreme," the emotion engine analyzes the user's tone of voice and facial expression when providing the feedback. From these results, it is determined that the user truly felt the lane change was extreme, and the server adjusts the system parameters for the next drive. For example, it may take measures such as reducing the speed when changing lanes.

[0231] In this way, by using an autonomous driving assistance system that combines an emotion engine, it becomes possible to provide advanced assistance that takes into account the driver's state and emotions, thereby improving safety and comfort.

[0232] The processing flow will be explained below.

[0233] Step 1:

[0234] The server initializes the system. When creating an instance of UrbanCohesionDriveAssist, it sets the driver's fatigue level to 0 and their driving skill to intermediate level (5). It also initializes the sensor data, emotion data, and feedback lists as empty.

[0235] Step 2:

[0236] The terminal calls the collect_sensor_data method to collect data from various sensors installed in the vehicle. The collected data includes speed, lane position, and distance to the vehicle ahead. This data is randomly generated and added to the sensor data list.

[0237] Step 3:

[0238] The device uses an emotion engine to collect emotional data from the driver's facial expressions, tone of voice, and body movements, for example, assessing whether the driver is tired or stressed and adding this to the emotional data list.

[0239] Step 4:

[0240] The device calls the evaluate_driver_state method to evaluate the driver's fatigue level based on the collected sensor data and emotional data. The fatigue level is calculated by an integrated analysis of the sensor data and emotional data. For example, if the driver's attention is distracted while driving and their facial expression indicates fatigue, the fatigue level is set high.

[0241] Step 5:

[0242] The device calls the evaluate_and_propose_autonomous_mode method to propose switching to autonomous driving mode if the driver's fatigue level or emotion data exceeds a certain threshold. If the proposal is accepted, the switch_to_autonomous_mode method is called to switch to autonomous driving mode.

[0243] Step 6:

[0244] The device calls the provide_safety_support method, analyzes sensor data and emotion data, and provides safe driving support. For example, if the distance to the vehicle ahead is too close and the emotion data indicates that the driver is nervous, the device will warn, "You are getting too close to the vehicle ahead. Please be careful."

[0245] Step 7:

[0246] Users provide feedback using the collect_feedback method. The feedback includes specific opinions about the driving experience, as well as emotional data at the time of the feedback. For example, the user's opinion that "the system's behavior was too aggressive" is recorded, along with the user's tone of voice and facial expression.

[0247] Step 8:

[0248] The server calls the improve_system_based_on_feedback method to analyze the collected feedback and emotion data. Based on the analysis results, the driver's driving skills are reevaluated and the system's operating parameters are adjusted. For example, if the feedback indicates that the driver's driving skills are "too aggressive," the system may adjust the lane change speed accordingly.

[0249] Example 2

[0250] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0251] While conventional automated driving assistance systems can partially take into account the driver's level of fatigue and driving skill, it is difficult to provide safe driving assistance that fully reflects the driver's emotional state or to improve the system based on feedback. Furthermore, emotional data is not sufficiently taken into account when maintaining a safe distance from the vehicle ahead or analyzing surrounding traffic conditions, leaving issues with safety and comfort. Therefore, there is a need for a system that can evaluate the driver's emotions in real time and provide appropriate assistance and suggestions.

[0252] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for evaluating the driver's fatigue level, a means for collecting and analyzing sensor data and emotional data, a means for evaluating the driver's driving skill and emotional state, a means for suggesting transition to an autonomous driving mode based on the driver's state and emotional data, and a means for collecting feedback from the user and analyzing the emotional data at the time of feedback to improve the system. This enables advanced support and suggestions that take the driver's emotional state into consideration, thereby improving safety and comfort.

[0253] "Fatigue level" refers to the degree of physical and mental fatigue a driver experiences while driving.

[0254] "Sensor data" refers to information collected by various sensors attached to a vehicle, and specifically includes data related to driving, such as speed, lane position, and distance to the vehicle ahead.

[0255] "Emotional data" refers to information about a driver's emotional state analyzed from their tone of voice, facial expressions, body movements, etc., and specifically includes stress levels and distractibility.

[0256] "Driving skill" is an evaluation of the driver's level of proficiency and skill in driving.

[0257] "Autonomous driving mode" is a mode in which the vehicle can operate autonomously without driver operation.

[0258] "Feedback" refers to opinions and evaluations provided by users to the system, and includes information that can be used to improve the system.

[0259] An "autonomous driving assistance system" is a system that assists drivers in driving and improves safety and comfort, and includes fatigue assessment, collection and analysis of sensor data and emotional data, driving skill assessment, autonomous driving mode proposals, feedback collection, and system improvement.

[0260] A "warning" is a notification or alert that notifies the driver of an impending hazard.

[0261] "Safe driving support" refers to advice and assistance provided to drivers to help them drive safely, and specifically includes maintaining a safe distance from the vehicle ahead and warning against dangerous driving.

[0262] MODE FOR CARRYING OUT THE INVENTION

[0263] This invention combines an emotion engine with an automated driving assistance system to improve the effectiveness of driver fatigue assessment and feedback collection, and to improve the accuracy of suggestions for switching to automated driving mode. The emotion engine recognizes emotions from the user's tone of voice, facial expressions, body movements, etc., and reflects them in the system.

[0264] System initialization

[0265] The server creates an instance of UrbanCohesionDriveAssist when the system is initialized. This instance is initialized with the driver's fatigue level, driving skill, sensor data, emotional data, and feedback. During initialization, the necessary data is synchronized using an existing database or cloud server.

[0266] Data collection

[0267] The device calls the collect_sensor_data method to collect data such as speed, lane position, and distance to the vehicle ahead from sensors installed in the vehicle. At the same time, the emotion engine analyzes the user's tone of voice, facial expressions, and body movements to collect emotional data such as stress level and distraction level. This provides both sensor data and emotional data to the system in real time.

[0268] Driver condition assessment

[0269] The device uses the evaluate_driver_state method to evaluate the driver's fatigue level based on the collected sensor data and emotional data. For example, a pale complexion and distractedness after a long drive could be used as evaluation points. The device also evaluates the driver's stress level based on the driver's tone of voice and facial expression, and combines this with the fatigue level evaluation to provide a comprehensive state evaluation.

[0270] Proposal for autonomous driving mode

[0271] The device calls the evaluate_and_propose_autonomous_mode method and proposes switching to autonomous driving mode if the driver's condition or emotional data exceeds a certain threshold. For example, if the user's facial expression indicates extreme fatigue, the device will issue a voice notification asking, "Do you want to switch to autonomous driving mode?"

[0272] Safe driving support

[0273] If sensor data indicates that the distance to the vehicle ahead is dangerously close, the device will issue a warning saying, "You are approaching the vehicle in front. Please be careful." If emotional data detects that the driver is anxious or distracted, the device will provide specific advice based on that emotional data, such as, "Take a deep breath and relax."

[0274] Gathering feedback and improving the system

[0275] When a user provides feedback after driving using the collect_feedback method, the emotion engine analyzes the tone of voice and facial expressions to evaluate the quality of the feedback. Based on this result, the server uses the improve_system_based_on_feedback method to improve the system. For example, taking into account past feedback and emotion data, the server can improve safety by reducing the lane-changing speed in the next driving session.

[0276] Specific examples

[0277] For example, if a driver has been driving for a long time, the server initializes the system and the device calls the appropriate method. The device uses the collect_sensor_data method to collect the vehicle's speed and distance from the vehicle ahead, and analyzes the driver's facial expression and tone of voice using the emotion engine. The emotion data and driving data are combined to evaluate the driver's condition. If the evaluation results indicate that the driver is fatigued and not paying attention, the device will suggest, "Do you want to switch to autonomous driving mode?" If the driver is dangerously close to the vehicle ahead, the device will warn, "You are very close to the vehicle ahead. Please be careful." After completing the drive, if the user provides feedback such as "My lane change was a bit aggressive," the emotion engine analyzes the user's tone of voice and facial expression to identify the specific reason why they thought the lane change was aggressive. The server uses this data to adjust the system parameters and reduce the lane change speed on the next drive.

[0278] An example of a prompt for a generative AI model is:

[0279] Please explain in natural language how the system detects the user's fatigue after a long drive and what suggestions it makes in response.

[0280] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0281] Step 1: Initialize the system

[0282] The server creates an instance of UrbanCohesionDriveAssist, which is initialized with the driver's fatigue level, driving skill, sensor data, emotional data, and feedback. The input is the initial setting data stored in an existing database or cloud server, and the output is the initialized system instance.

[0283] Step 2: Collecting sensor and emotion data

[0284] The device uses sensors installed in the vehicle and an emotion engine to collect driving-related sensor data and driver emotion data. Specifically, it calls the collect_sensor_data method to collect data such as speed, lane position, and distance to the vehicle ahead. It also analyzes stress levels and distraction levels from tone of voice, facial expressions, and body movements through the emotion engine. The inputs are the vehicle's sensor information and raw data from the emotion engine, and the output is analyzed sensor data and emotion data.

[0285] Step 3: Driver condition assessment

[0286] The device analyzes sensor data and emotion data to evaluate the driver's fatigue and stress levels. Specifically, it calls the evaluate_driver_state method to evaluate the facial expression and level of distraction of a driver who has been driving for a long time. The input is the collected sensor data and emotion data, and the output is the evaluation result, which includes the driver's fatigue and stress levels.

[0287] Step 4: Proposing an autonomous driving mode

[0288] The device proposes transitioning to autonomous driving mode based on the results of the driver's state evaluation. Specifically, it calls the evaluate_and_propose_autonomous_mode method, and if the driver's fatigue or stress level exceeds a certain threshold, it issues a voice notification suggesting, "Do you want to switch to autonomous driving mode?" The input is the driver's state evaluation result, and the output is a notification proposing transition to autonomous driving mode.

[0289] Step 5: Safe driving support

[0290] The device provides safe driving support to the driver based on the collected sensor data and emotional data. Specifically, it evaluates the distance to the vehicle ahead and issues a warning such as, "You are very close to the vehicle ahead. Please be careful." It also detects anxiety or distraction in the driver from emotional data and provides advice such as, "Take a deep breath and relax." The input is sensor data and emotional data, and the output is specific warnings and advice.

[0291] Step 6: Gather feedback

[0292] The user provides feedback after driving. The device uses the collect_feedback method to analyze the feedback along with the tone of voice and facial expression. The emotion engine is used to collect emotion data when providing the feedback and save the analysis results. The input is the user's feedback and emotion data, and the output is the analyzed feedback data.

[0293] Step 7: Improve your system

[0294] The server analyzes the collected feedback and emotion data and makes improvements to the system. Specifically, it calls the improve_system_based_on_feedback method to adjust system parameters based on the analyzed data. For example, it takes into account past feedback and emotion data and reduces the lane-changing speed in the next driving session. The input is the analyzed feedback data, and the output is the adjusted system parameters.

[0295] (Application example 2)

[0296] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0297] While conventional automated driving assistance systems can evaluate the driver's level of fatigue and driving skill, they do not adequately suggest switching to automated driving or provide safe driving support that accurately reflects the driver's emotional state. This means that the system is unable to respond quickly to sudden changes in the driver's physical condition or emotional fluctuations, which could compromise the safety and comfort of automated driving.

[0298] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for evaluating the driver's fatigue level, a means for collecting and analyzing sensor data, voice tone, and facial expressions, a means for evaluating the driver's driving skill and emotional state, a means for proposing transition to autonomous driving mode, and a means for collecting feedback from the user and improving the system. This makes it possible to evaluate the driver's state of fatigue, stress, and distraction in real time and propose transition to autonomous driving mode at an appropriate time.

[0299] "Driver fatigue level" refers to the degree of physical or mental fatigue a driver feels while driving.

[0300] "Sensor data" refers to information collected from sensors attached to vehicles and related equipment, and includes physical data such as speed, position, and distance.

[0301] "Voice tone" refers to characteristics of the driver's voice, such as pitch, strength, and intonation, and is information used to estimate the driver's emotional state.

[0302] "Facial expression" refers to the movement and changes of the driver's facial muscles, and is information used to analyze the driver's emotional state.

[0303] "Driving skills" refers to a driver's ability to operate a vehicle safely and efficiently, and its assessment includes the driver's experience and skill.

[0304] "Emotional state" refers to the driver's psychological state, including states such as stress, distraction, and relaxation.

[0305] "Autonomous driving mode" refers to a mode in which the vehicle drives automatically without driver intervention, aiming to improve safety and convenience.

[0306] "Feedback" refers to opinions and impressions from users, providing information about the performance and usability of the system.

[0307] "System Improvement" refers to actions taken to improve the performance and functionality of a system based on collected feedback and data.

[0308] This invention combines an emotion engine with an automated driving assistance system to evaluate the driver's fatigue level, driving skill, and emotional state, and proposes transitioning to automated driving mode at the appropriate time. The system uses smart glasses and other sensor devices to grasp the driver's emotional state in real time, and collects and analyzes feedback.

[0309] System configuration

[0310] Hardware:

[0311] Smart glasses: Equipped with a camera and microphone to collect the driver's facial expressions and voice.

[0312] Sensors installed in the vehicle collect driving data such as speed, lane position, and distance to the vehicle ahead.

[0313] software:

[0314] Emotion Engine: A model for analyzing a driver's emotional state from their facial expressions, vocal tone, and body movements.

[0315] Autonomous driving assistance software: A program that integrates sensor data and emotional data to assess the driver's condition and suggest autonomous driving modes.

[0316] System Operation

[0317] 1. Data Collection:

[0318] The device collects the driver's facial expressions and voice through the smart glasses' camera and microphone, and also acquires data such as speed, location, and distance to the vehicle ahead from sensors installed in the vehicle.

[0319] 2. Emotional state assessment:

[0320] The collected data is analyzed by an on-board emotion engine, which assesses the driver's fatigue, stress level and distraction.

[0321] 3. Driver Condition Assessment:

[0322] The device comprehensively evaluates the driver's emotional and driving data to analyze their fatigue and stress levels, and based on this evaluation, the driver's attention and driving skills are also taken into account.

[0323] 4. Proposal for autonomous driving mode:

[0324] If the driver's condition exceeds a certain threshold, the device will suggest transitioning to autonomous driving mode and notify the driver through audio and visual feedback.

[0325] 5. Safe driving support:

[0326] Based on sensor data and emotional data, the device provides safe driving support, for example by issuing a warning when the distance to the vehicle ahead is short or when the driver's stress level is high.

[0327] 6. Gathering feedback and improving the system:

[0328] The user provides feedback after completing a drive, and the emotion engine analyzes the user's emotional state. The server then uses this feedback to improve the system and apply it to the next drive.

[0329] Specific examples

[0330] During a long drive, the smart glasses suggest, "You seem tired. Would you like to switch to autonomous driving mode?" At this time, it is clear that the driver is not only tired, but also stressed from their facial expressions. Furthermore, when providing feedback after the drive that "your lane change was a bit extreme," the tone of voice and facial expressions at the time are analyzed to determine whether that was really how they felt, and the system will adjust accordingly for the next drive.

[0331] Prompt Sentence Examples

[0332] "You're driving a long distance. Your smart glasses might say: 'You're getting very close to the vehicle ahead. Be careful.' They might also suggest: 'You seem tired. Would you like to shift into self-driving mode?' If you provide feedback like 'You're changing lanes too aggressively,' the system will use this feedback and your emotional data to make adjustments the next time you drive."

[0333] In this way, by implementing the present invention, advanced automated driving assistance that takes into account the emotional state of the driver can be realized.

[0334] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0335] Step 1:

[0336] System initialization

[0337] The server initializes the system and initializes the driver's fatigue level, driving skill, sensor data, emotional data, and feedback. It uses the system's initial settings and past data as input, and generates an initial database as output. This process prepares the system for data collection and analysis.

[0338] Step 2:

[0339] Sensor and emotion data collection

[0340] The device collects data from sensors installed in the smart glasses and the vehicle. Specifically, the smart glasses' camera captures the driver's facial expressions, the microphone records audio, and the vehicle sensors collect driving data such as speed, position, and distance between vehicles. The device uses real-time data from the sensors as input and receives and stores all of this data in a consistent format as output.

[0341] Step 3:

[0342] Driver condition assessment

[0343] The device analyzes the collected sensor data and emotional data. Specifically, it uses an emotion engine to evaluate the driver's fatigue and stress levels based on their facial expressions and voice. The input is the data collected in the previous step, and the output is an evaluation of the driver's fatigue and emotional state. This evaluation allows the driver's current physical and mental state to be understood.

[0344] Step 4:

[0345] Proposal for autonomous driving mode

[0346] Based on the evaluation results, the device will suggest switching to autonomous driving mode. Specifically, if the evaluation exceeds a certain threshold, the device will notify the driver audibly and visually, saying, "You seem tired. Would you like to switch to autonomous driving mode?" The device uses the evaluation results of fatigue level and emotional state as input, and generates and displays a suggestion message as output.

[0347] Step 5:

[0348] Safe driving support

[0349] The device analyzes sensor data and emotional data in real time to provide safe driving support. Specifically, it issues a warning when the distance to the vehicle ahead is close or the driver is under high stress. It uses sensor data and emotional data as input and provides warning messages and advice in audio and visual formats as output.

[0350] Step 6:

[0351] Gathering feedback

[0352] The user provides feedback after completing the drive. The device collects this feedback data and analyzes the tone of voice and facial expressions at the time of providing the feedback using an emotion engine. The user's feedback and emotional data at that time are used as input, and the analysis results of the feedback are obtained as output.

[0353] Step 7:

[0354] System Improvements

[0355] The server analyzes the collected feedback and emotion data and adjusts the system parameters accordingly, specifically adjusting the system's response based on the feedback and applying it to the next drive. The server uses the feedback analysis results as input and obtains improved system settings as output.

[0356] This series of steps enables the driver's emotional state to be grasped in real time, allowing appropriate automated driving suggestions and safe driving support to be realized.

[0357] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0358] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0359] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0360] [Second embodiment]

[0361] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0362] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0363] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0364] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0365] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0366] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0367] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0368] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0369] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0370] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0371] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0372] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0373] This invention provides a specific method for evaluating driver fatigue and driving skills in an automated driving assistance system and proposing a transition to automated driving mode at an appropriate time. It also aims to enhance user trust and social acceptance by collecting user feedback and continuously improving the system.

[0374] Program processing

[0375] 1. System initialization

[0376] The server initializes the system. At the time of instantiation, the driver's current fatigue level, driving skill, collected sensor data, and past feedback are initialized.

[0377] 2. Sensor data collection

[0378] The device collects data from various sensors installed in the vehicle, including driving information such as the vehicle's speed, lane position, and distance to the vehicle ahead.

[0379] 3. Driver condition assessment

[0380] The device assesses the driver's level of fatigue, which is quantified based on certain criteria and indicates a decline in attention and reaction time while driving.

[0381] 4. Proposal for autonomous driving mode

[0382] The device will determine whether to switch to autonomous driving mode based on the collected data and the evaluation results, and will suggest switching to autonomous driving mode if the driver is highly fatigued or the driving conditions are difficult.

[0383] 5. Safe driving support

[0384] It analyzes sensor data and provides warnings and advice for safe driving as needed, for example, if the distance to the vehicle ahead is too close.

[0385] 6. Gathering Feedback

[0386] The user provides feedback about the driving experience, for example, by providing specific feedback such as whether the system is "too aggressive" or "too slow."

[0387] 7. System Improvements

[0388] The server improves the system based on the collected feedback. It analyzes the feedback, reevaluates the driver's driving skills based on the feedback, and adjusts the system's operating parameters.

[0389] Specific examples

[0390] Next, a specific example of the automated driving assistance system of the present invention will be shown. For example, if a driver has been driving for a long time, the system will evaluate the driver's fatigue level in real time. If the fatigue level exceeds a certain level, the device will suggest switching to automated driving mode. At this time, the driver will be notified by voice, "Do you want to switch the vehicle to automated driving mode?", ensuring a smooth transition to automated driving mode.

[0391] Additionally, if the distance to the vehicle ahead is significantly short while driving, the sensor will detect this information and the device will issue a warning saying, "You are getting too close to the vehicle ahead." This allows the driver to immediately recognize the situation and maintain safe driving.

[0392] Furthermore, suppose the user provides feedback after the drive, such as "My lane change was a bit extreme." The server analyzes this feedback and adjusts the system parameters for the next drive. In this way, the system is continuously improved based on the user's experience and opinions.

[0393] By combining these functions, the automated driving assistance system of the present invention can improve the safety and comfort of the driver.

[0394] The processing flow will be explained below.

[0395] Step 1:

[0396] The server initializes the system by creating an instance of UrbanCohesionDriveAssist, setting the driver's current fatigue level to 0, and the driving skill level to intermediate (5). It also initializes the sensor data and feedback lists as empty.

[0397] Step 2:

[0398] The terminal calls the collect_sensor_data method to collect data from various sensors installed in the vehicle. The collected data includes speed, lane position, and distance to the vehicle ahead. A random number is generated for each piece of data and recorded as sensor data.

[0399] Step 3:

[0400] The device calls the evaluate_driver_state method to evaluate the driver's fatigue level. The fatigue level increases by 1 and is adjusted downward according to the driver's driving skill. As a result, the fatigue level is set so that it will not fall below 0.

[0401] Step 4:

[0402] The device calls the evaluate_and_propose_autonomous_mode method to propose switching to autonomous driving mode if the driver's fatigue level exceeds a certain threshold (e.g., 10). If the proposal is accepted, the switch_to_autonomous_mode method is called to switch to autonomous driving mode.

[0403] Step 5:

[0404] The device calls the provide_safety_support method to provide safe driving support based on the collected sensor data. For example, if the distance to the vehicle ahead is less than 20 meters, the device warns the driver that the vehicle is "too close to the vehicle ahead."

[0405] Step 6:

[0406] The user provides feedback using the collect_feedback method. Feedback includes their opinion and evaluation of the driving experience, such as "the system is too aggressive" or "it's too slow."

[0407] Step 7:

[0408] The server calls the improve_system_based_on_feedback method to analyze the collected feedback and improve the system. It evaluates the feedback and adjusts the driver's driving skills and the system's operating parameters accordingly. The improved settings are then applied the next time the vehicle is driven.

[0409] Example 1

[0410] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0411] Conventional automated driving assistance systems have problems such as insufficient assessment of driver fatigue and driving skills, making it impossible to suggest switching to automated driving mode at the appropriate time. They also lack the ability to provide real-time safe driving support based on sensor data and improve the system based on user feedback. As a result, they have been unable to fully ensure driver safety and comfort.

[0412] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0413] In this invention, the server includes a means for assessing the driver's fatigue level, a means for collecting sensor data from various sensors and analyzing it in real time, a means for assessing the driver's driving skill, a means for determining whether or not switching to autonomous driving mode is necessary and making a recommendation, a means for providing warnings and advice for safe driving as necessary based on the collected sensor data, and a means for collecting feedback from users and improving the system based on that feedback. This makes it possible to accurately assess the driver's fatigue level and driving skill and to suggest switching to autonomous driving mode at an appropriate time. It also makes it possible to provide safe driving support in real time and improve the system based on that feedback.

[0414] The "means for assessing the driver's level of fatigue" refers to a device or system that analyzes the driver's eye opening and closing and facial movements to quantify the level of fatigue.

[0415] "Various sensors" refers to a group of sensors that collect driving information such as vehicle speed, lane position, and distance to the vehicle ahead.

[0416] "Real-time analysis means" refers to devices or software for processing and analyzing collected sensor data in real time.

[0417] "Means for evaluating a driver's driving skills" refers to a system that analyzes the vehicle's driving patterns and the driver's operations, and quantifies or evaluates the skills.

[0418] "Means for determining whether or not switching to autonomous driving mode is necessary and proposing this" refers to a device or system that determines whether or not switching to autonomous driving mode is necessary based on collected data and evaluation results and notifies the driver.

[0419] "Means for providing warnings and advice for safe driving" refers to devices or software that provide instructions or warnings to drivers for safe driving based on data analyzed in real time.

[0420] "Means for collecting user feedback" refers to an interface through which drivers can input their opinions and thoughts about their driving experience, and a system for collecting that data.

[0421] A "means for improving the system" is a device or software that analyzes the collected feedback and adjusts or refines the operating parameters of the system.

[0422] MODE FOR CARRYING OUT THE INVENTION

[0423] This invention provides a specific method for an automated driving assistance system that evaluates the driver's fatigue level and driving skill and suggests transitioning to automated driving mode at an appropriate time. It also aims to enhance driver safety and comfort by collecting user feedback and continuously improving the system.

[0424] System configuration

[0425] The main components of the automated driving assistance system are a server, a terminal, and a user interface. The role and operation of each component are explained below.

[0426] Server Roles

[0427] The server plays a central role in the entire system, specifically implementing the following functions:

[0428] System initialization:

[0429] The server initializes the system, including the driver's current fatigue level, driving skill, collected sensor data, and past feedback.

[0430] Data Processing:

[0431] The server analyzes the collected sensor data and evaluates the driver's level of fatigue and driving skills.

[0432] System Improvements:

[0433] The server analyzes user feedback and adjusts the system's operating parameters.

[0434] Device Role

[0435] The terminal collects and analyzes data in real time from various sensors installed in the vehicle. Specific functions are as follows:

[0436] Sensor data collection:

[0437] The device collects data from sensors installed in the vehicle, including the vehicle's speed, lane position, distance to the vehicle ahead, and a driver fatigue index based on facial recognition.

[0438] Driver Condition Rating:

[0439] The device evaluates the driver's level of fatigue and driving skills based on the data collected.

[0440] Autonomous driving mode suggestions:

[0441] The device determines whether it is necessary to switch to autonomous driving mode and notifies the driver.

[0442] Safe driving support:

[0443] The device analyzes the data in real time and issues warnings to the driver as needed.

[0444] User Interface

[0445] The user interface is primarily responsible for:

[0446] Providing feedback:

[0447] The system provides an interface that allows users to easily provide feedback, such as specific opinions like "too extreme" or "too slow."

[0448] Notification function:

[0449] When suggesting transition to autonomous driving mode, audio and visual notifications will be provided.

[0450] Hardware and software used

[0451] Hardware:

[0452] Various sensors (speed sensors, lane position sensors, distance sensors, face recognition cameras, etc.)

[0453] On-board computer (terminal)

[0454] server

[0455] software:

[0456] Data Analysis Software

[0457] Feedback Analysis Algorithm

[0458] Notification and warning interface program

[0459] Specific examples

[0460] Driver fatigue assessment and autonomous driving mode proposal

[0461] As a specific example, consider a driver who continues to drive for long periods of time. The device evaluates the driver's fatigue level in real time, and if it exceeds a certain level, a voice notification will be issued asking, "Do you want to switch the vehicle to autonomous driving mode?" At this point, the driver can choose to switch to autonomous driving mode via the voice notification.

[0462] Safe driving support function

[0463] If the distance to the vehicle ahead becomes too short while driving, the device will issue a warning saying, "You are getting too close to the vehicle ahead," prompting the driver to take immediate action to increase their distance.

[0464] Gathering feedback and improving the system

[0465] After completing the drive, the user provides feedback such as "My lane change was a bit extreme." The server analyzes this feedback and adjusts the system's operating parameters for the next drive, improving the user experience.

[0466] Prompt Sentence Examples

[0467] Examples of prompts to enter into a generative AI model include:

[0468] Prompt: Describe the specific steps for fatigue assessment and feedback processing in an automated driving assistance system.

[0469] Objective: Learn more about how to assess driver fatigue in real time and suggest automated driving modes at the right time.

[0470] With such settings, the automatic driving assistance system of the present invention can improve the safety and comfort of the driver.

[0471] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0472] The flow of this system's program processing

[0473] Step 1:

[0474] System initialization

[0475] The server initializes the system. As input, it obtains the driver's current fatigue level, driving skill, collected sensor data, and past feedback data from the database. Data processing generates a driver profile and initializes driving tendencies. As output, it obtains an initialized system and an initial driver profile. Specifically, the server reads driving history data from the database and prepares for a new driving session.

[0476] Step 2:

[0477] Sensor data collection

[0478] The terminal collects data in real time from various sensors attached to the vehicle. The input includes data obtained from speed sensors, lane position sensors, distance sensors, face recognition cameras, etc. Data processing involves storing each sensor data in a buffer and converting it into the required data format. As an output, real-time data from the sensors is stored in the buffer. Specifically, the terminal collects the latest driving information every second and sends that information to the server in real time.

[0479] Step 3:

[0480] Driver condition assessment

[0481] The driver's condition is evaluated based on the sensor data collected by the device. The input includes real-time sensor data stored in a buffer. For data calculation, facial recognition technology is used to analyze the opening and closing of the driver's eyes and facial movements to quantify the level of fatigue. Driving patterns are also analyzed to evaluate driving skills. The output is a driver fatigue score and driving skill score. Specifically, an AI model is used to evaluate real-time data, and the driver's fatigue level is displayed on the screen.

[0482] Step 4:

[0483] Proposal for autonomous driving mode

[0484] The device will suggest switching to autonomous driving mode based on the evaluation results. Inputs include the driver's fatigue score and driving skill score. Data calculations compare the evaluation score with preset reference values ​​to determine whether it is necessary. As an output, a notification suggesting switching to autonomous driving mode is generated for the driver. Specifically, the device will issue a voice notification asking, "Do you want to switch the vehicle to autonomous driving mode?"

[0485] Step 5:

[0486] Safe driving support

[0487] The device analyzes sensor data in real time and provides warnings and advice for safe driving as needed. The input includes continuously collected sensor data. Data processing monitors the distance to the vehicle ahead and lane departure, generating a warning if certain criteria are exceeded. The output generates audio and visual warnings for the driver. Specifically, the device displays a warning message such as "You are getting too close to the vehicle ahead."

[0488] Step 6:

[0489] Gathering feedback

[0490] The user provides feedback on the driving experience. The input includes feedback data provided through the user interface. Data processing classifies the feedback and stores it as analysis data for system improvement. The output is the collected feedback stored on the server. Specifically, the user enters "The lane change was a bit extreme" into the smartphone app.

[0491] Step 7:

[0492] System Improvements

[0493] The server improves the system based on the collected feedback. Inputs include user feedback and past driving data. Data calculations involve analyzing the feedback, reassessing the driver's driving skills, and adjusting the system's operating parameters. The output is improved system settings. Specifically, the server analyzes the feedback and configures the system to operate with the new parameter set the next time the vehicle is driven.

[0494] In this way, automated driving assistance systems can increase driver safety and comfort.

[0495] (Application example 1)

[0496] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0497] Current autonomous driving technology does not adequately assess driver fatigue and driving skills, making it difficult to fully guarantee driver safety and comfort. Furthermore, there is a lack of mechanisms for detecting driver fatigue and loss of concentration in real time during long periods of driving and prompting the vehicle to transition to autonomous driving mode at the appropriate time. Furthermore, existing systems lack sufficient processes for collecting user feedback and improving the system, resulting in low reliability and social acceptance. Therefore, there is a need for an autonomous driving assistance system that can be continuously improved while achieving both safety and comfort.

[0498] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0499] In this invention, the server includes a means for assessing the driver's fatigue level, a means for collecting and analyzing sensor data, a means for assessing the driver's driving skill, a means for proposing transition to autonomous driving mode, a means for collecting user feedback and improving the system, and a means for providing visual and audio warnings to support the driver's safe driving. This makes it possible to evaluate the driver's condition in real time and suggest transitioning to autonomous driving mode at an appropriate time. It also supports safe driving by notifying the driver of dangerous situations while driving with visual and audio warnings. Furthermore, by collecting user feedback and continuously improving the system based on that feedback, it is possible to increase reliability and social acceptance.

[0500] The "means for assessing driver fatigue" refers to a method and device for quantifying and assessing the driver's fatigue level in real time based on facial recognition, eye movements, etc.

[0501] "Means for collecting and analyzing sensor data" refers to the technology and equipment that collects data such as speed, lane position, and distance to the vehicle ahead from various sensors attached to the vehicle and analyzes it.

[0502] The "means for assessing the driver's driving skills" refers to algorithms and devices for assessing the driving skills based on collected sensor data and the driver's driving history.

[0503] "Means for proposing transition to autonomous driving mode" refers to technologies and devices that notify and suggest transition to autonomous driving mode to the driver at an appropriate time based on the evaluation results.

[0504] "Means for collecting user feedback and improving the system" refers to technologies and devices that collect feedback provided by drivers based on their driving experience, analyze it, adjust system parameters, and continuously improve the system.

[0505] "Means for providing visual and audio warnings to support safe driving" refers to technologies and devices that provide visual and audio warnings to drivers based on collected sensor data and evaluation results, and encourage appropriate safe driving behavior.

[0506] The following configurations are included as embodiments of the present invention.

[0507] The server initializes the system and provides a means to evaluate the driver's fatigue level and driving skills. It uses data from various sensors attached to the vehicle (speed, lane position, distance to the vehicle ahead, etc.) to collect and analyze sensor data. The analyzed data is sent to the server, which evaluates the driver's condition in real time.

[0508] Based on the evaluation results, the device provides a means to suggest switching to autonomous driving mode. This suggestion is made using visual and audio output devices such as smart glasses. For example, if the driver's fatigue level exceeds a certain level, the device will notify the driver via audio and visual means, saying, "Your fatigue level is high. Would you like to suggest switching to autonomous driving mode?" If the driver answers "Yes" via audio, the device will smoothly switch to autonomous driving mode.

[0509] In addition, the device provides visual and audio warnings to help drivers drive safely. For example, if the distance to the vehicle ahead is significantly short, the device will warn the driver that they are driving too close to the vehicle ahead, allowing them to immediately recognize the situation and take appropriate action.

[0510] After completing a drive, users can provide feedback to the system. The feedback is sent to the server and used to improve the system. For example, specific opinions such as "Your lane changes were a bit extreme" can be reflected and the system's operating parameters will be adjusted for the next drive. This allows the system to continuously improve, increasing its reliability and social acceptance.

[0511] Description of the hardware and software used

[0512] Hardware: Smart glasses (e.g., Microsoft HoloLens, Google Glass), various in-vehicle sensors

[0513] Software: Driver state assessment algorithm, smart glasses SDK (e.g. GoggleSDK)

[0514] Data processing and calculation

[0515] The data is processed as follows:

[0516] 1. Sensor data collection:

[0517] Data is collected from various sensors installed in the vehicle.

[0518] 2. Driver Condition Assessment:

[0519] Based on the collected sensor data, algorithms are applied to assess fatigue levels and driving skills.

[0520] 3. Autonomous driving mode suggestions and warnings:

[0521] -Based on the evaluation results, the device will suggest transitioning to autonomous driving mode and provide visual and audio warnings.

[0522] Examples of concrete examples and prompts

[0523] Specific examples

[0524] When a driver is driving for a long period of time, the smart glasses monitor the driver's eye movements and facial expressions in real time to detect an increase in fatigue. As a result, a notification will appear on the smart glasses' HUD saying, "Fatigue level is high. Would you like to suggest switching to autonomous driving mode?" If the driver answers "Yes" aloud, the smart glasses will automatically switch to autonomous driving mode.

[0525] Prompt Sentence Examples

[0526] Consider a scenario in which smart glasses monitor the eye movements and facial expressions of a driver in real time while driving for long periods of time to detect increased fatigue, and explain the appropriate message display and voice response procedures.

[0527] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0528] Step 1:

[0529] System initialization

[0530] The server initializes the system. The inputs are the driver's current fatigue level, driving skill, collected sensor data, and past feedback. Based on this, all initial parameters are set. The output is the initialized system state. Specifically, the server resets all parameters and performs the initial settings.

[0531] Step 2:

[0532] Sensor data collection

[0533] The terminal collects data from various sensors attached to the vehicle. The input is real-time data such as the vehicle's speed, lane position, and distance to the vehicle ahead. This data is collected by the sensors and sent to the terminal. The output is a set of collected sensor data. Specifically, the terminal collects sensor data at regular intervals and stores it for analysis.

[0534] Step 3:

[0535] Driver condition assessment

[0536] The device evaluates the driver's fatigue level and driving skills based on the sensor data collected. The sensor data collected in step 2 is used as input. A data analysis algorithm quantifies the fatigue level and evaluates the driving skills. The output is the quantified fatigue level and driving skill evaluation results. Specifically, the device executes the data analysis algorithm and calculates the evaluation results.

[0537] Step 4:

[0538] Proposal for autonomous driving mode

[0539] Based on the evaluation results, the device will suggest switching to autonomous driving mode. The inputs are the fatigue level and driving skill evaluation results obtained in step 3. When the fatigue level exceeds a certain value, the device will output a suggestion notification. Specifically, it will display a message on the smart glasses' HUD and notify the driver by voice, "Your fatigue level is high. Would you like to suggest switching to autonomous driving mode?"

[0540] Step 5:

[0541] Safe driving support

[0542] The device analyzes sensor data in real time and provides warnings and advice for safe driving. The input is sensor data and the driver's state evaluation results. The output is visual and audio warning and advice messages. Specifically, the device monitors the distance to the vehicle ahead and warns the driver that "you are too close to the vehicle ahead" if the distance is too close.

[0543] Step 6:

[0544] Gathering feedback

[0545] The user provides feedback about the driving experience. The input is the specific feedback provided by the driver, including their opinions and thoughts about the system's behavior. The output is the collected feedback data. The specific operation is that the terminal receives the feedback from the user and sends it to the server.

[0546] Step 7:

[0547] System Improvements

[0548] The server adjusts and improves the system's operating parameters based on the collected feedback. The input is the feedback data collected in step 6. The feedback is analyzed and necessary adjustments are made. The output is the improved system's operating parameters. Specifically, the server runs the feedback analysis algorithm and updates the system settings.

[0549] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0550] This invention combines an emotion engine with an automated driving assistance system to improve the effectiveness of driver fatigue assessment and feedback collection, and to improve the accuracy of suggestions for switching to automated driving mode. The emotion engine recognizes emotions from the user's tone of voice, facial expressions, body movements, etc., and reflects them in the system.

[0551] Program processing

[0552] 1. System initialization

[0553] The server initializes the system. When creating an instance of UrbanCohesionDriveAssist, it initializes the driver's fatigue level, driving skill, sensor data, emotional data, and feedback.

[0554] 2. Collecting sensor data and emotion data

[0555] The device calls the collect_sensor_data method to collect data from the vehicle's sensors and the emotion engine that recognizes the user's emotions. The collected data includes speed, lane position, distance to the vehicle ahead, and the user's emotional state (e.g., stress level, distraction level, etc.).

[0556] 3. Driver condition assessment

[0557] The device calls the evaluate_driver_state method to comprehensively evaluate the driver's fatigue level based on emotional data and driving data, taking into account, for example, whether the driver is not only tired but also in a high-stress state.

[0558] 4. Proposal for autonomous driving mode

[0559] The device calls the evaluate_and_propose_autonomous_mode method and suggests switching to autonomous driving mode if the driver's state or emotional data exceeds a certain threshold. Specifically, the device encourages switching to autonomous driving mode even if the driver is feeling highly stressed based on emotional data.

[0560] 5. Safe driving support

[0561] The device analyzes sensor data and emotional data to provide safe driving support. For example, it not only issues a warning when the distance to the vehicle ahead is short, but also considers the causes of driver distraction based on emotional data and provides warnings and advice based on that.

[0562] 6. Gathering Feedback

[0563] When a user provides feedback using the collect_feedback method, we also collect emotional data, such as tone of voice and facial expressions, to assess the user's true perception of the quality of the experience.

[0564] 7. System Improvements

[0565] The server calls the improve_system_based_on_feedback method to analyze the feedback and emotional data. Based on this, the system can be improved. For example, if the user perceives the system as "extreme," and the emotional data indicates a stress response, the server can adjust the driver's skill level or the system's response speed.

[0566] Specific examples

[0567] Next, a specific example of the present invention will be shown. For example, when a driver has been driving for a long time, the system evaluates the driver's fatigue level in real time. This evaluation includes emotional data such as the driver's pale complexion or nervous expression. If the fatigue level exceeds a certain threshold, the device will issue a voice notification saying, "You seem tired. Would you like to switch to autonomous driving mode?" and suggest a smooth transition to autonomous driving mode.

[0568] Additionally, if the distance to the vehicle ahead is significantly short while driving, the sensor data will detect this information, and the emotion engine will sense anxiety from the driver's facial expression. Based on this information, the device will issue a warning to the driver saying, "You are getting very close to the vehicle ahead. Please be careful."

[0569] Furthermore, when a user provides feedback after completing a drive, such as "My lane change was a bit extreme," the emotion engine analyzes the user's tone of voice and facial expression when providing the feedback. From these results, it is determined that the user truly felt the lane change was extreme, and the server adjusts the system parameters for the next drive. For example, it may take measures such as reducing the speed when changing lanes.

[0570] In this way, by using an autonomous driving assistance system that combines an emotion engine, it becomes possible to provide advanced assistance that takes into account the driver's state and emotions, thereby improving safety and comfort.

[0571] The processing flow will be explained below.

[0572] Step 1:

[0573] The server initializes the system. When creating an instance of UrbanCohesionDriveAssist, it sets the driver's fatigue level to 0 and their driving skill to intermediate level (5). It also initializes the sensor data, emotion data, and feedback lists as empty.

[0574] Step 2:

[0575] The terminal calls the collect_sensor_data method to collect data from various sensors installed in the vehicle. The collected data includes speed, lane position, and distance to the vehicle ahead. This data is randomly generated and added to the sensor data list.

[0576] Step 3:

[0577] The device uses an emotion engine to collect emotional data from the driver's facial expressions, tone of voice, and body movements, for example, assessing whether the driver is tired or stressed and adding this to the emotional data list.

[0578] Step 4:

[0579] The device calls the evaluate_driver_state method to evaluate the driver's fatigue level based on the collected sensor data and emotional data. The fatigue level is calculated by an integrated analysis of the sensor data and emotional data. For example, if the driver's attention is distracted while driving and their facial expression indicates fatigue, the fatigue level is set high.

[0580] Step 5:

[0581] The device calls the evaluate_and_propose_autonomous_mode method to propose switching to autonomous driving mode if the driver's fatigue level or emotion data exceeds a certain threshold. If the proposal is accepted, the switch_to_autonomous_mode method is called to switch to autonomous driving mode.

[0582] Step 6:

[0583] The device calls the provide_safety_support method, analyzes sensor data and emotion data, and provides safe driving support. For example, if the distance to the vehicle ahead is too close and the emotion data indicates that the driver is nervous, the device will warn, "You are getting too close to the vehicle ahead. Please be careful."

[0584] Step 7:

[0585] Users provide feedback using the collect_feedback method. The feedback includes specific opinions about the driving experience, as well as emotional data at the time of the feedback. For example, the user's opinion that "the system's behavior was too aggressive" is recorded, along with the user's tone of voice and facial expression.

[0586] Step 8:

[0587] The server calls the improve_system_based_on_feedback method to analyze the collected feedback and emotion data. Based on the analysis results, the driver's driving skills are reevaluated and the system's operating parameters are adjusted. For example, if the feedback indicates that the driver's driving skills are "too aggressive," the system may adjust the lane change speed accordingly.

[0588] Example 2

[0589] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0590] While conventional automated driving assistance systems can partially take into account the driver's level of fatigue and driving skill, it is difficult to provide safe driving assistance that fully reflects the driver's emotional state or to improve the system based on feedback. Furthermore, emotional data is not sufficiently taken into account when maintaining a safe distance from the vehicle ahead or analyzing surrounding traffic conditions, leaving issues with safety and comfort. Therefore, there is a need for a system that can evaluate the driver's emotions in real time and provide appropriate assistance and suggestions.

[0591] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for evaluating the driver's fatigue level, a means for collecting and analyzing sensor data and emotional data, a means for evaluating the driver's driving skill and emotional state, a means for suggesting transition to an autonomous driving mode based on the driver's state and emotional data, and a means for collecting feedback from the user and analyzing the emotional data at the time of feedback to improve the system. This enables advanced support and suggestions that take the driver's emotional state into consideration, thereby improving safety and comfort.

[0592] "Fatigue level" refers to the degree of physical and mental fatigue a driver experiences while driving.

[0593] "Sensor data" refers to information collected by various sensors attached to a vehicle, and specifically includes data related to driving, such as speed, lane position, and distance to the vehicle ahead.

[0594] "Emotional data" refers to information about a driver's emotional state analyzed from their tone of voice, facial expressions, body movements, etc., and specifically includes stress levels and distractibility.

[0595] "Driving skill" is an evaluation of the driver's level of proficiency and skill in driving.

[0596] "Autonomous driving mode" is a mode in which the vehicle can operate autonomously without driver operation.

[0597] "Feedback" refers to opinions and evaluations provided by users to the system, and includes information that can be used to improve the system.

[0598] An "autonomous driving assistance system" is a system that assists drivers in driving and improves safety and comfort, and includes fatigue assessment, collection and analysis of sensor data and emotional data, driving skill assessment, autonomous driving mode proposals, feedback collection, and system improvement.

[0599] A "warning" is a notification or alert that notifies the driver of an impending hazard.

[0600] "Safe driving support" refers to advice and assistance provided to drivers to help them drive safely, and specifically includes maintaining a safe distance from the vehicle ahead and warning against dangerous driving.

[0601] MODE FOR CARRYING OUT THE INVENTION

[0602] This invention combines an emotion engine with an automated driving assistance system to improve the effectiveness of driver fatigue assessment and feedback collection, and to improve the accuracy of suggestions for switching to automated driving mode. The emotion engine recognizes emotions from the user's tone of voice, facial expressions, body movements, etc., and reflects them in the system.

[0603] System initialization

[0604] The server creates an instance of UrbanCohesionDriveAssist when the system is initialized. This instance is initialized with the driver's fatigue level, driving skill, sensor data, emotional data, and feedback. During initialization, the necessary data is synchronized using an existing database or cloud server.

[0605] Data collection

[0606] The device calls the collect_sensor_data method to collect data such as speed, lane position, and distance to the vehicle ahead from sensors installed in the vehicle. At the same time, the emotion engine analyzes the user's tone of voice, facial expressions, and body movements to collect emotional data such as stress level and distraction level. This provides both sensor data and emotional data to the system in real time.

[0607] Driver condition assessment

[0608] The device uses the evaluate_driver_state method to evaluate the driver's fatigue level based on the collected sensor data and emotional data. For example, a pale complexion and distractedness after a long drive could be used as evaluation points. The device also evaluates the driver's stress level based on the driver's tone of voice and facial expression, and combines this with the fatigue level evaluation to provide a comprehensive state evaluation.

[0609] Proposal for autonomous driving mode

[0610] The device calls the evaluate_and_propose_autonomous_mode method and proposes switching to autonomous driving mode if the driver's condition or emotional data exceeds a certain threshold. For example, if the user's facial expression indicates extreme fatigue, the device will issue a voice notification asking, "Do you want to switch to autonomous driving mode?"

[0611] Safe driving support

[0612] If sensor data indicates that the distance to the vehicle ahead is dangerously close, the device will issue a warning saying, "You are approaching the vehicle in front. Please be careful." If emotional data detects that the driver is anxious or distracted, the device will provide specific advice based on that emotional data, such as, "Take a deep breath and relax."

[0613] Gathering feedback and improving the system

[0614] When a user provides feedback after driving using the collect_feedback method, the emotion engine analyzes the tone of voice and facial expressions to evaluate the quality of the feedback. Based on this result, the server uses the improve_system_based_on_feedback method to improve the system. For example, taking into account past feedback and emotion data, the server can improve safety by reducing the lane-changing speed in the next driving session.

[0615] Specific examples

[0616] For example, if a driver has been driving for a long time, the server initializes the system and the device calls the appropriate method. The device uses the collect_sensor_data method to collect the vehicle's speed and distance from the vehicle ahead, and analyzes the driver's facial expression and tone of voice using the emotion engine. The emotion data and driving data are combined to evaluate the driver's condition. If the evaluation results indicate that the driver is fatigued and not paying attention, the device will suggest, "Do you want to switch to autonomous driving mode?" If the driver is dangerously close to the vehicle ahead, the device will warn, "You are very close to the vehicle ahead. Please be careful." After completing the drive, if the user provides feedback such as "My lane change was a bit aggressive," the emotion engine analyzes the user's tone of voice and facial expression to identify the specific reason why they thought the lane change was aggressive. The server uses this data to adjust the system parameters and reduce the lane change speed on the next drive.

[0617] An example of a prompt for a generative AI model is:

[0618] Please explain in natural language how the system detects the user's fatigue after a long drive and what suggestions it makes in response.

[0619] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0620] Step 1: Initialize the system

[0621] The server creates an instance of UrbanCohesionDriveAssist, which is initialized with the driver's fatigue level, driving skill, sensor data, emotional data, and feedback. The input is the initial setting data stored in an existing database or cloud server, and the output is the initialized system instance.

[0622] Step 2: Collecting sensor and emotion data

[0623] The device uses sensors installed in the vehicle and an emotion engine to collect driving-related sensor data and driver emotion data. Specifically, it calls the collect_sensor_data method to collect data such as speed, lane position, and distance to the vehicle ahead. It also analyzes stress levels and distraction levels from tone of voice, facial expressions, and body movements through the emotion engine. The inputs are the vehicle's sensor information and raw data from the emotion engine, and the output is analyzed sensor data and emotion data.

[0624] Step 3: Driver condition assessment

[0625] The device analyzes sensor data and emotion data to evaluate the driver's fatigue and stress levels. Specifically, it calls the evaluate_driver_state method to evaluate the facial expression and level of distraction of a driver who has been driving for a long time. The input is the collected sensor data and emotion data, and the output is the evaluation result, which includes the driver's fatigue and stress levels.

[0626] Step 4: Proposing an autonomous driving mode

[0627] The device proposes transitioning to autonomous driving mode based on the results of the driver's state evaluation. Specifically, it calls the evaluate_and_propose_autonomous_mode method, and if the driver's fatigue or stress level exceeds a certain threshold, it issues a voice notification suggesting, "Do you want to switch to autonomous driving mode?" The input is the driver's state evaluation result, and the output is a notification proposing transition to autonomous driving mode.

[0628] Step 5: Safe driving support

[0629] The device provides safe driving support to the driver based on the collected sensor data and emotional data. Specifically, it evaluates the distance to the vehicle ahead and issues a warning such as, "You are very close to the vehicle ahead. Please be careful." It also detects anxiety or distraction in the driver from emotional data and provides advice such as, "Take a deep breath and relax." The input is sensor data and emotional data, and the output is specific warnings and advice.

[0630] Step 6: Gather feedback

[0631] The user provides feedback after driving. The device uses the collect_feedback method to analyze the feedback along with the tone of voice and facial expression. The emotion engine is used to collect emotion data when providing the feedback and save the analysis results. The input is the user's feedback and emotion data, and the output is the analyzed feedback data.

[0632] Step 7: Improve your system

[0633] The server analyzes the collected feedback and emotion data and makes improvements to the system. Specifically, it calls the improve_system_based_on_feedback method to adjust system parameters based on the analyzed data. For example, it takes into account past feedback and emotion data and reduces the lane-changing speed in the next driving session. The input is the analyzed feedback data, and the output is the adjusted system parameters.

[0634] (Application example 2)

[0635] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0636] While conventional automated driving assistance systems can evaluate the driver's level of fatigue and driving skill, they do not adequately suggest switching to automated driving or provide safe driving support that accurately reflects the driver's emotional state. This means that the system is unable to respond quickly to sudden changes in the driver's physical condition or emotional fluctuations, which could compromise the safety and comfort of automated driving.

[0637] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for evaluating the driver's fatigue level, a means for collecting and analyzing sensor data, voice tone, and facial expressions, a means for evaluating the driver's driving skill and emotional state, a means for proposing transition to autonomous driving mode, and a means for collecting feedback from the user and improving the system. This makes it possible to evaluate the driver's state of fatigue, stress, and distraction in real time and propose transition to autonomous driving mode at an appropriate time.

[0638] "Driver fatigue level" refers to the degree of physical or mental fatigue a driver feels while driving.

[0639] "Sensor data" refers to information collected from sensors attached to vehicles and related equipment, and includes physical data such as speed, position, and distance.

[0640] "Voice tone" refers to characteristics of the driver's voice, such as pitch, strength, and intonation, and is information used to estimate the driver's emotional state.

[0641] "Facial expression" refers to the movement and changes of the driver's facial muscles, and is information used to analyze the driver's emotional state.

[0642] "Driving skills" refers to a driver's ability to operate a vehicle safely and efficiently, and its assessment includes the driver's experience and skill.

[0643] "Emotional state" refers to the driver's psychological state, including states such as stress, distraction, and relaxation.

[0644] "Autonomous driving mode" refers to a mode in which the vehicle drives automatically without driver intervention, aiming to improve safety and convenience.

[0645] "Feedback" refers to opinions and impressions from users, providing information about the performance and usability of the system.

[0646] "System Improvement" refers to actions taken to improve the performance and functionality of a system based on collected feedback and data.

[0647] This invention combines an emotion engine with an automated driving assistance system to evaluate the driver's fatigue level, driving skill, and emotional state, and proposes transitioning to automated driving mode at the appropriate time. The system uses smart glasses and other sensor devices to grasp the driver's emotional state in real time, and collects and analyzes feedback.

[0648] System configuration

[0649] Hardware:

[0650] Smart glasses: Equipped with a camera and microphone to collect the driver's facial expressions and voice.

[0651] Sensors installed in the vehicle collect driving data such as speed, lane position, and distance to the vehicle ahead.

[0652] software:

[0653] Emotion Engine: A model for analyzing a driver's emotional state from their facial expressions, vocal tone, and body movements.

[0654] Autonomous driving assistance software: A program that integrates sensor data and emotional data to assess the driver's condition and suggest autonomous driving modes.

[0655] System Operation

[0656] 1. Data Collection:

[0657] The device collects the driver's facial expressions and voice through the smart glasses' camera and microphone, and also acquires data such as speed, location, and distance to the vehicle ahead from sensors installed in the vehicle.

[0658] 2. Emotional state assessment:

[0659] The collected data is analyzed by an on-board emotion engine, which assesses the driver's fatigue, stress level and distraction.

[0660] 3. Driver Condition Assessment:

[0661] The device comprehensively evaluates the driver's emotional and driving data to analyze their fatigue and stress levels, and based on this evaluation, the driver's attention and driving skills are also taken into account.

[0662] 4. Proposal for autonomous driving mode:

[0663] If the driver's condition exceeds a certain threshold, the device will suggest transitioning to autonomous driving mode and notify the driver through audio and visual feedback.

[0664] 5. Safe driving support:

[0665] Based on sensor data and emotional data, the device provides safe driving support, for example by issuing a warning when the distance to the vehicle ahead is short or when the driver's stress level is high.

[0666] 6. Gathering feedback and improving the system:

[0667] The user provides feedback after completing a drive, and the emotion engine analyzes the user's emotional state. The server then uses this feedback to improve the system and apply it to the next drive.

[0668] Specific examples

[0669] During a long drive, the smart glasses suggest, "You seem tired. Would you like to switch to autonomous driving mode?" At this time, it is clear that the driver is not only tired, but also stressed from their facial expressions. Furthermore, when providing feedback after the drive that "your lane change was a bit extreme," the tone of voice and facial expressions at the time are analyzed to determine whether that was really how they felt, and the system will adjust accordingly for the next drive.

[0670] Prompt Sentence Examples

[0671] "You're driving a long distance. Your smart glasses might say: 'You're getting very close to the vehicle ahead. Be careful.' They might also suggest: 'You seem tired. Would you like to shift into self-driving mode?' If you provide feedback like 'You're changing lanes too aggressively,' the system will use this feedback and your emotional data to make adjustments the next time you drive."

[0672] In this way, by implementing the present invention, advanced automated driving assistance that takes into account the emotional state of the driver can be realized.

[0673] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0674] Step 1:

[0675] System initialization

[0676] The server initializes the system and initializes the driver's fatigue level, driving skill, sensor data, emotional data, and feedback. It uses the system's initial settings and past data as input, and generates an initial database as output. This process prepares the system for data collection and analysis.

[0677] Step 2:

[0678] Sensor and emotion data collection

[0679] The device collects data from sensors installed in the smart glasses and the vehicle. Specifically, the smart glasses' camera captures the driver's facial expressions, the microphone records audio, and the vehicle sensors collect driving data such as speed, position, and distance between vehicles. The device uses real-time data from the sensors as input and receives and stores all of this data in a consistent format as output.

[0680] Step 3:

[0681] Driver condition assessment

[0682] The device analyzes the collected sensor data and emotional data. Specifically, it uses an emotion engine to evaluate the driver's fatigue and stress levels based on their facial expressions and voice. The input is the data collected in the previous step, and the output is an evaluation of the driver's fatigue and emotional state. This evaluation allows the driver's current physical and mental state to be understood.

[0683] Step 4:

[0684] Proposal for autonomous driving mode

[0685] Based on the evaluation results, the device will suggest switching to autonomous driving mode. Specifically, if the evaluation exceeds a certain threshold, the device will notify the driver audibly and visually, saying, "You seem tired. Would you like to switch to autonomous driving mode?" The device uses the evaluation results of fatigue level and emotional state as input, and generates and displays a suggestion message as output.

[0686] Step 5:

[0687] Safe driving support

[0688] The device analyzes sensor data and emotional data in real time to provide safe driving support. Specifically, it issues a warning when the distance to the vehicle ahead is close or the driver is under high stress. It uses sensor data and emotional data as input and provides warning messages and advice in audio and visual formats as output.

[0689] Step 6:

[0690] Gathering feedback

[0691] The user provides feedback after completing the drive. The device collects this feedback data and analyzes the tone of voice and facial expressions at the time of providing the feedback using an emotion engine. The user's feedback and emotional data at that time are used as input, and the analysis results of the feedback are obtained as output.

[0692] Step 7:

[0693] System Improvements

[0694] The server analyzes the collected feedback and emotion data and adjusts the system parameters accordingly, specifically adjusting the system's response based on the feedback and applying it to the next drive. The server uses the feedback analysis results as input and obtains improved system settings as output.

[0695] This series of steps enables the driver's emotional state to be grasped in real time, allowing appropriate automated driving suggestions and safe driving support to be realized.

[0696] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0697] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0698] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0699] [Third embodiment]

[0700] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0701] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0702] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0703] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0704] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0705] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0706] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0707] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0708] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0709] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0710] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0711] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0712] This invention provides a specific method for evaluating driver fatigue and driving skills in an automated driving assistance system and proposing a transition to automated driving mode at an appropriate time. It also aims to enhance user trust and social acceptance by collecting user feedback and continuously improving the system.

[0713] Program processing

[0714] 1. System initialization

[0715] The server initializes the system. At the time of instantiation, the driver's current fatigue level, driving skill, collected sensor data, and past feedback are initialized.

[0716] 2. Sensor data collection

[0717] The device collects data from various sensors installed in the vehicle, including driving information such as the vehicle's speed, lane position, and distance to the vehicle ahead.

[0718] 3. Driver condition assessment

[0719] The device assesses the driver's level of fatigue, which is quantified based on certain criteria and indicates a decline in attention and reaction time while driving.

[0720] 4. Proposal for autonomous driving mode

[0721] The device will determine whether to switch to autonomous driving mode based on the collected data and the evaluation results, and will suggest switching to autonomous driving mode if the driver is highly fatigued or the driving conditions are difficult.

[0722] 5. Safe driving support

[0723] It analyzes sensor data and provides warnings and advice for safe driving as needed, for example, if the distance to the vehicle ahead is too close.

[0724] 6. Gathering Feedback

[0725] The user provides feedback about the driving experience, for example, by providing specific feedback such as whether the system is "too aggressive" or "too slow."

[0726] 7. System Improvements

[0727] The server improves the system based on the collected feedback. It analyzes the feedback, reevaluates the driver's driving skills based on the feedback, and adjusts the system's operating parameters.

[0728] Specific examples

[0729] Next, a specific example of the automated driving assistance system of the present invention will be shown. For example, if a driver has been driving for a long time, the system will evaluate the driver's fatigue level in real time. If the fatigue level exceeds a certain level, the device will suggest switching to automated driving mode. At this time, the driver will be notified by voice, "Do you want to switch the vehicle to automated driving mode?", ensuring a smooth transition to automated driving mode.

[0730] Additionally, if the distance to the vehicle ahead is significantly short while driving, the sensor will detect this information and the device will issue a warning saying, "You are getting too close to the vehicle ahead." This allows the driver to immediately recognize the situation and maintain safe driving.

[0731] Furthermore, suppose the user provides feedback after the drive, such as "My lane change was a bit extreme." The server analyzes this feedback and adjusts the system parameters for the next drive. In this way, the system is continuously improved based on the user's experience and opinions.

[0732] By combining these functions, the automated driving assistance system of the present invention can improve the safety and comfort of the driver.

[0733] The processing flow will be explained below.

[0734] Step 1:

[0735] The server initializes the system by creating an instance of UrbanCohesionDriveAssist, setting the driver's current fatigue level to 0, and the driving skill level to intermediate (5). It also initializes the sensor data and feedback lists as empty.

[0736] Step 2:

[0737] The terminal calls the collect_sensor_data method to collect data from various sensors installed in the vehicle. The collected data includes speed, lane position, and distance to the vehicle ahead. A random number is generated for each piece of data and recorded as sensor data.

[0738] Step 3:

[0739] The device calls the evaluate_driver_state method to evaluate the driver's fatigue level. The fatigue level increases by 1 and is adjusted downward according to the driver's driving skill. As a result, the fatigue level is set so that it will not fall below 0.

[0740] Step 4:

[0741] The device calls the evaluate_and_propose_autonomous_mode method to propose switching to autonomous driving mode if the driver's fatigue level exceeds a certain threshold (e.g., 10). If the proposal is accepted, the switch_to_autonomous_mode method is called to switch to autonomous driving mode.

[0742] Step 5:

[0743] The device calls the provide_safety_support method to provide safe driving support based on the collected sensor data. For example, if the distance to the vehicle ahead is less than 20 meters, the device warns the driver that the vehicle is "too close to the vehicle ahead."

[0744] Step 6:

[0745] The user provides feedback using the collect_feedback method. Feedback includes their opinion and evaluation of the driving experience, such as "the system is too aggressive" or "it's too slow."

[0746] Step 7:

[0747] The server calls the improve_system_based_on_feedback method to analyze the collected feedback and improve the system. It evaluates the feedback and adjusts the driver's driving skills and the system's operating parameters accordingly. The improved settings are then applied the next time the vehicle is driven.

[0748] Example 1

[0749] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0750] Conventional automated driving assistance systems have problems such as insufficient assessment of driver fatigue and driving skills, making it impossible to suggest switching to automated driving mode at the appropriate time. They also lack the ability to provide real-time safe driving support based on sensor data and improve the system based on user feedback. As a result, they have been unable to fully ensure driver safety and comfort.

[0751] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0752] In this invention, the server includes a means for assessing the driver's fatigue level, a means for collecting sensor data from various sensors and analyzing it in real time, a means for assessing the driver's driving skill, a means for determining whether or not switching to autonomous driving mode is necessary and making a recommendation, a means for providing warnings and advice for safe driving as necessary based on the collected sensor data, and a means for collecting feedback from users and improving the system based on that feedback. This makes it possible to accurately assess the driver's fatigue level and driving skill and to suggest switching to autonomous driving mode at an appropriate time. It also makes it possible to provide safe driving support in real time and improve the system based on that feedback.

[0753] The "means for assessing the driver's level of fatigue" refers to a device or system that analyzes the driver's eye opening and closing and facial movements to quantify the level of fatigue.

[0754] "Various sensors" refers to a group of sensors that collect driving information such as vehicle speed, lane position, and distance to the vehicle ahead.

[0755] "Real-time analysis means" refers to devices or software for processing and analyzing collected sensor data in real time.

[0756] "Means for evaluating a driver's driving skills" refers to a system that analyzes the vehicle's driving patterns and the driver's operations, and quantifies or evaluates the skills.

[0757] "Means for determining whether or not switching to autonomous driving mode is necessary and proposing this" refers to a device or system that determines whether or not switching to autonomous driving mode is necessary based on collected data and evaluation results and notifies the driver.

[0758] "Means for providing warnings and advice for safe driving" refers to devices or software that provide instructions or warnings to drivers for safe driving based on data analyzed in real time.

[0759] "Means for collecting user feedback" refers to an interface through which drivers can input their opinions and thoughts about their driving experience, and a system for collecting that data.

[0760] A "means for improving the system" is a device or software that analyzes the collected feedback and adjusts or refines the operating parameters of the system.

[0761] MODE FOR CARRYING OUT THE INVENTION

[0762] This invention provides a specific method for an automated driving assistance system that evaluates the driver's fatigue level and driving skill and suggests transitioning to automated driving mode at an appropriate time. It also aims to enhance driver safety and comfort by collecting user feedback and continuously improving the system.

[0763] System configuration

[0764] The main components of the automated driving assistance system are a server, a terminal, and a user interface. The role and operation of each component are explained below.

[0765] Server Roles

[0766] The server plays a central role in the entire system, specifically implementing the following functions:

[0767] System initialization:

[0768] The server initializes the system, including the driver's current fatigue level, driving skill, collected sensor data, and past feedback.

[0769] Data Processing:

[0770] The server analyzes the collected sensor data and evaluates the driver's level of fatigue and driving skills.

[0771] System Improvements:

[0772] The server analyzes user feedback and adjusts the system's operating parameters.

[0773] Device Role

[0774] The terminal collects and analyzes data in real time from various sensors installed in the vehicle. Specific functions are as follows:

[0775] Sensor data collection:

[0776] The device collects data from sensors installed in the vehicle, including the vehicle's speed, lane position, distance to the vehicle ahead, and a driver fatigue index based on facial recognition.

[0777] Driver Condition Rating:

[0778] The device evaluates the driver's level of fatigue and driving skills based on the data collected.

[0779] Autonomous driving mode suggestions:

[0780] The device determines whether it is necessary to switch to autonomous driving mode and notifies the driver.

[0781] Safe driving support:

[0782] The device analyzes the data in real time and issues warnings to the driver as needed.

[0783] User Interface

[0784] The user interface is primarily responsible for:

[0785] Providing feedback:

[0786] The system provides an interface that allows users to easily provide feedback, such as specific opinions like "too extreme" or "too slow."

[0787] Notification function:

[0788] When suggesting transition to autonomous driving mode, audio and visual notifications will be provided.

[0789] Hardware and software used

[0790] Hardware:

[0791] Various sensors (speed sensors, lane position sensors, distance sensors, face recognition cameras, etc.)

[0792] On-board computer (terminal)

[0793] server

[0794] software:

[0795] Data Analysis Software

[0796] Feedback Analysis Algorithm

[0797] Notification and warning interface program

[0798] Specific examples

[0799] Driver fatigue assessment and autonomous driving mode proposal

[0800] As a specific example, consider a driver who continues to drive for long periods of time. The device evaluates the driver's fatigue level in real time, and if it exceeds a certain level, a voice notification will be issued asking, "Do you want to switch the vehicle to autonomous driving mode?" At this point, the driver can choose to switch to autonomous driving mode via the voice notification.

[0801] Safe driving support function

[0802] If the distance to the vehicle ahead becomes too short while driving, the device will issue a warning saying, "You are getting too close to the vehicle ahead," prompting the driver to take immediate action to increase their distance.

[0803] Gathering feedback and improving the system

[0804] After completing the drive, the user provides feedback such as "My lane change was a bit extreme." The server analyzes this feedback and adjusts the system's operating parameters for the next drive, improving the user experience.

[0805] Prompt Sentence Examples

[0806] Examples of prompts to enter into a generative AI model include:

[0807] Prompt: Describe the specific steps for fatigue assessment and feedback processing in an automated driving assistance system.

[0808] Objective: Learn more about how to assess driver fatigue in real time and suggest automated driving modes at the right time.

[0809] With such settings, the automatic driving assistance system of the present invention can improve the safety and comfort of the driver.

[0810] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0811] The flow of this system's program processing

[0812] Step 1:

[0813] System initialization

[0814] The server initializes the system. As input, it obtains the driver's current fatigue level, driving skill, collected sensor data, and past feedback data from the database. Data processing generates a driver profile and initializes driving tendencies. As output, it obtains an initialized system and an initial driver profile. Specifically, the server reads driving history data from the database and prepares for a new driving session.

[0815] Step 2:

[0816] Sensor data collection

[0817] The terminal collects data in real time from various sensors attached to the vehicle. The input includes data obtained from speed sensors, lane position sensors, distance sensors, face recognition cameras, etc. Data processing involves storing each sensor data in a buffer and converting it into the required data format. As an output, real-time data from the sensors is stored in the buffer. Specifically, the terminal collects the latest driving information every second and sends that information to the server in real time.

[0818] Step 3:

[0819] Driver condition assessment

[0820] The driver's condition is evaluated based on the sensor data collected by the device. The input includes real-time sensor data stored in a buffer. For data calculation, facial recognition technology is used to analyze the opening and closing of the driver's eyes and facial movements to quantify the level of fatigue. Driving patterns are also analyzed to evaluate driving skills. The output is a driver fatigue score and driving skill score. Specifically, an AI model is used to evaluate real-time data, and the driver's fatigue level is displayed on the screen.

[0821] Step 4:

[0822] Proposal for autonomous driving mode

[0823] The device will suggest switching to autonomous driving mode based on the evaluation results. Inputs include the driver's fatigue score and driving skill score. Data calculations compare the evaluation score with preset reference values ​​to determine whether it is necessary. As an output, a notification suggesting switching to autonomous driving mode is generated for the driver. Specifically, the device will issue a voice notification asking, "Do you want to switch the vehicle to autonomous driving mode?"

[0824] Step 5:

[0825] Safe driving support

[0826] The device analyzes sensor data in real time and provides warnings and advice for safe driving as needed. The input includes continuously collected sensor data. Data processing monitors the distance to the vehicle ahead and lane departure, generating a warning if certain criteria are exceeded. The output generates audio and visual warnings for the driver. Specifically, the device displays a warning message such as "You are getting too close to the vehicle ahead."

[0827] Step 6:

[0828] Gathering feedback

[0829] The user provides feedback on the driving experience. The input includes feedback data provided through the user interface. Data processing classifies the feedback and stores it as analysis data for system improvement. The output is the collected feedback stored on the server. Specifically, the user enters "The lane change was a bit extreme" into the smartphone app.

[0830] Step 7:

[0831] System Improvements

[0832] The server improves the system based on the collected feedback. Inputs include user feedback and past driving data. Data calculations involve analyzing the feedback, reassessing the driver's driving skills, and adjusting the system's operating parameters. The output is improved system settings. Specifically, the server analyzes the feedback and configures the system to operate with the new parameter set the next time the vehicle is driven.

[0833] In this way, automated driving assistance systems can increase driver safety and comfort.

[0834] (Application example 1)

[0835] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0836] Current autonomous driving technology does not adequately assess driver fatigue and driving skills, making it difficult to fully guarantee driver safety and comfort. Furthermore, there is a lack of mechanisms for detecting driver fatigue and loss of concentration in real time during long periods of driving and prompting the vehicle to transition to autonomous driving mode at the appropriate time. Furthermore, existing systems lack sufficient processes for collecting user feedback and improving the system, resulting in low reliability and social acceptance. Therefore, there is a need for an autonomous driving assistance system that can be continuously improved while achieving both safety and comfort.

[0837] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0838] In this invention, the server includes a means for assessing the driver's fatigue level, a means for collecting and analyzing sensor data, a means for assessing the driver's driving skill, a means for proposing transition to autonomous driving mode, a means for collecting user feedback and improving the system, and a means for providing visual and audio warnings to support the driver's safe driving. This makes it possible to evaluate the driver's condition in real time and suggest transitioning to autonomous driving mode at an appropriate time. It also supports safe driving by notifying the driver of dangerous situations while driving with visual and audio warnings. Furthermore, by collecting user feedback and continuously improving the system based on that feedback, it is possible to increase reliability and social acceptance.

[0839] The "means for assessing driver fatigue" refers to a method and device for quantifying and assessing the driver's fatigue level in real time based on facial recognition, eye movements, etc.

[0840] "Means for collecting and analyzing sensor data" refers to the technology and equipment that collects data such as speed, lane position, and distance to the vehicle ahead from various sensors attached to the vehicle and analyzes it.

[0841] The "means for assessing the driver's driving skills" refers to algorithms and devices for assessing the driving skills based on collected sensor data and the driver's driving history.

[0842] "Means for proposing transition to autonomous driving mode" refers to technologies and devices that notify and suggest transition to autonomous driving mode to the driver at an appropriate time based on the evaluation results.

[0843] "Means for collecting user feedback and improving the system" refers to technologies and devices that collect feedback provided by drivers based on their driving experience, analyze it, adjust system parameters, and continuously improve the system.

[0844] "Means for providing visual and audio warnings to support safe driving" refers to technologies and devices that provide visual and audio warnings to drivers based on collected sensor data and evaluation results, and encourage appropriate safe driving behavior.

[0845] The following configurations are included as embodiments of the present invention.

[0846] The server initializes the system and provides a means to evaluate the driver's fatigue level and driving skills. It uses data from various sensors attached to the vehicle (speed, lane position, distance to the vehicle ahead, etc.) to collect and analyze sensor data. The analyzed data is sent to the server, which evaluates the driver's condition in real time.

[0847] Based on the evaluation results, the device provides a means to suggest switching to autonomous driving mode. This suggestion is made using visual and audio output devices such as smart glasses. For example, if the driver's fatigue level exceeds a certain level, the device will notify the driver via audio and visual means, saying, "Your fatigue level is high. Would you like to suggest switching to autonomous driving mode?" If the driver answers "Yes" via audio, the device will smoothly switch to autonomous driving mode.

[0848] In addition, the device provides visual and audio warnings to help drivers drive safely. For example, if the distance to the vehicle ahead is significantly short, the device will warn the driver that they are driving too close to the vehicle ahead, allowing them to immediately recognize the situation and take appropriate action.

[0849] After completing a drive, users can provide feedback to the system. The feedback is sent to the server and used to improve the system. For example, specific opinions such as "Your lane changes were a bit extreme" can be reflected and the system's operating parameters will be adjusted for the next drive. This allows the system to continuously improve, increasing its reliability and social acceptance.

[0850] Description of the hardware and software used

[0851] Hardware: Smart glasses (e.g., Microsoft HoloLens, Google Glass), various in-vehicle sensors

[0852] Software: Driver state assessment algorithm, smart glasses SDK (e.g. GoggleSDK)

[0853] Data processing and calculation

[0854] The data is processed as follows:

[0855] 1. Sensor data collection:

[0856] Data is collected from various sensors installed in the vehicle.

[0857] 2. Driver Condition Assessment:

[0858] Based on the collected sensor data, algorithms are applied to assess fatigue levels and driving skills.

[0859] 3. Autonomous driving mode suggestions and warnings:

[0860] -Based on the evaluation results, the device will suggest transitioning to autonomous driving mode and provide visual and audio warnings.

[0861] Examples of concrete examples and prompts

[0862] Specific examples

[0863] When a driver is driving for a long period of time, the smart glasses monitor the driver's eye movements and facial expressions in real time to detect an increase in fatigue. As a result, a notification will appear on the smart glasses' HUD saying, "Fatigue level is high. Would you like to suggest switching to autonomous driving mode?" If the driver answers "Yes" aloud, the smart glasses will automatically switch to autonomous driving mode.

[0864] Prompt Sentence Examples

[0865] Consider a scenario in which smart glasses monitor the eye movements and facial expressions of a driver in real time while driving for long periods of time to detect increased fatigue, and explain the appropriate message display and voice response procedures.

[0866] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0867] Step 1:

[0868] System initialization

[0869] The server initializes the system. The inputs are the driver's current fatigue level, driving skill, collected sensor data, and past feedback. Based on this, all initial parameters are set. The output is the initialized system state. Specifically, the server resets all parameters and performs the initial settings.

[0870] Step 2:

[0871] Sensor data collection

[0872] The terminal collects data from various sensors attached to the vehicle. The input is real-time data such as the vehicle's speed, lane position, and distance to the vehicle ahead. This data is collected by the sensors and sent to the terminal. The output is a set of collected sensor data. Specifically, the terminal collects sensor data at regular intervals and stores it for analysis.

[0873] Step 3:

[0874] Driver condition assessment

[0875] The device evaluates the driver's fatigue level and driving skills based on the sensor data collected. The sensor data collected in step 2 is used as input. A data analysis algorithm quantifies the fatigue level and evaluates the driving skills. The output is the quantified fatigue level and driving skill evaluation results. Specifically, the device executes the data analysis algorithm and calculates the evaluation results.

[0876] Step 4:

[0877] Proposal for autonomous driving mode

[0878] Based on the evaluation results, the device will suggest switching to autonomous driving mode. The inputs are the fatigue level and driving skill evaluation results obtained in step 3. When the fatigue level exceeds a certain value, the device will output a suggestion notification. Specifically, it will display a message on the smart glasses' HUD and notify the driver by voice, "Your fatigue level is high. Would you like to suggest switching to autonomous driving mode?"

[0879] Step 5:

[0880] Safe driving support

[0881] The device analyzes sensor data in real time and provides warnings and advice for safe driving. The input is sensor data and the driver's state evaluation results. The output is visual and audio warning and advice messages. Specifically, the device monitors the distance to the vehicle ahead and warns the driver that "you are too close to the vehicle ahead" if the distance is too close.

[0882] Step 6:

[0883] Gathering feedback

[0884] The user provides feedback about the driving experience. The input is the specific feedback provided by the driver, including their opinions and thoughts about the system's behavior. The output is the collected feedback data. The specific operation is that the terminal receives the feedback from the user and sends it to the server.

[0885] Step 7:

[0886] System Improvements

[0887] The server adjusts and improves the system's operating parameters based on the collected feedback. The input is the feedback data collected in step 6. The feedback is analyzed and necessary adjustments are made. The output is the improved system's operating parameters. Specifically, the server runs the feedback analysis algorithm and updates the system settings.

[0888] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0889] This invention combines an emotion engine with an automated driving assistance system to improve the effectiveness of driver fatigue assessment and feedback collection, and to improve the accuracy of suggestions for switching to automated driving mode. The emotion engine recognizes emotions from the user's tone of voice, facial expressions, body movements, etc., and reflects them in the system.

[0890] Program processing

[0891] 1. System initialization

[0892] The server initializes the system. When creating an instance of UrbanCohesionDriveAssist, it initializes the driver's fatigue level, driving skill, sensor data, emotional data, and feedback.

[0893] 2. Collecting sensor data and emotion data

[0894] The device calls the collect_sensor_data method to collect data from the vehicle's sensors and the emotion engine that recognizes the user's emotions. The collected data includes speed, lane position, distance to the vehicle ahead, and the user's emotional state (e.g., stress level, distraction level, etc.).

[0895] 3. Driver condition assessment

[0896] The device calls the evaluate_driver_state method to comprehensively evaluate the driver's fatigue level based on emotional data and driving data, taking into account, for example, whether the driver is not only tired but also in a high-stress state.

[0897] 4. Proposal for autonomous driving mode

[0898] The device calls the evaluate_and_propose_autonomous_mode method and suggests switching to autonomous driving mode if the driver's state or emotional data exceeds a certain threshold. Specifically, the device encourages switching to autonomous driving mode even if the driver is feeling highly stressed based on emotional data.

[0899] 5. Safe driving support

[0900] The device analyzes sensor data and emotional data to provide safe driving support. For example, it not only issues a warning when the distance to the vehicle ahead is short, but also considers the causes of driver distraction based on emotional data and provides warnings and advice based on that.

[0901] 6. Gathering Feedback

[0902] When a user provides feedback using the collect_feedback method, we also collect emotional data, such as tone of voice and facial expressions, to assess the user's true perception of the quality of the experience.

[0903] 7. System Improvements

[0904] The server calls the improve_system_based_on_feedback method to analyze the feedback and emotional data. Based on this, the system can be improved. For example, if the user perceives the system as "extreme," and the emotional data indicates a stress response, the server can adjust the driver's skill level or the system's response speed.

[0905] Specific examples

[0906] Next, a specific example of the present invention will be shown. For example, when a driver has been driving for a long time, the system evaluates the driver's fatigue level in real time. This evaluation includes emotional data such as the driver's pale complexion or nervous expression. If the fatigue level exceeds a certain threshold, the device will issue a voice notification saying, "You seem tired. Would you like to switch to autonomous driving mode?" and suggest a smooth transition to autonomous driving mode.

[0907] Additionally, if the distance to the vehicle ahead is significantly short while driving, the sensor data will detect this information, and the emotion engine will sense anxiety from the driver's facial expression. Based on this information, the device will issue a warning to the driver saying, "You are getting very close to the vehicle ahead. Please be careful."

[0908] Furthermore, when a user provides feedback after completing a drive, such as "My lane change was a bit extreme," the emotion engine analyzes the user's tone of voice and facial expression when providing the feedback. From these results, it is determined that the user truly felt the lane change was extreme, and the server adjusts the system parameters for the next drive. For example, it may take measures such as reducing the speed when changing lanes.

[0909] In this way, by using an autonomous driving assistance system that combines an emotion engine, it becomes possible to provide advanced assistance that takes into account the driver's state and emotions, thereby improving safety and comfort.

[0910] The processing flow will be explained below.

[0911] Step 1:

[0912] The server initializes the system. When creating an instance of UrbanCohesionDriveAssist, it sets the driver's fatigue level to 0 and their driving skill to intermediate level (5). It also initializes the sensor data, emotion data, and feedback lists as empty.

[0913] Step 2:

[0914] The terminal calls the collect_sensor_data method to collect data from various sensors installed in the vehicle. The collected data includes speed, lane position, and distance to the vehicle ahead. This data is randomly generated and added to the sensor data list.

[0915] Step 3:

[0916] The device uses an emotion engine to collect emotional data from the driver's facial expressions, tone of voice, and body movements, for example, assessing whether the driver is tired or stressed and adding this to the emotional data list.

[0917] Step 4:

[0918] The device calls the evaluate_driver_state method to evaluate the driver's fatigue level based on the collected sensor data and emotional data. The fatigue level is calculated by an integrated analysis of the sensor data and emotional data. For example, if the driver's attention is distracted while driving and their facial expression indicates fatigue, the fatigue level is set high.

[0919] Step 5:

[0920] The device calls the evaluate_and_propose_autonomous_mode method to propose switching to autonomous driving mode if the driver's fatigue level or emotion data exceeds a certain threshold. If the proposal is accepted, the switch_to_autonomous_mode method is called to switch to autonomous driving mode.

[0921] Step 6:

[0922] The device calls the provide_safety_support method, analyzes sensor data and emotion data, and provides safe driving support. For example, if the distance to the vehicle ahead is too close and the emotion data indicates that the driver is nervous, the device will warn, "You are getting too close to the vehicle ahead. Please be careful."

[0923] Step 7:

[0924] Users provide feedback using the collect_feedback method. The feedback includes specific opinions about the driving experience, as well as emotional data at the time of the feedback. For example, the user's opinion that "the system's behavior was too aggressive" is recorded, along with the user's tone of voice and facial expression.

[0925] Step 8:

[0926] The server calls the improve_system_based_on_feedback method to analyze the collected feedback and emotion data. Based on the analysis results, the driver's driving skills are reevaluated and the system's operating parameters are adjusted. For example, if the feedback indicates that the driver's driving skills are "too aggressive," the system may adjust the lane change speed accordingly.

[0927] Example 2

[0928] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0929] While conventional automated driving assistance systems can partially take into account the driver's level of fatigue and driving skill, it is difficult to provide safe driving assistance that fully reflects the driver's emotional state or to improve the system based on feedback. Furthermore, emotional data is not sufficiently taken into account when maintaining a safe distance from the vehicle ahead or analyzing surrounding traffic conditions, leaving issues with safety and comfort. Therefore, there is a need for a system that can evaluate the driver's emotions in real time and provide appropriate assistance and suggestions.

[0930] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for evaluating the driver's fatigue level, a means for collecting and analyzing sensor data and emotional data, a means for evaluating the driver's driving skill and emotional state, a means for suggesting transition to an autonomous driving mode based on the driver's state and emotional data, and a means for collecting feedback from the user and analyzing the emotional data at the time of feedback to improve the system. This enables advanced support and suggestions that take the driver's emotional state into consideration, thereby improving safety and comfort.

[0931] "Fatigue level" refers to the degree of physical and mental fatigue a driver experiences while driving.

[0932] "Sensor data" refers to information collected by various sensors attached to a vehicle, and specifically includes data related to driving, such as speed, lane position, and distance to the vehicle ahead.

[0933] "Emotional data" refers to information about a driver's emotional state analyzed from their tone of voice, facial expressions, body movements, etc., and specifically includes stress levels and distractibility.

[0934] "Driving skill" is an evaluation of the driver's level of proficiency and skill in driving.

[0935] "Autonomous driving mode" is a mode in which the vehicle can operate autonomously without driver operation.

[0936] "Feedback" refers to opinions and evaluations provided by users to the system, and includes information that can be used to improve the system.

[0937] An "autonomous driving assistance system" is a system that assists drivers in driving and improves safety and comfort, and includes fatigue assessment, collection and analysis of sensor data and emotional data, driving skill assessment, autonomous driving mode proposals, feedback collection, and system improvement.

[0938] A "warning" is a notification or alert that notifies the driver of an impending hazard.

[0939] "Safe driving support" refers to advice and assistance provided to drivers to help them drive safely, and specifically includes maintaining a safe distance from the vehicle ahead and warning against dangerous driving.

[0940] MODE FOR CARRYING OUT THE INVENTION

[0941] This invention combines an emotion engine with an automated driving assistance system to improve the effectiveness of driver fatigue assessment and feedback collection, and to improve the accuracy of suggestions for switching to automated driving mode. The emotion engine recognizes emotions from the user's tone of voice, facial expressions, body movements, etc., and reflects them in the system.

[0942] System initialization

[0943] The server creates an instance of UrbanCohesionDriveAssist when the system is initialized. This instance is initialized with the driver's fatigue level, driving skill, sensor data, emotional data, and feedback. During initialization, the necessary data is synchronized using an existing database or cloud server.

[0944] Data collection

[0945] The device calls the collect_sensor_data method to collect data such as speed, lane position, and distance to the vehicle ahead from sensors installed in the vehicle. At the same time, the emotion engine analyzes the user's tone of voice, facial expressions, and body movements to collect emotional data such as stress level and distraction level. This provides both sensor data and emotional data to the system in real time.

[0946] Driver condition assessment

[0947] The device uses the evaluate_driver_state method to evaluate the driver's fatigue level based on the collected sensor data and emotional data. For example, a pale complexion and distractedness after a long drive could be used as evaluation points. The device also evaluates the driver's stress level based on the driver's tone of voice and facial expression, and combines this with the fatigue level evaluation to provide a comprehensive state evaluation.

[0948] Proposal for autonomous driving mode

[0949] The device calls the evaluate_and_propose_autonomous_mode method and proposes switching to autonomous driving mode if the driver's condition or emotional data exceeds a certain threshold. For example, if the user's facial expression indicates extreme fatigue, the device will issue a voice notification asking, "Do you want to switch to autonomous driving mode?"

[0950] Safe driving support

[0951] If sensor data indicates that the distance to the vehicle ahead is dangerously close, the device will issue a warning saying, "You are approaching the vehicle in front. Please be careful." If emotional data detects that the driver is anxious or distracted, the device will provide specific advice based on that emotional data, such as, "Take a deep breath and relax."

[0952] Gathering feedback and improving the system

[0953] When a user provides feedback after driving using the collect_feedback method, the emotion engine analyzes the tone of voice and facial expressions to evaluate the quality of the feedback. Based on this result, the server uses the improve_system_based_on_feedback method to improve the system. For example, taking into account past feedback and emotion data, the server can improve safety by reducing the lane-changing speed in the next driving session.

[0954] Specific examples

[0955] For example, if a driver has been driving for a long time, the server initializes the system and the device calls the appropriate method. The device uses the collect_sensor_data method to collect the vehicle's speed and distance from the vehicle ahead, and analyzes the driver's facial expression and tone of voice using the emotion engine. The emotion data and driving data are combined to evaluate the driver's condition. If the evaluation results indicate that the driver is fatigued and not paying attention, the device will suggest, "Do you want to switch to autonomous driving mode?" If the driver is dangerously close to the vehicle ahead, the device will warn, "You are very close to the vehicle ahead. Please be careful." After completing the drive, if the user provides feedback such as "My lane change was a bit aggressive," the emotion engine analyzes the user's tone of voice and facial expression to identify the specific reason why they thought the lane change was aggressive. The server uses this data to adjust the system parameters and reduce the lane change speed on the next drive.

[0956] An example of a prompt for a generative AI model is:

[0957] Please explain in natural language how the system detects the user's fatigue after a long drive and what suggestions it makes in response.

[0958] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0959] Step 1: Initialize the system

[0960] The server creates an instance of UrbanCohesionDriveAssist, which is initialized with the driver's fatigue level, driving skill, sensor data, emotional data, and feedback. The input is the initial setting data stored in an existing database or cloud server, and the output is the initialized system instance.

[0961] Step 2: Collecting sensor and emotion data

[0962] The device uses sensors installed in the vehicle and an emotion engine to collect driving-related sensor data and driver emotion data. Specifically, it calls the collect_sensor_data method to collect data such as speed, lane position, and distance to the vehicle ahead. It also analyzes stress levels and distraction levels from tone of voice, facial expressions, and body movements through the emotion engine. The inputs are the vehicle's sensor information and raw data from the emotion engine, and the output is analyzed sensor data and emotion data.

[0963] Step 3: Driver condition assessment

[0964] The device analyzes sensor data and emotion data to evaluate the driver's fatigue and stress levels. Specifically, it calls the evaluate_driver_state method to evaluate the facial expression and level of distraction of a driver who has been driving for a long time. The input is the collected sensor data and emotion data, and the output is the evaluation result, which includes the driver's fatigue and stress levels.

[0965] Step 4: Proposing an autonomous driving mode

[0966] The device proposes transitioning to autonomous driving mode based on the results of the driver's state evaluation. Specifically, it calls the evaluate_and_propose_autonomous_mode method, and if the driver's fatigue or stress level exceeds a certain threshold, it issues a voice notification suggesting, "Do you want to switch to autonomous driving mode?" The input is the driver's state evaluation result, and the output is a notification proposing transition to autonomous driving mode.

[0967] Step 5: Safe driving support

[0968] The device provides safe driving support to the driver based on the collected sensor data and emotional data. Specifically, it evaluates the distance to the vehicle ahead and issues a warning such as, "You are very close to the vehicle ahead. Please be careful." It also detects anxiety or distraction in the driver from emotional data and provides advice such as, "Take a deep breath and relax." The input is sensor data and emotional data, and the output is specific warnings and advice.

[0969] Step 6: Gather feedback

[0970] The user provides feedback after driving. The device uses the collect_feedback method to analyze the feedback along with the tone of voice and facial expression. The emotion engine is used to collect emotion data when providing the feedback and save the analysis results. The input is the user's feedback and emotion data, and the output is the analyzed feedback data.

[0971] Step 7: Improve your system

[0972] The server analyzes the collected feedback and emotion data and makes improvements to the system. Specifically, it calls the improve_system_based_on_feedback method to adjust system parameters based on the analyzed data. For example, it takes into account past feedback and emotion data and reduces the lane-changing speed in the next driving session. The input is the analyzed feedback data, and the output is the adjusted system parameters.

[0973] (Application example 2)

[0974] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0975] While conventional automated driving assistance systems can evaluate the driver's level of fatigue and driving skill, they do not adequately suggest switching to automated driving or provide safe driving support that accurately reflects the driver's emotional state. This means that the system is unable to respond quickly to sudden changes in the driver's physical condition or emotional fluctuations, which could compromise the safety and comfort of automated driving.

[0976] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for evaluating the driver's fatigue level, a means for collecting and analyzing sensor data, voice tone, and facial expressions, a means for evaluating the driver's driving skill and emotional state, a means for proposing transition to autonomous driving mode, and a means for collecting feedback from the user and improving the system. This makes it possible to evaluate the driver's state of fatigue, stress, and distraction in real time and propose transition to autonomous driving mode at an appropriate time.

[0977] "Driver fatigue level" refers to the degree of physical or mental fatigue a driver feels while driving.

[0978] "Sensor data" refers to information collected from sensors attached to vehicles and related equipment, and includes physical data such as speed, position, and distance.

[0979] "Voice tone" refers to characteristics of the driver's voice, such as pitch, strength, and intonation, and is information used to estimate the driver's emotional state.

[0980] "Facial expression" refers to the movement and changes of the driver's facial muscles, and is information used to analyze the driver's emotional state.

[0981] "Driving skills" refers to a driver's ability to operate a vehicle safely and efficiently, and its assessment includes the driver's experience and skill.

[0982] "Emotional state" refers to the driver's psychological state, including states such as stress, distraction, and relaxation.

[0983] "Autonomous driving mode" refers to a mode in which the vehicle drives automatically without driver intervention, aiming to improve safety and convenience.

[0984] "Feedback" refers to opinions and impressions from users, providing information about the performance and usability of the system.

[0985] "System Improvement" refers to actions taken to improve the performance and functionality of a system based on collected feedback and data.

[0986] This invention combines an emotion engine with an automated driving assistance system to evaluate the driver's fatigue level, driving skill, and emotional state, and proposes transitioning to automated driving mode at the appropriate time. The system uses smart glasses and other sensor devices to grasp the driver's emotional state in real time, and collects and analyzes feedback.

[0987] System configuration

[0988] Hardware:

[0989] Smart glasses: Equipped with a camera and microphone to collect the driver's facial expressions and voice.

[0990] Sensors installed in the vehicle collect driving data such as speed, lane position, and distance to the vehicle ahead.

[0991] software:

[0992] Emotion Engine: A model for analyzing a driver's emotional state from their facial expressions, vocal tone, and body movements.

[0993] Autonomous driving assistance software: A program that integrates sensor data and emotional data to assess the driver's condition and suggest autonomous driving modes.

[0994] System Operation

[0995] 1. Data Collection:

[0996] The device collects the driver's facial expressions and voice through the smart glasses' camera and microphone, and also acquires data such as speed, location, and distance to the vehicle ahead from sensors installed in the vehicle.

[0997] 2. Emotional state assessment:

[0998] The collected data is analyzed by an on-board emotion engine, which assesses the driver's fatigue, stress level and distraction.

[0999] 3. Driver Condition Assessment:

[1000] The device comprehensively evaluates the driver's emotional and driving data to analyze their fatigue and stress levels, and based on this evaluation, the driver's attention and driving skills are also taken into account.

[1001] 4. Proposal for autonomous driving mode:

[1002] If the driver's condition exceeds a certain threshold, the device will suggest transitioning to autonomous driving mode and notify the driver through audio and visual feedback.

[1003] 5. Safe driving support:

[1004] Based on sensor data and emotional data, the device provides safe driving support, for example by issuing a warning when the distance to the vehicle ahead is short or when the driver's stress level is high.

[1005] 6. Gathering feedback and improving the system:

[1006] The user provides feedback after completing a drive, and the emotion engine analyzes the user's emotional state. The server then uses this feedback to improve the system and apply it to the next drive.

[1007] Specific examples

[1008] During a long drive, the smart glasses suggest, "You seem tired. Would you like to switch to autonomous driving mode?" At this time, it is clear that the driver is not only tired, but also stressed from their facial expressions. Furthermore, when providing feedback after the drive that "your lane change was a bit extreme," the tone of voice and facial expressions at the time are analyzed to determine whether that was really how they felt, and the system will adjust accordingly for the next drive.

[1009] Prompt Sentence Examples

[1010] "You're driving a long distance. Your smart glasses might say: 'You're getting very close to the vehicle ahead. Be careful.' They might also suggest: 'You seem tired. Would you like to shift into self-driving mode?' If you provide feedback like 'You're changing lanes too aggressively,' the system will use this feedback and your emotional data to make adjustments the next time you drive."

[1011] In this way, by implementing the present invention, advanced automated driving assistance that takes into account the emotional state of the driver can be realized.

[1012] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1013] Step 1:

[1014] System initialization

[1015] The server initializes the system and initializes the driver's fatigue level, driving skill, sensor data, emotional data, and feedback. It uses the system's initial settings and past data as input, and generates an initial database as output. This process prepares the system for data collection and analysis.

[1016] Step 2:

[1017] Sensor and emotion data collection

[1018] The device collects data from sensors installed in the smart glasses and the vehicle. Specifically, the smart glasses' camera captures the driver's facial expressions, the microphone records audio, and the vehicle sensors collect driving data such as speed, position, and distance between vehicles. The device uses real-time data from the sensors as input and receives and stores all of this data in a consistent format as output.

[1019] Step 3:

[1020] Driver condition assessment

[1021] The device analyzes the collected sensor data and emotional data. Specifically, it uses an emotion engine to evaluate the driver's fatigue and stress levels based on their facial expressions and voice. The input is the data collected in the previous step, and the output is an evaluation of the driver's fatigue and emotional state. This evaluation allows the driver's current physical and mental state to be understood.

[1022] Step 4:

[1023] Proposal for autonomous driving mode

[1024] Based on the evaluation results, the device will suggest switching to autonomous driving mode. Specifically, if the evaluation exceeds a certain threshold, the device will notify the driver audibly and visually, saying, "You seem tired. Would you like to switch to autonomous driving mode?" The device uses the evaluation results of fatigue level and emotional state as input, and generates and displays a suggestion message as output.

[1025] Step 5:

[1026] Safe driving support

[1027] The device analyzes sensor data and emotional data in real time to provide safe driving support. Specifically, it issues a warning when the distance to the vehicle ahead is close or the driver is under high stress. It uses sensor data and emotional data as input and provides warning messages and advice in audio and visual formats as output.

[1028] Step 6:

[1029] Gathering feedback

[1030] The user provides feedback after completing the drive. The device collects this feedback data and analyzes the tone of voice and facial expressions at the time of providing the feedback using an emotion engine. The user's feedback and emotional data at that time are used as input, and the analysis results of the feedback are obtained as output.

[1031] Step 7:

[1032] System Improvements

[1033] The server analyzes the collected feedback and emotion data and adjusts the system parameters accordingly, specifically adjusting the system's response based on the feedback and applying it to the next drive. The server uses the feedback analysis results as input and obtains improved system settings as output.

[1034] This series of steps enables the driver's emotional state to be grasped in real time, allowing appropriate automated driving suggestions and safe driving support to be realized.

[1035] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1036] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1037] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1038] [Fourth embodiment]

[1039] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1040] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1041] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1042] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1043] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1044] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1045] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1046] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1047] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1048] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1049] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1050] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1051] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1052] This invention provides a specific method for evaluating driver fatigue and driving skills in an automated driving assistance system and proposing a transition to automated driving mode at an appropriate time. It also aims to enhance user trust and social acceptance by collecting user feedback and continuously improving the system.

[1053] Program processing

[1054] 1. System initialization

[1055] The server initializes the system. At the time of instantiation, the driver's current fatigue level, driving skill, collected sensor data, and past feedback are initialized.

[1056] 2. Sensor data collection

[1057] The device collects data from various sensors installed in the vehicle, including driving information such as the vehicle's speed, lane position, and distance to the vehicle ahead.

[1058] 3. Driver condition assessment

[1059] The device assesses the driver's level of fatigue, which is quantified based on certain criteria and indicates a decline in attention and reaction time while driving.

[1060] 4. Proposal for autonomous driving mode

[1061] The device will determine whether to switch to autonomous driving mode based on the collected data and the evaluation results, and will suggest switching to autonomous driving mode if the driver is highly fatigued or the driving conditions are difficult.

[1062] 5. Safe driving support

[1063] It analyzes sensor data and provides warnings and advice for safe driving as needed, for example, if the distance to the vehicle ahead is too close.

[1064] 6. Gathering Feedback

[1065] The user provides feedback about the driving experience, for example, by providing specific feedback such as whether the system is "too aggressive" or "too slow."

[1066] 7. System Improvements

[1067] The server improves the system based on the collected feedback. It analyzes the feedback, reevaluates the driver's driving skills based on the feedback, and adjusts the system's operating parameters.

[1068] Specific examples

[1069] Next, a specific example of the automated driving assistance system of the present invention will be shown. For example, if a driver has been driving for a long time, the system will evaluate the driver's fatigue level in real time. If the fatigue level exceeds a certain level, the device will suggest switching to automated driving mode. At this time, the driver will be notified by voice, "Do you want to switch the vehicle to automated driving mode?", ensuring a smooth transition to automated driving mode.

[1070] Additionally, if the distance to the vehicle ahead is significantly short while driving, the sensor will detect this information and the device will issue a warning saying, "You are getting too close to the vehicle ahead." This allows the driver to immediately recognize the situation and maintain safe driving.

[1071] Furthermore, suppose the user provides feedback after the drive, such as "My lane change was a bit extreme." The server analyzes this feedback and adjusts the system parameters for the next drive. In this way, the system is continuously improved based on the user's experience and opinions.

[1072] By combining these functions, the automated driving assistance system of the present invention can improve the safety and comfort of the driver.

[1073] The processing flow will be explained below.

[1074] Step 1:

[1075] The server initializes the system by creating an instance of UrbanCohesionDriveAssist, setting the driver's current fatigue level to 0, and the driving skill level to intermediate (5). It also initializes the sensor data and feedback lists as empty.

[1076] Step 2:

[1077] The terminal calls the collect_sensor_data method to collect data from various sensors installed in the vehicle. The collected data includes speed, lane position, and distance to the vehicle ahead. A random number is generated for each piece of data and recorded as sensor data.

[1078] Step 3:

[1079] The device calls the evaluate_driver_state method to evaluate the driver's fatigue level. The fatigue level increases by 1 and is adjusted downward according to the driver's driving skill. As a result, the fatigue level is set so that it will not fall below 0.

[1080] Step 4:

[1081] The device calls the evaluate_and_propose_autonomous_mode method to propose switching to autonomous driving mode if the driver's fatigue level exceeds a certain threshold (e.g., 10). If the proposal is accepted, the switch_to_autonomous_mode method is called to switch to autonomous driving mode.

[1082] Step 5:

[1083] The device calls the provide_safety_support method to provide safe driving support based on the collected sensor data. For example, if the distance to the vehicle ahead is less than 20 meters, the device warns the driver that the vehicle is "too close to the vehicle ahead."

[1084] Step 6:

[1085] The user provides feedback using the collect_feedback method. Feedback includes their opinion and evaluation of the driving experience, such as "the system is too aggressive" or "it's too slow."

[1086] Step 7:

[1087] The server calls the improve_system_based_on_feedback method to analyze the collected feedback and improve the system. It evaluates the feedback and adjusts the driver's driving skills and the system's operating parameters accordingly. The improved settings are then applied the next time the vehicle is driven.

[1088] Example 1

[1089] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1090] Conventional automated driving assistance systems have problems such as insufficient assessment of driver fatigue and driving skills, making it impossible to suggest switching to automated driving mode at the appropriate time. They also lack the ability to provide real-time safe driving support based on sensor data and improve the system based on user feedback. As a result, they have been unable to fully ensure driver safety and comfort.

[1091] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1092] In this invention, the server includes a means for assessing the driver's fatigue level, a means for collecting sensor data from various sensors and analyzing it in real time, a means for assessing the driver's driving skill, a means for determining whether or not switching to autonomous driving mode is necessary and making a recommendation, a means for providing warnings and advice for safe driving as necessary based on the collected sensor data, and a means for collecting feedback from users and improving the system based on that feedback. This makes it possible to accurately assess the driver's fatigue level and driving skill and to suggest switching to autonomous driving mode at an appropriate time. It also makes it possible to provide safe driving support in real time and improve the system based on that feedback.

[1093] The "means for assessing the driver's level of fatigue" refers to a device or system that analyzes the driver's eye opening and closing and facial movements to quantify the level of fatigue.

[1094] "Various sensors" refers to a group of sensors that collect driving information such as vehicle speed, lane position, and distance to the vehicle ahead.

[1095] "Real-time analysis means" refers to devices or software for processing and analyzing collected sensor data in real time.

[1096] "Means for evaluating a driver's driving skills" refers to a system that analyzes the vehicle's driving patterns and the driver's operations, and quantifies or evaluates the skills.

[1097] "Means for determining whether or not switching to autonomous driving mode is necessary and proposing this" refers to a device or system that determines whether or not switching to autonomous driving mode is necessary based on collected data and evaluation results and notifies the driver.

[1098] "Means for providing warnings and advice for safe driving" refers to devices or software that provide instructions or warnings to drivers for safe driving based on data analyzed in real time.

[1099] "Means for collecting user feedback" refers to an interface through which drivers can input their opinions and thoughts about their driving experience, and a system for collecting that data.

[1100] A "means for improving the system" is a device or software that analyzes the collected feedback and adjusts or refines the operating parameters of the system.

[1101] MODE FOR CARRYING OUT THE INVENTION

[1102] This invention provides a specific method for an automated driving assistance system that evaluates the driver's fatigue level and driving skill and suggests transitioning to automated driving mode at an appropriate time. It also aims to enhance driver safety and comfort by collecting user feedback and continuously improving the system.

[1103] System configuration

[1104] The main components of the automated driving assistance system are a server, a terminal, and a user interface. The role and operation of each component are explained below.

[1105] Server Roles

[1106] The server plays a central role in the entire system, specifically implementing the following functions:

[1107] System initialization:

[1108] The server initializes the system, including the driver's current fatigue level, driving skill, collected sensor data, and past feedback.

[1109] Data Processing:

[1110] The server analyzes the collected sensor data and evaluates the driver's level of fatigue and driving skills.

[1111] System Improvements:

[1112] The server analyzes user feedback and adjusts the system's operating parameters.

[1113] Device Role

[1114] The terminal collects and analyzes data in real time from various sensors installed in the vehicle. Specific functions are as follows:

[1115] Sensor data collection:

[1116] The device collects data from sensors installed in the vehicle, including the vehicle's speed, lane position, distance to the vehicle ahead, and a driver fatigue index based on facial recognition.

[1117] Driver Condition Rating:

[1118] The device evaluates the driver's level of fatigue and driving skills based on the data collected.

[1119] Autonomous driving mode suggestions:

[1120] The device determines whether it is necessary to switch to autonomous driving mode and notifies the driver.

[1121] Safe driving support:

[1122] The device analyzes the data in real time and issues warnings to the driver as needed.

[1123] User Interface

[1124] The user interface is primarily responsible for:

[1125] Providing feedback:

[1126] The system provides an interface that allows users to easily provide feedback, such as specific opinions like "too extreme" or "too slow."

[1127] Notification function:

[1128] When suggesting transition to autonomous driving mode, audio and visual notifications will be provided.

[1129] Hardware and software used

[1130] Hardware:

[1131] Various sensors (speed sensors, lane position sensors, distance sensors, face recognition cameras, etc.)

[1132] On-board computer (terminal)

[1133] server

[1134] software:

[1135] Data Analysis Software

[1136] Feedback Analysis Algorithm

[1137] Notification and warning interface program

[1138] Specific examples

[1139] Driver fatigue assessment and autonomous driving mode proposal

[1140] As a specific example, consider a driver who continues to drive for long periods of time. The device evaluates the driver's fatigue level in real time, and if it exceeds a certain level, a voice notification will be issued asking, "Do you want to switch the vehicle to autonomous driving mode?" At this point, the driver can choose to switch to autonomous driving mode via the voice notification.

[1141] Safe driving support function

[1142] If the distance to the vehicle ahead becomes too short while driving, the device will issue a warning saying, "You are getting too close to the vehicle ahead," prompting the driver to take immediate action to increase their distance.

[1143] Gathering feedback and improving the system

[1144] After completing the drive, the user provides feedback such as "My lane change was a bit extreme." The server analyzes this feedback and adjusts the system's operating parameters for the next drive, improving the user experience.

[1145] Prompt Sentence Examples

[1146] Examples of prompts to enter into a generative AI model include:

[1147] Prompt: Describe the specific steps for fatigue assessment and feedback processing in an automated driving assistance system.

[1148] Objective: Learn more about how to assess driver fatigue in real time and suggest automated driving modes at the right time.

[1149] With such settings, the automatic driving assistance system of the present invention can improve the safety and comfort of the driver.

[1150] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1151] The flow of this system's program processing

[1152] Step 1:

[1153] System initialization

[1154] The server initializes the system. As input, it obtains the driver's current fatigue level, driving skill, collected sensor data, and past feedback data from the database. Data processing generates a driver profile and initializes driving tendencies. As output, it obtains an initialized system and an initial driver profile. Specifically, the server reads driving history data from the database and prepares for a new driving session.

[1155] Step 2:

[1156] Sensor data collection

[1157] The terminal collects data in real time from various sensors attached to the vehicle. The input includes data obtained from speed sensors, lane position sensors, distance sensors, face recognition cameras, etc. Data processing involves storing each sensor data in a buffer and converting it into the required data format. As an output, real-time data from the sensors is stored in the buffer. Specifically, the terminal collects the latest driving information every second and sends that information to the server in real time.

[1158] Step 3:

[1159] Driver condition assessment

[1160] The driver's condition is evaluated based on the sensor data collected by the device. The input includes real-time sensor data stored in a buffer. For data calculation, facial recognition technology is used to analyze the opening and closing of the driver's eyes and facial movements to quantify the level of fatigue. Driving patterns are also analyzed to evaluate driving skills. The output is a driver fatigue score and driving skill score. Specifically, an AI model is used to evaluate real-time data, and the driver's fatigue level is displayed on the screen.

[1161] Step 4:

[1162] Proposal for autonomous driving mode

[1163] The device will suggest switching to autonomous driving mode based on the evaluation results. Inputs include the driver's fatigue score and driving skill score. Data calculations compare the evaluation score with preset reference values ​​to determine whether it is necessary. As an output, a notification suggesting switching to autonomous driving mode is generated for the driver. Specifically, the device will issue a voice notification asking, "Do you want to switch the vehicle to autonomous driving mode?"

[1164] Step 5:

[1165] Safe driving support

[1166] The device analyzes sensor data in real time and provides warnings and advice for safe driving as needed. The input includes continuously collected sensor data. Data processing monitors the distance to the vehicle ahead and lane departure, generating a warning if certain criteria are exceeded. The output generates audio and visual warnings for the driver. Specifically, the device displays a warning message such as "You are getting too close to the vehicle ahead."

[1167] Step 6:

[1168] Gathering feedback

[1169] The user provides feedback on the driving experience. The input includes feedback data provided through the user interface. Data processing classifies the feedback and stores it as analysis data for system improvement. The output is the collected feedback stored on the server. Specifically, the user enters "The lane change was a bit extreme" into the smartphone app.

[1170] Step 7:

[1171] System Improvements

[1172] The server improves the system based on the collected feedback. Inputs include user feedback and past driving data. Data calculations involve analyzing the feedback, reassessing the driver's driving skills, and adjusting the system's operating parameters. The output is improved system settings. Specifically, the server analyzes the feedback and configures the system to operate with the new parameter set the next time the vehicle is driven.

[1173] In this way, automated driving assistance systems can increase driver safety and comfort.

[1174] (Application example 1)

[1175] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1176] Current autonomous driving technology does not adequately assess driver fatigue and driving skills, making it difficult to fully guarantee driver safety and comfort. Furthermore, there is a lack of mechanisms for detecting driver fatigue and loss of concentration in real time during long periods of driving and prompting the vehicle to transition to autonomous driving mode at the appropriate time. Furthermore, existing systems lack sufficient processes for collecting user feedback and improving the system, resulting in low reliability and social acceptance. Therefore, there is a need for an autonomous driving assistance system that can be continuously improved while achieving both safety and comfort.

[1177] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1178] In this invention, the server includes a means for assessing the driver's fatigue level, a means for collecting and analyzing sensor data, a means for assessing the driver's driving skill, a means for proposing transition to autonomous driving mode, a means for collecting user feedback and improving the system, and a means for providing visual and audio warnings to support the driver's safe driving. This makes it possible to evaluate the driver's condition in real time and suggest transitioning to autonomous driving mode at an appropriate time. It also supports safe driving by notifying the driver of dangerous situations while driving with visual and audio warnings. Furthermore, by collecting user feedback and continuously improving the system based on that feedback, it is possible to increase reliability and social acceptance.

[1179] The "means for assessing driver fatigue" refers to a method and device for quantifying and assessing the driver's fatigue level in real time based on facial recognition, eye movements, etc.

[1180] "Means for collecting and analyzing sensor data" refers to the technology and equipment that collects data such as speed, lane position, and distance to the vehicle ahead from various sensors attached to the vehicle and analyzes it.

[1181] The "means for assessing the driver's driving skills" refers to algorithms and devices for assessing the driving skills based on collected sensor data and the driver's driving history.

[1182] "Means for proposing transition to autonomous driving mode" refers to technologies and devices that notify and suggest transition to autonomous driving mode to the driver at an appropriate time based on the evaluation results.

[1183] "Means for collecting user feedback and improving the system" refers to technologies and devices that collect feedback provided by drivers based on their driving experience, analyze it, adjust system parameters, and continuously improve the system.

[1184] "Means for providing visual and audio warnings to support safe driving" refers to technologies and devices that provide visual and audio warnings to drivers based on collected sensor data and evaluation results, and encourage appropriate safe driving behavior.

[1185] The following configurations are included as embodiments of the present invention.

[1186] The server initializes the system and provides a means to evaluate the driver's fatigue level and driving skills. It uses data from various sensors attached to the vehicle (speed, lane position, distance to the vehicle ahead, etc.) to collect and analyze sensor data. The analyzed data is sent to the server, which evaluates the driver's condition in real time.

[1187] Based on the evaluation results, the device provides a means to suggest switching to autonomous driving mode. This suggestion is made using visual and audio output devices such as smart glasses. For example, if the driver's fatigue level exceeds a certain level, the device will notify the driver via audio and visual means, saying, "Your fatigue level is high. Would you like to suggest switching to autonomous driving mode?" If the driver answers "Yes" via audio, the device will smoothly switch to autonomous driving mode.

[1188] In addition, the device provides visual and audio warnings to help drivers drive safely. For example, if the distance to the vehicle ahead is significantly short, the device will warn the driver that they are driving too close to the vehicle ahead, allowing them to immediately recognize the situation and take appropriate action.

[1189] After completing a drive, users can provide feedback to the system. The feedback is sent to the server and used to improve the system. For example, specific opinions such as "Your lane changes were a bit extreme" can be reflected and the system's operating parameters will be adjusted for the next drive. This allows the system to continuously improve, increasing its reliability and social acceptance.

[1190] Description of the hardware and software used

[1191] Hardware: Smart glasses (e.g., Microsoft HoloLens, Google Glass), various in-vehicle sensors

[1192] Software: Driver state assessment algorithm, smart glasses SDK (e.g. GoggleSDK)

[1193] Data processing and calculation

[1194] The data is processed as follows:

[1195] 1. Sensor data collection:

[1196] Data is collected from various sensors installed in the vehicle.

[1197] 2. Driver Condition Assessment:

[1198] Based on the collected sensor data, algorithms are applied to assess fatigue levels and driving skills.

[1199] 3. Autonomous driving mode suggestions and warnings:

[1200] -Based on the evaluation results, the device will suggest transitioning to autonomous driving mode and provide visual and audio warnings.

[1201] Examples of concrete examples and prompts

[1202] Specific examples

[1203] When a driver is driving for a long period of time, the smart glasses monitor the driver's eye movements and facial expressions in real time to detect an increase in fatigue. As a result, a notification will appear on the smart glasses' HUD saying, "Fatigue level is high. Would you like to suggest switching to autonomous driving mode?" If the driver answers "Yes" aloud, the smart glasses will automatically switch to autonomous driving mode.

[1204] Prompt Sentence Examples

[1205] Consider a scenario in which smart glasses monitor the eye movements and facial expressions of a driver in real time while driving for long periods of time to detect increased fatigue, and explain the appropriate message display and voice response procedures.

[1206] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1207] Step 1:

[1208] System initialization

[1209] The server initializes the system. The inputs are the driver's current fatigue level, driving skill, collected sensor data, and past feedback. Based on this, all initial parameters are set. The output is the initialized system state. Specifically, the server resets all parameters and performs the initial settings.

[1210] Step 2:

[1211] Sensor data collection

[1212] The terminal collects data from various sensors attached to the vehicle. The input is real-time data such as the vehicle's speed, lane position, and distance to the vehicle ahead. This data is collected by the sensors and sent to the terminal. The output is a set of collected sensor data. Specifically, the terminal collects sensor data at regular intervals and stores it for analysis.

[1213] Step 3:

[1214] Driver condition assessment

[1215] The device evaluates the driver's fatigue level and driving skills based on the sensor data collected. The sensor data collected in step 2 is used as input. A data analysis algorithm quantifies the fatigue level and evaluates the driving skills. The output is the quantified fatigue level and driving skill evaluation results. Specifically, the device executes the data analysis algorithm and calculates the evaluation results.

[1216] Step 4:

[1217] Proposal for autonomous driving mode

[1218] Based on the evaluation results, the device will suggest switching to autonomous driving mode. The inputs are the fatigue level and driving skill evaluation results obtained in step 3. When the fatigue level exceeds a certain value, the device will output a suggestion notification. Specifically, it will display a message on the smart glasses' HUD and notify the driver by voice, "Your fatigue level is high. Would you like to suggest switching to autonomous driving mode?"

[1219] Step 5:

[1220] Safe driving support

[1221] The device analyzes sensor data in real time and provides warnings and advice for safe driving. The input is sensor data and the driver's state evaluation results. The output is visual and audio warning and advice messages. Specifically, the device monitors the distance to the vehicle ahead and warns the driver that "you are too close to the vehicle ahead" if the distance is too close.

[1222] Step 6:

[1223] Gathering feedback

[1224] The user provides feedback about the driving experience. The input is the specific feedback provided by the driver, including their opinions and thoughts about the system's behavior. The output is the collected feedback data. The specific operation is that the terminal receives the feedback from the user and sends it to the server.

[1225] Step 7:

[1226] System Improvements

[1227] The server adjusts and improves the system's operating parameters based on the collected feedback. The input is the feedback data collected in step 6. The feedback is analyzed and necessary adjustments are made. The output is the improved system's operating parameters. Specifically, the server runs the feedback analysis algorithm and updates the system settings.

[1228] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1229] This invention combines an emotion engine with an automated driving assistance system to improve the effectiveness of driver fatigue assessment and feedback collection, and to improve the accuracy of suggestions for switching to automated driving mode. The emotion engine recognizes emotions from the user's tone of voice, facial expressions, body movements, etc., and reflects them in the system.

[1230] Program processing

[1231] 1. System initialization

[1232] The server initializes the system. When creating an instance of UrbanCohesionDriveAssist, it initializes the driver's fatigue level, driving skill, sensor data, emotional data, and feedback.

[1233] 2. Collecting sensor data and emotion data

[1234] The device calls the collect_sensor_data method to collect data from the vehicle's sensors and the emotion engine that recognizes the user's emotions. The collected data includes speed, lane position, distance to the vehicle ahead, and the user's emotional state (e.g., stress level, distraction level, etc.).

[1235] 3. Driver condition assessment

[1236] The device calls the evaluate_driver_state method to comprehensively evaluate the driver's fatigue level based on emotional data and driving data, taking into account, for example, whether the driver is not only tired but also in a high-stress state.

[1237] 4. Proposal for autonomous driving mode

[1238] The device calls the evaluate_and_propose_autonomous_mode method and suggests switching to autonomous driving mode if the driver's state or emotional data exceeds a certain threshold. Specifically, the device encourages switching to autonomous driving mode even if the driver is feeling highly stressed based on emotional data.

[1239] 5. Safe driving support

[1240] The device analyzes sensor data and emotional data to provide safe driving support. For example, it not only issues a warning when the distance to the vehicle ahead is short, but also considers the causes of driver distraction based on emotional data and provides warnings and advice based on that.

[1241] 6. Gathering Feedback

[1242] When a user provides feedback using the collect_feedback method, we also collect emotional data, such as tone of voice and facial expressions, to assess the user's true perception of the quality of the experience.

[1243] 7. System Improvements

[1244] The server calls the improve_system_based_on_feedback method to analyze the feedback and emotional data. Based on this, the system can be improved. For example, if the user perceives the system as "extreme," and the emotional data indicates a stress response, the server can adjust the driver's skill level or the system's response speed.

[1245] Specific examples

[1246] Next, a specific example of the present invention will be shown. For example, when a driver has been driving for a long time, the system evaluates the driver's fatigue level in real time. This evaluation includes emotional data such as the driver's pale complexion or nervous expression. If the fatigue level exceeds a certain threshold, the device will issue a voice notification saying, "You seem tired. Would you like to switch to autonomous driving mode?" and suggest a smooth transition to autonomous driving mode.

[1247] Additionally, if the distance to the vehicle ahead is significantly short while driving, the sensor data will detect this information, and the emotion engine will sense anxiety from the driver's facial expression. Based on this information, the device will issue a warning to the driver saying, "You are getting very close to the vehicle ahead. Please be careful."

[1248] Furthermore, when a user provides feedback after completing a drive, such as "My lane change was a bit extreme," the emotion engine analyzes the user's tone of voice and facial expression when providing the feedback. From these results, it is determined that the user truly felt the lane change was extreme, and the server adjusts the system parameters for the next drive. For example, it may take measures such as reducing the speed when changing lanes.

[1249] In this way, by using an autonomous driving assistance system that combines an emotion engine, it becomes possible to provide advanced assistance that takes into account the driver's state and emotions, thereby improving safety and comfort.

[1250] The processing flow will be explained below.

[1251] Step 1:

[1252] The server initializes the system. When creating an instance of UrbanCohesionDriveAssist, it sets the driver's fatigue level to 0 and their driving skill to intermediate level (5). It also initializes the sensor data, emotion data, and feedback lists as empty.

[1253] Step 2:

[1254] The terminal calls the collect_sensor_data method to collect data from various sensors installed in the vehicle. The collected data includes speed, lane position, and distance to the vehicle ahead. This data is randomly generated and added to the sensor data list.

[1255] Step 3:

[1256] The device uses an emotion engine to collect emotional data from the driver's facial expressions, tone of voice, and body movements, for example, assessing whether the driver is tired or stressed and adding this to the emotional data list.

[1257] Step 4:

[1258] The device calls the evaluate_driver_state method to evaluate the driver's fatigue level based on the collected sensor data and emotional data. The fatigue level is calculated by an integrated analysis of the sensor data and emotional data. For example, if the driver's attention is distracted while driving and their facial expression indicates fatigue, the fatigue level is set high.

[1259] Step 5:

[1260] The device calls the evaluate_and_propose_autonomous_mode method to propose switching to autonomous driving mode if the driver's fatigue level or emotion data exceeds a certain threshold. If the proposal is accepted, the switch_to_autonomous_mode method is called to switch to autonomous driving mode.

[1261] Step 6:

[1262] The device calls the provide_safety_support method, analyzes sensor data and emotion data, and provides safe driving support. For example, if the distance to the vehicle ahead is too close and the emotion data indicates that the driver is nervous, the device will warn, "You are getting too close to the vehicle ahead. Please be careful."

[1263] Step 7:

[1264] Users provide feedback using the collect_feedback method. The feedback includes specific opinions about the driving experience, as well as emotional data at the time of the feedback. For example, the user's opinion that "the system's behavior was too aggressive" is recorded, along with the user's tone of voice and facial expression.

[1265] Step 8:

[1266] The server calls the improve_system_based_on_feedback method to analyze the collected feedback and emotion data. Based on the analysis results, the driver's driving skills are reevaluated and the system's operating parameters are adjusted. For example, if the feedback indicates that the driver's driving skills are "too aggressive," the system may adjust the lane change speed accordingly.

[1267] Example 2

[1268] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1269] While conventional automated driving assistance systems can partially take into account the driver's level of fatigue and driving skill, it is difficult to provide safe driving assistance that fully reflects the driver's emotional state or to improve the system based on feedback. Furthermore, emotional data is not sufficiently taken into account when maintaining a safe distance from the vehicle ahead or analyzing surrounding traffic conditions, leaving issues with safety and comfort. Therefore, there is a need for a system that can evaluate the driver's emotions in real time and provide appropriate assistance and suggestions.

[1270] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for evaluating the driver's fatigue level, a means for collecting and analyzing sensor data and emotional data, a means for evaluating the driver's driving skill and emotional state, a means for suggesting transition to an autonomous driving mode based on the driver's state and emotional data, and a means for collecting feedback from the user and analyzing the emotional data at the time of feedback to improve the system. This enables advanced support and suggestions that take the driver's emotional state into consideration, thereby improving safety and comfort.

[1271] "Fatigue level" refers to the degree of physical and mental fatigue a driver experiences while driving.

[1272] "Sensor data" refers to information collected by various sensors attached to a vehicle, and specifically includes data related to driving, such as speed, lane position, and distance to the vehicle ahead.

[1273] "Emotional data" refers to information about a driver's emotional state analyzed from their tone of voice, facial expressions, body movements, etc., and specifically includes stress levels and distractibility.

[1274] "Driving skill" is an evaluation of the driver's level of proficiency and skill in driving.

[1275] "Autonomous driving mode" is a mode in which the vehicle can operate autonomously without driver operation.

[1276] "Feedback" refers to opinions and evaluations provided by users to the system, and includes information that can be used to improve the system.

[1277] An "autonomous driving assistance system" is a system that assists drivers in driving and improves safety and comfort, and includes fatigue assessment, collection and analysis of sensor data and emotional data, driving skill assessment, autonomous driving mode proposals, feedback collection, and system improvement.

[1278] A "warning" is a notification or alert that notifies the driver of an impending hazard.

[1279] "Safe driving support" refers to advice and assistance provided to drivers to help them drive safely, and specifically includes maintaining a safe distance from the vehicle ahead and warning against dangerous driving.

[1280] MODE FOR CARRYING OUT THE INVENTION

[1281] This invention combines an emotion engine with an automated driving assistance system to improve the effectiveness of driver fatigue assessment and feedback collection, and to improve the accuracy of suggestions for switching to automated driving mode. The emotion engine recognizes emotions from the user's tone of voice, facial expressions, body movements, etc., and reflects them in the system.

[1282] System initialization

[1283] The server creates an instance of UrbanCohesionDriveAssist when the system is initialized. This instance is initialized with the driver's fatigue level, driving skill, sensor data, emotional data, and feedback. During initialization, the necessary data is synchronized using an existing database or cloud server.

[1284] Data collection

[1285] The device calls the collect_sensor_data method to collect data such as speed, lane position, and distance to the vehicle ahead from sensors installed in the vehicle. At the same time, the emotion engine analyzes the user's tone of voice, facial expressions, and body movements to collect emotional data such as stress level and distraction level. This provides both sensor data and emotional data to the system in real time.

[1286] Driver condition assessment

[1287] The device uses the evaluate_driver_state method to evaluate the driver's fatigue level based on the collected sensor data and emotional data. For example, a pale complexion and distractedness after a long drive could be used as evaluation points. The device also evaluates the driver's stress level based on the driver's tone of voice and facial expression, and combines this with the fatigue level evaluation to provide a comprehensive state evaluation.

[1288] Proposal for autonomous driving mode

[1289] The device calls the evaluate_and_propose_autonomous_mode method and proposes switching to autonomous driving mode if the driver's condition or emotional data exceeds a certain threshold. For example, if the user's facial expression indicates extreme fatigue, the device will issue a voice notification asking, "Do you want to switch to autonomous driving mode?"

[1290] Safe driving support

[1291] If sensor data indicates that the distance to the vehicle ahead is dangerously close, the device will issue a warning saying, "You are approaching the vehicle in front. Please be careful." If emotional data detects that the driver is anxious or distracted, the device will provide specific advice based on that emotional data, such as, "Take a deep breath and relax."

[1292] Gathering feedback and improving the system

[1293] When a user provides feedback after driving using the collect_feedback method, the emotion engine analyzes the tone of voice and facial expressions to evaluate the quality of the feedback. Based on this result, the server uses the improve_system_based_on_feedback method to improve the system. For example, taking into account past feedback and emotion data, the server can improve safety by reducing the lane-changing speed in the next driving session.

[1294] Specific examples

[1295] For example, if a driver has been driving for a long time, the server initializes the system and the device calls the appropriate method. The device uses the collect_sensor_data method to collect the vehicle's speed and distance from the vehicle ahead, and analyzes the driver's facial expression and tone of voice using the emotion engine. The emotion data and driving data are combined to evaluate the driver's condition. If the evaluation results indicate that the driver is fatigued and not paying attention, the device will suggest, "Do you want to switch to autonomous driving mode?" If the driver is dangerously close to the vehicle ahead, the device will warn, "You are very close to the vehicle ahead. Please be careful." After completing the drive, if the user provides feedback such as "My lane change was a bit aggressive," the emotion engine analyzes the user's tone of voice and facial expression to identify the specific reason why they thought the lane change was aggressive. The server uses this data to adjust the system parameters and reduce the lane change speed on the next drive.

[1296] An example of a prompt for a generative AI model is:

[1297] Please explain in natural language how the system detects the user's fatigue after a long drive and what suggestions it makes in response.

[1298] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1299] Step 1: Initialize the system

[1300] The server creates an instance of UrbanCohesionDriveAssist, which is initialized with the driver's fatigue level, driving skill, sensor data, emotional data, and feedback. The input is the initial setting data stored in an existing database or cloud server, and the output is the initialized system instance.

[1301] Step 2: Collecting sensor and emotion data

[1302] The device uses sensors installed in the vehicle and an emotion engine to collect driving-related sensor data and driver emotion data. Specifically, it calls the collect_sensor_data method to collect data such as speed, lane position, and distance to the vehicle ahead. It also analyzes stress levels and distraction levels from tone of voice, facial expressions, and body movements through the emotion engine. The inputs are the vehicle's sensor information and raw data from the emotion engine, and the output is analyzed sensor data and emotion data.

[1303] Step 3: Driver condition assessment

[1304] The device analyzes sensor data and emotion data to evaluate the driver's fatigue and stress levels. Specifically, it calls the evaluate_driver_state method to evaluate the facial expression and level of distraction of a driver who has been driving for a long time. The input is the collected sensor data and emotion data, and the output is the evaluation result, which includes the driver's fatigue and stress levels.

[1305] Step 4: Proposing an autonomous driving mode

[1306] The device proposes transitioning to autonomous driving mode based on the results of the driver's state evaluation. Specifically, it calls the evaluate_and_propose_autonomous_mode method, and if the driver's fatigue or stress level exceeds a certain threshold, it issues a voice notification suggesting, "Do you want to switch to autonomous driving mode?" The input is the driver's state evaluation result, and the output is a notification proposing transition to autonomous driving mode.

[1307] Step 5: Safe driving support

[1308] The device provides safe driving support to the driver based on the collected sensor data and emotional data. Specifically, it evaluates the distance to the vehicle ahead and issues a warning such as, "You are very close to the vehicle ahead. Please be careful." It also detects anxiety or distraction in the driver from emotional data and provides advice such as, "Take a deep breath and relax." The input is sensor data and emotional data, and the output is specific warnings and advice.

[1309] Step 6: Gather feedback

[1310] The user provides feedback after driving. The device uses the collect_feedback method to analyze the feedback along with the tone of voice and facial expression. The emotion engine is used to collect emotion data when providing the feedback and save the analysis results. The input is the user's feedback and emotion data, and the output is the analyzed feedback data.

[1311] Step 7: Improve your system

[1312] The server analyzes the collected feedback and emotion data and makes improvements to the system. Specifically, it calls the improve_system_based_on_feedback method to adjust system parameters based on the analyzed data. For example, it takes into account past feedback and emotion data and reduces the lane-changing speed in the next driving session. The input is the analyzed feedback data, and the output is the adjusted system parameters.

[1313] (Application example 2)

[1314] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1315] While conventional automated driving assistance systems can evaluate the driver's level of fatigue and driving skill, they do not adequately suggest switching to automated driving or provide safe driving support that accurately reflects the driver's emotional state. This means that the system is unable to respond quickly to sudden changes in the driver's physical condition or emotional fluctuations, which could compromise the safety and comfort of automated driving.

[1316] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for evaluating the driver's fatigue level, a means for collecting and analyzing sensor data, voice tone, and facial expressions, a means for evaluating the driver's driving skill and emotional state, a means for proposing transition to autonomous driving mode, and a means for collecting feedback from the user and improving the system. This makes it possible to evaluate the driver's state of fatigue, stress, and distraction in real time and propose transition to autonomous driving mode at an appropriate time.

[1317] "Driver fatigue level" refers to the degree of physical or mental fatigue a driver feels while driving.

[1318] "Sensor data" refers to information collected from sensors attached to vehicles and related equipment, and includes physical data such as speed, position, and distance.

[1319] "Voice tone" refers to characteristics of the driver's voice, such as pitch, strength, and intonation, and is information used to estimate the driver's emotional state.

[1320] "Facial expression" refers to the movement and changes of the driver's facial muscles, and is information used to analyze the driver's emotional state.

[1321] "Driving skills" refers to a driver's ability to operate a vehicle safely and efficiently, and its assessment includes the driver's experience and skill.

[1322] "Emotional state" refers to the driver's psychological state, including states such as stress, distraction, and relaxation.

[1323] "Autonomous driving mode" refers to a mode in which the vehicle drives automatically without driver intervention, aiming to improve safety and convenience.

[1324] "Feedback" refers to opinions and impressions from users, providing information about the performance and usability of the system.

[1325] "System Improvement" refers to actions taken to improve the performance and functionality of a system based on collected feedback and data.

[1326] This invention combines an emotion engine with an automated driving assistance system to evaluate the driver's fatigue level, driving skill, and emotional state, and proposes transitioning to automated driving mode at the appropriate time. The system uses smart glasses and other sensor devices to grasp the driver's emotional state in real time, and collects and analyzes feedback.

[1327] System configuration

[1328] Hardware:

[1329] Smart glasses: Equipped with a camera and microphone to collect the driver's facial expressions and voice.

[1330] Sensors installed in the vehicle collect driving data such as speed, lane position, and distance to the vehicle ahead.

[1331] software:

[1332] Emotion Engine: A model for analyzing a driver's emotional state from their facial expressions, vocal tone, and body movements.

[1333] Autonomous driving assistance software: A program that integrates sensor data and emotional data to assess the driver's condition and suggest autonomous driving modes.

[1334] System Operation

[1335] 1. Data Collection:

[1336] The device collects the driver's facial expressions and voice through the smart glasses' camera and microphone, and also acquires data such as speed, location, and distance to the vehicle ahead from sensors installed in the vehicle.

[1337] 2. Emotional state assessment:

[1338] The collected data is analyzed by an on-board emotion engine, which assesses the driver's fatigue, stress level and distraction.

[1339] 3. Driver Condition Assessment:

[1340] The device comprehensively evaluates the driver's emotional and driving data to analyze their fatigue and stress levels, and based on this evaluation, the driver's attention and driving skills are also taken into account.

[1341] 4. Proposal for autonomous driving mode:

[1342] If the driver's condition exceeds a certain threshold, the device will suggest transitioning to autonomous driving mode and notify the driver through audio and visual feedback.

[1343] 5. Safe driving support:

[1344] Based on sensor data and emotional data, the device provides safe driving support, for example by issuing a warning when the distance to the vehicle ahead is short or when the driver's stress level is high.

[1345] 6. Gathering feedback and improving the system:

[1346] The user provides feedback after completing a drive, and the emotion engine analyzes the user's emotional state. The server then uses this feedback to improve the system and apply it to the next drive.

[1347] Specific examples

[1348] During a long drive, the smart glasses suggest, "You seem tired. Would you like to switch to autonomous driving mode?" At this time, it is clear that the driver is not only tired, but also stressed from their facial expressions. Furthermore, when providing feedback after the drive that "your lane change was a bit extreme," the tone of voice and facial expressions at the time are analyzed to determine whether that was really how they felt, and the system will adjust accordingly for the next drive.

[1349] Prompt Sentence Examples

[1350] "You're driving a long distance. Your smart glasses might say: 'You're getting very close to the vehicle ahead. Be careful.' They might also suggest: 'You seem tired. Would you like to shift into self-driving mode?' If you provide feedback like 'You're changing lanes too aggressively,' the system will use this feedback and your emotional data to make adjustments the next time you drive."

[1351] In this way, by implementing the present invention, advanced automated driving assistance that takes into account the emotional state of the driver can be realized.

[1352] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1353] Step 1:

[1354] System initialization

[1355] The server initializes the system and initializes the driver's fatigue level, driving skill, sensor data, emotional data, and feedback. It uses the system's initial settings and past data as input, and generates an initial database as output. This process prepares the system for data collection and analysis.

[1356] Step 2:

[1357] Sensor and emotion data collection

[1358] The device collects data from sensors installed in the smart glasses and the vehicle. Specifically, the smart glasses' camera captures the driver's facial expressions, the microphone records audio, and the vehicle sensors collect driving data such as speed, position, and distance between vehicles. The device uses real-time data from the sensors as input and receives and stores all of this data in a consistent format as output.

[1359] Step 3:

[1360] Driver condition assessment

[1361] The device analyzes the collected sensor data and emotional data. Specifically, it uses an emotion engine to evaluate the driver's fatigue and stress levels based on their facial expressions and voice. The input is the data collected in the previous step, and the output is an evaluation of the driver's fatigue and emotional state. This evaluation allows the driver's current physical and mental state to be understood.

[1362] Step 4:

[1363] Proposal for autonomous driving mode

[1364] Based on the evaluation results, the device will suggest switching to autonomous driving mode. Specifically, if the evaluation exceeds a certain threshold, the device will notify the driver audibly and visually, saying, "You seem tired. Would you like to switch to autonomous driving mode?" The device uses the evaluation results of fatigue level and emotional state as input, and generates and displays a suggestion message as output.

[1365] Step 5:

[1366] Safe driving support

[1367] The device analyzes sensor data and emotional data in real time to provide safe driving support. Specifically, it issues a warning when the distance to the vehicle ahead is close or the driver is under high stress. It uses sensor data and emotional data as input and provides warning messages and advice in audio and visual formats as output.

[1368] Step 6:

[1369] Gathering feedback

[1370] The user provides feedback after completing the drive. The device collects this feedback data and analyzes the tone of voice and facial expressions at the time of providing the feedback using an emotion engine. The user's feedback and emotional data at that time are used as input, and the analysis results of the feedback are obtained as output.

[1371] Step 7:

[1372] System Improvements

[1373] The server analyzes the collected feedback and emotion data and adjusts the system parameters accordingly, specifically adjusting the system's response based on the feedback and applying it to the next drive. The server uses the feedback analysis results as input and obtains improved system settings as output.

[1374] This series of steps enables the driver's emotional state to be grasped in real time, allowing appropriate automated driving suggestions and safe driving support to be realized.

[1375] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1376] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1377] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1378] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1379] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1380] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1381] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1382] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1383] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1384] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1385] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1386] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1387] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1388] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1389] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1390] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1391] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1392] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1393] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1394] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1395] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1396] The following is further disclosed regarding the above embodiment.

[1397] (Claim 1)

[1398] a means for assessing driver fatigue;

[1399] a means for collecting and analyzing sensor data;

[1400] a means for assessing the driver's driving skills;

[1401] A means of proposing a transition to autonomous driving mode;

[1402] a means of collecting user feedback and improving the system;

[1403] Autonomous driving assistance systems, including

[1404] (Claim 2)

[1405] 2. The automated driving assistance system of claim 1, wherein the system evaluates the distance to a vehicle ahead and provides a warning to maintain a safe distance.

[1406] (Claim 3)

[1407] 2. The automated driving assistance system according to claim 1, which analyzes surrounding traffic conditions based on collected sensor data and provides safe driving support.

[1408] "Example 1"

[1409] (Claim 1)

[1410] a means for assessing driver fatigue;

[1411] A means of collecting sensor data from various sensors and analyzing it in real time,

[1412] a means for assessing the driver's driving skills;

[1413] A means of determining whether or not a transition to autonomous driving mode is necessary and making a proposal;

[1414] A means to provide warnings and advice for safe driving as needed based on collected sensor data;

[1415] a means for collecting feedback from users and improving the system based on that feedback;

[1416] A system including:

[1417] (Claim 2)

[1418] 10. The system of claim 1, wherein the system provides a warning when the vehicle ahead is too close.

[1419] (Claim 3)

[1420] The system of claim 1 notifies the driver of the transition to autonomous driving mode via a voice or visual interface based on the collected sensor data.

[1421] "Application Example 1"

[1422] (Claim 1)

[1423] a means for assessing driver fatigue;

[1424] a means for collecting and analyzing sensor data;

[1425] a means for assessing the driver's driving skills;

[1426] A means of proposing a transition to autonomous driving mode;

[1427] a means of collecting user feedback and improving the system;

[1428] A means of providing visual and audio warnings to assist drivers in driving safely;

[1429] A system including:

[1430] (Claim 2)

[1431] 10. The system of claim 1, wherein the system assesses the distance to a vehicle ahead and provides a warning to maintain a safe following distance.

[1432] (Claim 3)

[1433] The system according to claim 1, which analyzes surrounding traffic conditions based on collected sensor data and provides safe driving support.

[1434] "Example 2: Combining Emotion Engines"

[1435] (Claim 1)

[1436] a means for assessing driver fatigue;

[1437] means for collecting and analyzing sensor data and emotion data;

[1438] a means for assessing the driver's driving skills and emotional state;

[1439] A means for suggesting transition to autonomous driving mode based on driver state and emotion data; and

[1440] A means for collecting feedback from users and analyzing emotional data at the time of feedback to improve the system;

[1441] A system including:

[1442] (Claim 2)

[1443] 10. The system of claim 1, wherein the system evaluates the distance to the vehicle ahead, provides warnings to maintain a safe distance, and provides advice to the driver based on emotion data.

[1444] (Claim 3)

[1445] The system according to claim 1, wherein the system analyzes surrounding traffic conditions and the driver's emotional state based on the collected sensor data and emotional data, and provides safe driving support.

[1446] "Application example 2 when combining emotion engines"

[1447] (Claim 1)

[1448] a means for assessing driver fatigue;

[1449] A means of collecting and analyzing sensor data, voice tone, and facial expressions;

[1450] a means for assessing the driver's driving skills and emotional state;

[1451] A means of proposing a transition to autonomous driving mode;

[1452] a means of collecting user feedback and improving the system;

[1453] A system including:

[1454] (Claim 2)

[1455] 10. The system of claim 1, wherein the system assesses the distance to a vehicle ahead and provides a warning to maintain a safe following distance.

[1456] (Claim 3)

[1457] The system according to claim 1, which analyzes surrounding traffic conditions and provides safe driving support based on collected sensor data and emotion data. [Explanation of symbols]

[1458] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for assessing driver fatigue; a means for collecting and analyzing sensor data; a means for assessing the driver's driving skills; A means of proposing a transition to autonomous driving mode; a means of collecting user feedback and improving the system; Autonomous driving assistance systems, including

2. 10. The automated driving assistance system of claim 1, wherein the system evaluates the distance to a vehicle ahead and provides a warning to maintain a safe distance.

3. The automated driving assistance system according to claim 1, wherein the system analyzes surrounding traffic conditions based on collected sensor data and provides safe driving support.

Images