System

Abstract

To provide a system for accurately determining a fatigue degree of a driver and urging the driver to take a rest.SOLUTION: The system further includes means for collecting information on the surroundings of the driver and the vehicle, means for using a generation AI model that analyzes the information in real time, means for notifying the driver based on the analysis result, means for analyzing facial expressions and blinks of the driver to determine a degree of fatigue, and means for notifying the driver of a rest according to the degree of fatigue, means for setting a destination and a waypoint on the way of the driver by voice, means for acquiring real-time traffic congestion information and calculating an optimal route, and means for guiding the driver by voice and display about the calculated route.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional driving assistance systems have difficulty in grasping information about surrounding pedestrians, motorbikes, and bicycles in real time and providing appropriate notifications for safe driving. Furthermore, they lack the ability to accurately assess the driver's fatigue level and encourage rest, which increases the risk of accidents during long drives. Furthermore, they do not provide optimal route guidance that takes into account traffic congestion and traffic light information, which reduces driving efficiency. [Means for solving the problem]

[0005] The present invention solves these problems by providing a system that includes a means for collecting information about the driver and the vehicle's surroundings, a means for analyzing the information in real time using a generative AI model, a means for notifying the driver based on the analysis results, a means for analyzing the driver's facial expressions and blinking to determine the driver's fatigue level, and a means for notifying the driver to take a break based on the fatigue level. The system also includes a means for the driver to set the driver's destination and stopovers via voice, a means for acquiring real-time traffic congestion information and calculating an optimal route, and a means for providing the driver with audio and visual guidance along the calculated route, thereby improving driving efficiency and safety. Furthermore, the vehicle's surroundings information includes information about pedestrians, motorcycles, and bicycles, and a means for notifying the driver in real time via an in-vehicle display and voice assistant, thereby enabling the system to predict dangers while driving in advance and encourage appropriate responses.

[0006] "Driver" means a person who operates and drives a vehicle.

[0007] "Vehicle" refers to a means of transportation such as a car or motorcycle that is intended to travel on roads.

[0008] "Surrounding information" refers to data about objects and situations around the vehicle, including pedestrians, motorbikes, bicycles, and the status of traffic lights.

[0009] "Means of collection" refers to methods and devices for obtaining information about the surrounding area using in-vehicle cameras, sensors, etc.

[0010] "Real-time" refers to immediate processing to respond to events and changes that occur almost simultaneously.

[0011] A "generative AI model" refers to an artificial intelligence model that learns patterns and features from given data and makes predictions and classifications for new data.

[0012] "Means for analysis" refers to methods or devices that process collected data and extract meaningful information.

[0013] "Means of notification" refers to devices and methods for conveying information to the driver based on the analysis results, and specifically includes notifications via display or voice assistant.

[0014] "Facial expressions" refers to the movements of a driver's face to indicate emotion or state.

[0015] "Blinking" refers to the movement of a driver's eyelids opening and closing.

[0016] "Fatigue level" refers to the driver's level of fatigue and indicates a condition that may affect driving speed and safety.

[0017] "Rest reminder notification" refers to a method of conveying information advising the driver to take a rest depending on their level of fatigue.

[0018] "Destination" refers to the final location the driver is heading for.

[0019] "Stopover" refers to a place where you stop on the way to your destination.

[0020] "Voice setting means" refers to a method or device that allows the driver to specify the destination and stopovers by voice.

[0021] "Traffic congestion information" refers to data regarding traffic congestion on roads.

[0022] An "optimal route" refers to the route that allows a driver to reach their destination most efficiently.

[0023] "Calculation means" refers to a method or device that calculates the optimal route based on information.

[0024] "Audio and visual guidance means" refers to devices and methods for conveying calculated route information to the driver, and specifically includes audio guidance and display.

[0025] "Means for providing warnings" refers to methods or devices that notify drivers of potential dangers that may occur while driving. [Brief explanation of the drawings]

[0026] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0027] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0028] First, the terms used in the following description will be explained.

[0029] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0030] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0031] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0032] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0033] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0034] [First embodiment]

[0035] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0036] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0037] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0038] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0039] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0040] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0041] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0042] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0043] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0044] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0045] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0046] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0047] The present invention relates to a system that collects information about the driver and the vehicle's surroundings, analyzes it in real time, and notifies the driver. The system aims to improve driver safety and driving efficiency, and has functions such as judging the driver's fatigue level and encouraging them to take a break as necessary.

[0048] System Configuration

[0049] The system of the present invention consists of the following main components:

[0050] 1. Terminal: Installed in the vehicle, it collects information about the surrounding area using on-board cameras and various sensors.

[0051] 2. Server: Receives information sent from the device and analyzes it using a generative AI model.

[0052] 3. Notification mechanism: Display and voice assistant to notify the driver based on the analysis results.

[0053] Program processing

[0054] The program processing in this system is carried out in the following manner.

[0055] 1. Data Collection:

[0056] The device collects real-time data from on-board cameras and sensors, including the movement of pedestrians, motorcyclists, and cyclists, traffic light status, and the driver's facial expressions and eye blinks.

[0057] 2. Data transmission:

[0058] The terminals transmit the collected data to the server in real time in an appropriate format.

[0059] 3. Data Analysis:

[0060] The server receives the data sent from the device and analyzes it using a generative AI model. The analysis results include predictions of pedestrians running out into the road at a specific location, situations in which a motorbike is approaching suddenly, and the driver's fatigue level.

[0061] 4. Notification information generation:

[0062] Based on the analysis results, the server generates appropriate notification and advice messages for the driver, such as "There is a risk of a pedestrian jumping out into the road at the intersection 100 meters ahead" or "A motorcycle is rapidly approaching from behind."

[0063] 5. Notification Implementation:

[0064] The server then sends the created notification data to the device, which then displays it on the in-car display and notifies the driver via the voice assistant. For example, a warning may be displayed on the screen and a voice message may be heard saying, "There is a pedestrian ahead. Please be careful."

[0065] Specific examples

[0066] Scenario 1: Risk prediction notification

[0067] When a user approaches a pedestrian intersection ahead while driving, the in-car display suddenly displays the message, "A pedestrian may jump out into the road at the intersection 100 meters ahead," and at the same time, the voice assistant notifies the user, "A pedestrian may jump out into the road ahead. Please be careful," prompting the user to immediately slow down.

[0068] Scenario 2: Fatigue level assessment

[0069] If the user continues driving for a long time, the in-car camera will analyze the user's facial expressions and blinking to determine that they are fatigued. In this case, the display will show "You are fatigued. Please take a break" and a voice message will be given saying "We recommend taking a break," encouraging the user to decide to take a break.

[0070] Scenario 3: Route guidance taking traffic congestion into account

[0071] When the user voice-selects "a nearby cafe" as their destination, the server calculates the optimal route based on the latest traffic information and sends it to the device. The device then provides route guidance via voice prompts and a display: "Turn right at the next intersection, then turn left at the next traffic light."

[0072] In this way, the system of the present invention efficiently collects and analyzes information about the driver and the vehicle's surroundings, and provides notifications at the appropriate time, thereby supporting safe and comfortable driving.

[0073] The processing flow will be explained below.

[0074] Step 1:

[0075] The device collects real-time data from on-board cameras and sensors, including the movement of pedestrians, motorcyclists, and bicycles, the status of traffic lights, and the driver's facial expressions and blinking.

[0076] Step 2:

[0077] The terminal transmits the collected data to a server in real time via wireless communication in an appropriate format.

[0078] Step 3:

[0079] The server receives the data sent from the device and inputs it into the generative AI model to analyze the surrounding situation and the driver's state.

[0080] Step 4:

[0081] The server analyzes various factors, such as the approach of pedestrians, motorbikes, and bicycles, and the driver's fatigue level, to generate risk prediction data and data on the driver's condition, such as when there is a high possibility of a pedestrian jumping out into the intersection ahead or when the driver is highly fatigued.

[0082] Step 5:

[0083] The server generates notification data based on the analysis results, such as a warning message such as "A pedestrian may jump out into the street at the intersection 100 meters ahead" or a message encouraging the driver to take a break such as "You are fatigued. Please take a break."

[0084] Step 6:

[0085] The server sends the created notification data to the terminal. The data is designed to be sent quickly.

[0086] Step 7:

[0087] The device displays the received notification data on the in-car display and notifies the driver via the voice assistant. Specifically, the display will show "There is a pedestrian ahead. Please be careful," and at the same time a voice will say, "A pedestrian has jumped out ahead. Please be careful."

[0088] Step 8:

[0089] The device continuously analyzes the driver's facial expressions and blinking to monitor their fatigue level. If it determines that the driver is fatigued, it will notify the driver, saying, "You are fatigued. Please take a break."

[0090] Step 9:

[0091] The server obtains real-time traffic congestion information from external traffic information services and calculates the optimal route to the driver's destination and intermediate stops based on the information obtained.

[0092] Step 10:

[0093] The server sends the calculated route data to the device, which receives it and starts providing guidance to the driver via voice and display, providing specific instructions such as "turn right at the next intersection."

[0094] Through these steps, the system provides support to improve driver safety and efficiency.

[0095] Example 1

[0096] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0097] To improve driver safety and driving efficiency, it is important to collect and analyze information about the driver and the vehicle's surroundings in real time and provide notifications at the appropriate time. However, conventional systems have had problems with insufficient collection and analysis of information, making it difficult to provide notifications at the appropriate time. There are also few systems that can determine the driver's fatigue level in real time and prompt them to take appropriate breaks. Furthermore, there are also insufficient systems that allow the driver to set their destination via voice input or provide optimal routes based on the latest traffic congestion information.

[0098] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0099] In this invention, the server includes means for collecting information about the driver and the surroundings of the vehicle, means for using a generative AI model to analyze the information in real time, means for notifying the driver based on the analysis results, means for analyzing the driver's facial expressions and blinking to determine the level of fatigue, means for notifying the driver to take a break based on the level of fatigue, means for transmitting the collected data to the server in an appropriate format, means for generating a notification message based on the analysis results using the generative AI model, and means for communicating the notification message to the driver via an in-vehicle display or a voice assistant. This makes it possible to improve driver safety, increase driving efficiency, and reduce the burden on the driver.

[0100] "Driver" means a person who drives a vehicle.

[0101] "Vehicle" refers to land transportation such as automobiles and motorcycles.

[0102] "Information" refers to data about the vehicle and its surroundings, such as the driver's facial expressions, blinking, the movements of pedestrians, motorbikes and bicycles, and the status of traffic lights.

[0103] A "generative AI model" refers to an artificial intelligence model that analyzes collected information in real time and generates appropriate notifications and advice for drivers.

[0104] "Notification" refers to warnings and advice given to the driver based on the analysis results.

[0105] A "break notification" refers to the transmission of a message encouraging a driver to take a break when the driver's level of fatigue is judged to be high.

[0106] "Format" refers to the standard or format for properly organizing and converting data.

[0107] An "in-vehicle display" refers to a device installed inside a vehicle that displays various information on a screen.

[0108] "Voice assistant" refers to a system that uses voice to provide notifications and advice to drivers.

[0109] MODE FOR CARRYING OUT THE INVENTION

[0110] The present invention relates to a system that collects information about the driver and the vehicle's surroundings, analyzes it in real time, and notifies the driver. The system aims to improve driver safety and driving efficiency, and has the function of judging the driver's fatigue level and encouraging them to take a break as necessary.

[0111] System Configuration

[0112] The system consists of the following main components:

[0113] 1. Device:

[0114] - The device is installed in the vehicle and collects information about the surrounding area using on-board cameras and various sensors.

[0115] - The device transmits the collected data in the appropriate format to the server in real time.

[0116] - The device displays the notification message received from the server on the in-vehicle display and notifies the driver via the voice assistant.

[0117] 2. Server:

[0118] - The server receives the information sent from the device and analyzes it using the generative AI model.

[0119] - The server generates an appropriate notification message based on the analysis results.

[0120] 3. Notification mechanism:

[0121] - The notification mechanism consists of the device display and voice assistant.

[0122] - The notification mechanism provides real-time warnings and advice to the driver.

[0123] Hardware and software used

[0124] 1. Hardware:

[0125] - In-car camera: A camera that captures the driver's facial expressions, blinking, and surrounding pedestrians and vehicles.

[0126] - Various sensors: Sensors for detecting vehicle movement and surrounding information using accelerometers, gyro sensors, LIDAR, etc.

[0127] - Terminal: A terminal installed in a vehicle that collects and transmits data.

[0128] - Display and voice assistant: Output devices for notifying the driver.

[0129] 2. Software:

[0130] - Generative AI model: An artificial intelligence model for performing data analysis on the server.

[0131] - Data transmission software: Software for transmitting data from the terminal to the server.

[0132] - Notification generation software: Software for generating notification messages based on analysis results.

[0133] Specific examples

[0134] Scenario 1: Risk prediction notification

[0135] When a user approaches a pedestrian intersection while driving, the in-car display suddenly displays the message, "A pedestrian may jump out into the intersection 100 meters ahead," and at the same time, the voice assistant notifies the user, "A pedestrian has jumped out into the intersection ahead. Please be careful." This allows the user to immediately slow down.

[0136] Prompt Sentence: Generate a message to notify the driver of dangerous situations in advance while driving. For example, "A pedestrian may jump out at the intersection 100 meters ahead."

[0137] Scenario 2: Fatigue level assessment

[0138] If the user continues driving for a long time, the in-car camera analyzes facial expressions and blinking to determine that the user is highly fatigued. In this case, the display will show "You are fatigued. Please take a break" and a voice message will be played saying "We recommend taking a break." This will encourage the user to decide to take a break.

[0139] Prompt Sentence: Analyze the fatigue level of the user who drives for a long time and generate a message to prompt appropriate rest. For example, generate "You are tired. Please take a break."

[0140] Scenario 3: Route guidance taking traffic congestion into account

[0141] When the user voice-selects "a nearby cafe" as their destination, the server calculates the optimal route based on the latest traffic information and sends it to the device. The device then provides route guidance via voice prompts and a display: "Turn right at the next intersection, then turn left at the next traffic light."

[0142] Prompt: Calculate the optimal route to reach the specified destination and generate a message containing the instructions. For example, "Turn right at the next intersection, then left at the next traffic light."

[0143] In this way, the system of the present invention efficiently collects and analyzes information about the driver and vehicle's surroundings, and provides notifications at appropriate times to support safe and comfortable driving.

[0144] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0145] Step 1: Data collection

[0146] The device collects data in real time using onboard cameras and various sensors.

[0147] Input: Information captured by the in-vehicle camera, such as the driver's facial expressions and blinking, the movements of pedestrians, motorbikes, and bicycles, and the status of traffic lights. Data on the vehicle's movements and surrounding conditions from sensors.

[0148] Data processing / calculation: Each data point is temporarily stored in a buffer and converted into an easily readable format.

[0149] Output: A collection of recorded data.

[0150] Specific operation: The device uses the onboard camera to capture the driver's facial expressions and blinks, uses accelerometers and gyro sensors to detect vehicle movement, and uses LIDAR to scan the location of surrounding objects and pedestrians.

[0151] Step 2: Send data

[0152] The terminal transmits the collected data to the server in real time.

[0153] Input: Various data stored on the device.

[0154] Data processing / computation: Converting data into an appropriate format (e.g., JSON or XML) and encrypting it.

[0155] Output: The formatted encrypted data.

[0156] Specific operation: The device converts the collected data into an appropriate format, encrypts it for security purposes, and sends it to a server over the Internet.

[0157] Step 3: Data analysis

[0158] The server receives the data sent from the device and analyzes it using a generative AI model.

[0159] Input: Encrypted data sent from the device after format conversion.

[0160] Data processing / calculation: Data is input into a generative AI model to predict when pedestrians will suddenly run out into the road, when motorcycles will suddenly approach, and determine the driver's level of fatigue.

[0161] Output: Analysis results (e.g., prediction of pedestrians running out into the road, determination of driver fatigue level, etc.).

[0162] Specific operation: The server temporarily stores the received data in a database, extracts the necessary data from the database, inputs it into the generative AI model for analysis, and predicts the movements of pedestrians and the state of the driver as a result.

[0163] Step 4: Generate notification information

[0164] The server generates a message to notify the driver based on the analysis results.

[0165] Input: Analysis results from the generative AI model.

[0166] Data processing / calculation: Based on the analysis results, an appropriate notification message is generated for the driver.

[0167] Output: The generated notification message.

[0168] Specific operation: The server generates an appropriate notification message based on the analysis results obtained by the generative AI model. For example, it creates a message such as "A pedestrian may jump out into the street at the intersection 100 meters ahead."

[0169] Step 5: Notification

[0170] The server sends the created notification data to the terminal, which then notifies the driver via the in-car display and voice assistant.

[0171] Input: The generated notification message.

[0172] Data processing / calculation: Converts the notification message into an appropriate format and sends it to the terminal.

[0173] Output: Notification messages sent to the terminal.

[0174] Specific operation: The server sends the generated notification message to the device, and the device displays the received notification data on the in-car display and notifies the driver via the voice assistant. For example, the display may display "There is a pedestrian ahead. Please be careful" and a voice message may also be issued saying "There is a pedestrian ahead. Please be careful."

[0175] In this way, through the specific operations of each step, the system can provide a safe driving environment for the driver and improve driving efficiency.

[0176] (Application example 1)

[0177] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0178] In today's modern transportation society, there is a demand for improved driver safety and driving efficiency. In particular, the risk of accidents is increasing due to fatigue from long periods of driving and inattention to surrounding traffic conditions. Therefore, technology is needed to enable drivers to continue driving safely and efficiently. Another challenge is to monitor and analyze traffic conditions and driver status in real time and provide notifications at the appropriate time.

[0179] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0180] In this invention, the server includes means for collecting information about the driver and the surrounding area of ​​the vehicle, means for using a generative AI model to analyze the information in real time, means for notifying the driver based on the analysis results, means for analyzing the driver's facial expressions and blinking to determine the level of fatigue, means for issuing a notification encouraging the driver to take a break based on the level of fatigue, means for monitoring the movements of nearby pedestrians, motorbikes, and bicycles and the status of traffic lights to determine the level of danger, and means for issuing notifications via an in-vehicle display or voice assistant. This enables safe and efficient driving by analyzing the driver's level of fatigue and traffic conditions in real time and accurately notifying the driver of necessary information.

[0181] A "driver" is a person who drives a vehicle and is the entity that performs driving duties.

[0182] "Vehicle" means a means of transportation such as an automobile or motorcycle designed and manufactured for travel on land.

[0183] "Means of collecting information" refers to devices and technologies that use on-board cameras and various sensors to detect the situation around the driver and vehicle.

[0184] A "generative AI model" is an artificial intelligence model that uses algorithms learned from large amounts of data to solve specific problems.

[0185] "Means of notification" refers to devices and technologies such as displays and voice assistants that convey analysis results to the driver.

[0186] "Means for analyzing facial expressions and blinking" refers to technology that captures the driver's face in real time using an in-car camera or other device, and analyzes facial expressions and blink frequency using specific algorithms.

[0187] The "means for determining fatigue level" refers to technology and equipment for estimating a driver's fatigue level based on analyzed facial expressions and blinking data.

[0188] "Means for notifying drivers to take a break" refers to display or audio notification technology that advises drivers to take a break when it is determined that the driver is highly fatigued.

[0189] "Means for monitoring the movement of pedestrians, motorbikes, and bicycles" refers to devices and technologies that use on-board sensors and cameras to detect the position and movement of objects in real time.

[0190] "Means for monitoring the status of traffic lights" refers to technology that uses in-vehicle cameras and sensors to detect the color and lighting status of traffic lights in real time.

[0191] "Means for determining the level of danger" refers to technology and devices that use generative AI models to analyze surrounding information and the driver's condition and estimate potential danger.

[0192] An "in-vehicle display" is a display device installed inside a vehicle that displays driving information and warnings.

[0193] A "voice assistant" is a device equipped with voice recognition and voice synthesis technology that interacts with the driver based on voice input and provides necessary notifications and information.

[0194] The present invention relates to a system that collects information about the driver and the vehicle's surroundings, analyzes it in real time, and notifies the driver. The system aims to improve driver safety and driving efficiency, and has functions such as judging the driver's fatigue level and encouraging them to take a break as necessary.

[0195] System Configuration

[0196] The system of the present invention consists of the following main components:

[0197] 1. Device:

[0198] The car uses on-board cameras and various sensors to collect information about the surrounding area. Specifically, the cameras capture the driver's facial expressions and blinks, while the sensors monitor nearby pedestrians, motorbikes, bicycles, and traffic lights.

[0199] 2. Server:

[0200] The system receives information sent from the device and analyzes it using a generative AI model. Specifically, the system analyzes the data using a trained AI model (using frameworks such as Google TensorFlow or PyTorch). The analysis results include predictions of pedestrians jumping out at specific locations, situations in which a motorcycle is suddenly approaching, and the driver's fatigue level.

[0201] 3. Notification mechanism:

[0202] This includes a display and voice assistant to notify the driver based on the analysis results. For example, necessary information can be immediately conveyed to the driver by displaying a warning on the in-car display or using a voice assistant (general name).

[0203] Program processing

[0204] Hardware:

[0205] In-vehicle camera: Captures the driver's facial expressions and the surrounding environment in real time.

[0206] Sensors: Capture surrounding activity (e.g. LIDAR, radar).

[0207] In-car display: Show notifications.

[0208] Voice assistant: Provides voice notifications.

[0209] software:

[0210] Generative AI models: Use frameworks such as Google TensorFlow and PyTorch.

[0211] Data analysis platform: Apache Kafka (data streaming), Apache Hadoop (data storage).

[0212] Specific examples

[0213] Scenario 1: Fatigue assessment while driving

[0214] The device captures the driver's facial expressions and blinks in real time, and the server analyzes them using a generative AI model. If the driver is judged to be fatigued, a notification will be sent saying, "You are fatigued. Please take a break."

[0215] Scenario 2: Risk prediction notification

[0216] The device collects information on the movements of pedestrians, motorbikes, and bicycles in the vicinity, as well as the status of traffic lights, which the server analyzes using a generative AI model. If a dangerous situation is predicted, a notification will be sent to the user, stating, "There is a possibility that a pedestrian may suddenly jump out at the intersection 100 meters ahead."

[0217] Scenario 3: Route guidance taking traffic congestion into account

[0218] When the user voice-selects a destination as "a nearby cafe," the server calculates the optimal route based on the latest traffic information, and the device then provides route guidance through voice prompts and a display, such as "Turn right at the next intersection and left at the next traffic light."

[0219] Prompt Sentence Examples

[0220] "The system analyzes the driver's real-time facial expressions and blinking data to determine their fatigue level. If the level of fatigue is high, the system will notify the driver by voice, saying, 'We recommend you take a break.'"

[0221] As described above, the present invention is a system that efficiently collects and analyzes information about the driver and the vehicle's surroundings using an in-vehicle camera and various sensors, thereby improving the driver's safety and driving efficiency.

[0222] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0223] Step 1: Data collection

[0224] The device uses on-board cameras and various sensors to capture the driver's facial expressions and eye blinks, as well as the status of surrounding pedestrians, motorcyclists, bicycles, and traffic lights in real time. Input data includes video feeds and sensor data. As an output, this data is pre-processed and converted into a format for analysis.

[0225] Step 2: Send data

[0226] The device sends pre-processed data to the server in real time. The input data includes captured video feeds and sensor data. These data are converted into an appropriate data format (e.g., JSON, Protobuf) and sent to the server. The output is the data packets arriving at the server.

[0227] Step 3: Data analysis

[0228] The server analyzes the received data using a generative AI model. Input data includes the video feed and sensor data sent from the device. The analysis results include the driver's fatigue level, the movement of nearby pedestrians, the approach of motorcycles and bicycles, and the status of traffic lights. The output is the analyzed results data.

[0229] Step 4: Generate notification information

[0230] The server generates notification and advice messages for the driver based on the analysis results. The input data includes the analysis results data. The generated notification messages include warning information such as "There is a risk of a pedestrian jumping out at the intersection 100 meters ahead" or "You are fatigued. Please take a break." The output is a specific notification message.

[0231] Step 5: Notification

[0232] The server sends the created notification data to the terminal, which displays it on the in-car display and notifies the driver via the voice assistant. The input data includes the notification message sent from the server. Specific actions include displaying a warning on the display and notifying the driver by voice, "There is a pedestrian ahead. Please be careful." The output is the notification being conveyed to the driver.

[0233] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0234] This invention combines a system that collects information about the driver and vehicle's surroundings and analyzes it in real time with an emotion engine that recognizes the driver's emotions. This system aims to improve driver safety and driving efficiency, and has the function of analyzing the driver's emotional state and providing appropriate advice and notifications accordingly.

[0235] System Configuration

[0236] The system of the present invention consists of the following main components:

[0237] 1. Terminal: Installed in the vehicle, it uses onboard cameras and various sensors to collect information about the surrounding area, as well as the driver's facial expressions and voice data.

[0238] 2. Server: Receives information sent from the device and analyzes it using the generative AI model and emotion engine.

[0239] 3. Notification mechanism: Display and voice assistant to notify the driver based on the analysis results.

[0240] 4. Emotion engine: Analyzes the driver's facial expressions and voice data to recognize emotions.

[0241] Program processing

[0242] The program processing in this system is carried out in the following manner.

[0243] Data collection and transmission

[0244] The device collects data in real time from on-board cameras and sensors. The collected data includes the movement of pedestrians, motorbikes, and bicycles, the status of traffic lights, and the driver's facial expressions and voice data. The device then transmits the collected data to a server. The data is transmitted in real time in an appropriate format.

[0245] Data analysis and emotion recognition

[0246] The server receives the data sent from the device and uses the generative AI model to analyze the surrounding situation and the driver's state. Furthermore, the emotion engine analyzes the driver's facial expressions and voice data to recognize the driver's emotional state. For example, it determines whether the driver is stressed or relaxed.

[0247] Generate and send notification information

[0248] The server generates notification data based on the analysis results and emotion recognition results. For example, it creates a warning message such as "A pedestrian may jump out into the intersection 100 meters ahead" or an advice message based on the driver's emotion, such as "Please concentrate on driving." The server then sends the created notification data to the device.

[0249] Notification implementation

[0250] The device displays the received notification data on the in-car display and notifies the driver via the voice assistant. Specifically, the display will show "There is a pedestrian ahead. Please be careful," and at the same time a voice message will say, "A pedestrian has jumped out ahead. Please be careful." Advice based on the driver's emotions will also be displayed and spoken.

[0251] Specific examples

[0252] Scenario 1: Risk prediction notification

[0253] When a user approaches a pedestrian intersection ahead while driving, the in-car display suddenly displays the message, "A pedestrian may jump out at the intersection 100 meters ahead," and at the same time, the voice assistant notifies the user, "A pedestrian has jumped out ahead. Please be careful." This allows the user to immediately slow down and prevent an accident.

[0254] Scenario 2: Fatigue and Emotion Recognition

[0255] If the user continues driving for a long time, the in-car camera analyzes facial expressions and blinking to determine a high level of fatigue, and the emotion engine determines the driver's stress level. In this case, the system displays a message saying "You are fatigued. Please take a break" and also provides a voice message saying "We recommend taking a break." If the stress level is high, the system will also provide a notification suggesting relaxation methods.

[0256] Scenario 3: Emotional Advice

[0257] If the emotion engine determines that the user is becoming irritated while driving, the server will detect this and provide visual and audio advice such as "Please relax while driving." At the same time, it is also possible to provide car music and relaxation guides.

[0258] In this way, the system of the present invention supports safe and comfortable driving by efficiently collecting and analyzing information about the driver and the vehicle's surroundings and notifying them at the appropriate time. It also aims to achieve even safer driving by recognizing the driver's emotional state and providing advice and support accordingly.

[0259] The processing flow will be explained below.

[0260] Step 1:

[0261] The device collects real-time data from onboard cameras and sensors, including the movement of pedestrians, motorcyclists, and bicycles, the status of traffic lights, and the driver's facial expressions and voice data.

[0262] Step 2:

[0263] The terminal transmits the collected data to a server in real time via wireless communication in an appropriate format.

[0264] Step 3:

[0265] The server receives the data sent from the device and inputs it into the generative AI model to analyze the surrounding situation and the driver's state.

[0266] Step 4:

[0267] The server analyzes various factors, such as the approach of pedestrians, motorbikes, and bicycles, and the driver's fatigue level. For example, it analyzes the prediction of pedestrians jumping out at a specific location, the situation of a motorbike suddenly approaching, the driver's facial expressions, and the frequency of blinking to assess the driver's fatigue level.

[0268] Step 5:

[0269] The server generates risk prediction data from the analysis results. For example, if there is a high possibility that a pedestrian will suddenly jump out into the intersection ahead, it will generate the necessary warning message.

[0270] Step 6:

[0271] The server uses facial expressions and voice data to recognize the driver's emotions through an emotion engine, for example, determining whether the driver is stressed or relaxed.

[0272] Step 7:

[0273] The server generates appropriate advice and notification messages for the driver based on the analysis and emotion recognition results, such as "A pedestrian may jump out into the road at the intersection 100 meters ahead" or "Please concentrate on driving."

[0274] Step 8:

[0275] The server sends the created notification data to the terminal. The data is designed to be sent quickly.

[0276] Step 9:

[0277] The device displays the received notification data on the in-car display and notifies the driver via the voice assistant. Specifically, the display will show "There is a pedestrian ahead. Please be careful," and at the same time a voice will say, "A pedestrian has jumped out ahead. Please be careful."

[0278] Step 10:

[0279] The device continuously monitors the driver's facial expressions and blinking, and evaluates the driver's emotional state using an emotion engine. If the device determines that the driver is fatigued, it will notify the driver, saying, "You are fatigued. Please take a break." If the driver's stress level is high, it will also notify the driver and suggest relaxation methods.

[0280] Step 11:

[0281] The server obtains real-time traffic congestion information from external traffic information services and calculates the optimal route to the driver's destination and intermediate stops based on the information obtained.

[0282] Step 12:

[0283] The server sends the calculated route data to the device, which receives it and starts providing guidance to the driver via voice and display, providing specific instructions such as "turn right at the next intersection."

[0284] In this way, the system of the present invention analyzes various information in real time and notifies drivers at the appropriate time to improve driver safety and efficiency. In addition, by combining it with an emotion engine, it provides advice according to the driver's emotional state, aiming to further safer driving and reduce driver stress.

[0285] Example 2

[0286] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0287] In modern society, ensuring safe and comfortable driving requires accurate understanding of the driver's situation and emotional state, and providing appropriate advice and notifications accordingly. Conventional systems lack the ability to accurately recognize the driver's emotions and provide notifications in real time, creating the risk that driver stress and fatigue could impede safe driving. Furthermore, technology for quickly analyzing surrounding information and providing appropriate advice and warnings is also not yet mature. This makes it difficult to ensure the safety of the driver and other road users.

[0288] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0289] In this invention, the server includes a means for collecting information about the driver and the vehicle's surroundings, a means for analyzing the information in real time using a generative AI model, a means for analyzing the driver's facial expressions and voice data to recognize the driver's emotional state, a means for providing advice or notification based on the emotional state, and a means for notifying the driver based on the analysis results. This allows the server to accurately recognize the driver's emotional state and provide appropriate advice or warnings accordingly. It also efficiently analyzes information about the surrounding environment to support safe driving.

[0290] "Means for collecting information about the driver and the vehicle's surroundings" refers to devices and functions that use on-board cameras, microphones, and various sensors to collect data on the driver's condition and the vehicle's surrounding environment in real time.

[0291] "Means using generative AI models for real-time analysis" refers to technologies that use generative AI models to instantly analyze collected data and understand the surrounding environment and the driver's situation.

[0292] "Means for notifying the driver based on the analysis results" refers to devices or functions for providing appropriate notifications and advice to the driver based on the analysis results of the generative AI model.

[0293] "Means for analyzing the driver's facial expressions and voice data to recognize the emotional state" refers to technology or devices that analyze the driver's facial expressions and voice data and recognize the driver's emotional state (e.g., stress, relaxation, etc.) from that information.

[0294] "Means for providing advice or notifications based on emotional state" refers to a function or device that provides appropriate advice or warnings to the driver in real time based on the driver's recognized emotional state.

[0295] "Means for setting the driver's destination and intermediate points by voice" refers to technology or devices that allow the driver to set the destination and intermediate points using voice commands.

[0296] "Means of obtaining real-time traffic congestion information and calculating the optimal route" refers to technology or devices that obtain current traffic conditions and traffic congestion information in real time and calculate the optimal driving route based on that information.

[0297] "Means for providing audio and visual guidance to the driver of the calculated route" refers to a function or device that provides audio and visual guidance to the driver of the calculated driving route.

[0298] "Means including pedestrian, motorbike, and bicycle information" refers to technology or devices that can detect the presence of and collect information about pedestrians, motorbikes, and bicycles moving around the vehicle.

[0299] "Means for notifying the driver of warnings in real time via an in-vehicle display and voice assistant" refers to a function or device that notifies the driver of warning messages in real time using an in-vehicle display and voice assistant.

[0300] The present invention is a system that efficiently collects and analyzes information about the driver and the vehicle's surroundings, and recognizes the driver's emotional state to support safe and comfortable driving. Specific embodiments of the present invention are described below.

[0301] System Configuration

[0302] The system of the present invention consists of the following main components:

[0303] 1. Terminal

[0304] 2. Server

[0305] 3. Means of notification

[0306] 4. Emotion Engine

[0307] Terminal

[0308] The device is installed in the vehicle and uses an onboard camera, microphone, and various sensors to collect information about the surrounding area, as well as the driver's facial expression and voice data. The collected data includes the movements of pedestrians, motorbikes, and bicycles, the status of traffic lights, and the driver's facial expression and voice data. The device transmits this data to a server in an appropriate format (e.g., JSON format).

[0309] server

[0310] The server receives the data sent from the device and uses a generative AI model to analyze the surrounding situation and the driver's state. This analysis includes object recognition technology and traffic light status analysis. It also uses an emotion engine to analyze the driver's facial expressions and voice data to recognize the driver's emotional state. For example, it determines whether the driver is stressed or relaxed.

[0311] Notification means

[0312] The notification means include an in-vehicle display and a voice assistant. The server generates notification information based on the analysis results and emotion recognition results and sends it to the terminal. The terminal displays the received notification information on the in-vehicle display and notifies the driver via the voice assistant.

[0313] Emotion Engine

[0314] The emotion engine analyzes the driver's facial expressions and voice data to recognize their emotional state, for example, determining their stress level and fatigue level from eye movements, blinking frequency, and tone of voice.

[0315] Specific examples

[0316] Scenario 1: Risk prediction notification

[0317] When a user approaches a pedestrian intersection ahead while driving, the device sends video footage captured by the onboard camera to a server. The server analyzes this data and detects the possibility of a pedestrian jumping out at the intersection 100 meters ahead. The analysis results are generated as notification information and sent to the device. The device receives this information and displays "A pedestrian may jump out at the intersection 100 meters ahead" on the in-car display, and the voice assistant also notifies the user. This allows the user to immediately slow down and prevent an accident.

[0318] Scenario 2: Fatigue and Emotion Recognition

[0319] When a user continues driving for a long period of time, the in-car camera captures the user's facial expressions and blinking in real time and sends the data to a server. The server analyzes this data and determines that the level of fatigue is high. At the same time, the emotion engine determines the driver's stress level. Based on this result, the server generates a message saying, "You are fatigued. Please take a break," and sends it to the device. The device then displays a notification on the in-car display and also issues a voice message saying, "We recommend that you take a break."

[0320] Scenario 3: Emotional Advice

[0321] If the emotion engine determines that the user is becoming irritated while driving, the server detects this and generates advice such as "Please relax while driving." At the same time, it also creates relaxing car music and a relaxation guide and sends these to the device. The device then notifies the user of this information via the display and voice assistant.

[0322] Prompt Sentence Examples

[0323] Please explain the specific processing flow of a safety drive system that analyzes the driver's emotions and provides appropriate advice.

[0324] In this way, the system of the present invention supports safe and comfortable driving by efficiently collecting and analyzing information about the driver and vehicle's surroundings and notifying them at the appropriate time. Furthermore, it aims to achieve even safer driving by recognizing the driver's emotional state and providing advice and support accordingly.

[0325] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0326] Step 1: Data collection

[0327] The device uses onboard cameras, microphones, and various sensors to collect real-time information about the driver and the vehicle's surroundings.

[0328] Input: Driver's facial expression, voice data, video footage of the vehicle's surroundings, sensor data (vehicle speed, brake status, etc.)

[0329] Data processing: Capture camera footage frame by frame, record audio as sample data, and log sensor data by time.

[0330] Output: Collected real-time data (camera footage, audio data, sensor data)

[0331] Step 2: Send data

[0332] The device sends the collected data to the server in an appropriate format (e.g., JSON format).

[0333] Input: Collected real-time data (camera footage, audio data, sensor data)

[0334] Data processing: Convert the data into JSON format, split it into packets, and prepare it for transmission.

[0335] Output: JSON formatted data packet

[0336] Step 3: Receiving data

[0337] The server receives the data packets sent from the terminal.

[0338] Input: JSON formatted data packet

[0339] Data processing: Reconstructing received packets and restoring them to their original data format.

[0340] Output: Recovered real-time data (camera video, audio data, sensor data)

[0341] Step 4: Data analysis

[0342] The server uses a generative AI model to analyze the data it receives, specifically using object recognition technology to detect the presence of pedestrians, motorbikes, and bicycles, as well as analyze the status of traffic lights.

[0343] Input: Recovered real-time data (camera video, audio data, sensor data)

[0344] Data Computation: Uses generative AI models to perform object recognition and analyze traffic light status.

[0345] Output: Analysis results (presence of pedestrians, traffic light status, etc.)

[0346] Step 5: Emotion Recognition

[0347] The server uses an emotion engine to analyze the driver's facial expressions and voice data to recognize their emotional state.

[0348] Input: Driver's facial expression data and voice data

[0349] Data Computation: Using the emotion engine, facial and vocal features are extracted to classify emotional states.

[0350] Output: Emotion recognition results (e.g., stress, relaxation, fatigue, etc.)

[0351] Step 6: Generate notification information

[0352] The server generates notification information based on the analysis and emotion recognition results, such as a warning message such as "A pedestrian may jump out into the road at the intersection 100 meters ahead" or an advice message such as "Please concentrate on driving."

[0353] Input: Data analysis results, emotion recognition results

[0354] Data processing: Generate appropriate messages based on analysis results and emotion recognition results

[0355] Output: Notification information (warning messages, advice messages, etc.)

[0356] Step 7: Send notification data

[0357] The server transmits the generated notification information to the terminal.

[0358] Input: Notification information

[0359] Data processing: The notification message is converted into a data packet and prepared for transmission.

[0360] Output: Notification data packet

[0361] Step 8: Notification

[0362] The device displays the received notification information on the in-car display and notifies the driver via the voice assistant.

[0363] Input: Notification data packet

[0364] Data processing: Analyzes the notification data packet and converts it into a format for display and audio output.

[0365] Output: Warning message on the display, notification via voice assistant

[0366] Specifically, when a driver approaches a pedestrian intersection ahead, the camera image is sent to the server, which then generates an analysis result as notification information and sends it to the device. The device receives this information and issues a warning on the display and via the voice assistant. This series of steps allows the user to take appropriate action.

[0367] (Application example 2)

[0368] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0369] In recent years, as autonomous vehicles have become more widely used, there has been a demand for understanding the driver's fatigue level and emotional state to provide a safe and comfortable driving experience. However, conventional technologies lack the means to accurately analyze the driver's emotional state in real time and provide appropriate notifications and advice accordingly. Furthermore, systems that can instantly notify the driver using mobile devices such as smartphones and smart glasses have not been fully established. As a result, there have been concerns about reduced driver safety and driving efficiency.

[0370] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0371] In this invention, the server includes: means for collecting information about the driver and the surroundings of the vehicle; means for using a generative AI model to analyze the information in real time; means for notifying the driver based on the analysis results; means for analyzing the driver's facial expressions and blinking to determine the driver's fatigue level; means for notifying the driver to take a break based on the fatigue level; means for recognizing the driver's emotional state and providing appropriate advice or notification based on the emotional state; and means for providing notifications to the driver using a mobile device such as a smartphone or smart glasses. This makes it possible to analyze the driver's emotional state in real time and notify the driver at the appropriate time, thereby supporting safe and smooth driving.

[0372] "Driver" means a person who operates a vehicle.

[0373] "Vehicle" means a moving object designed to travel on roads.

[0374] "Surrounding information" refers to all data about the vehicle's external environment, including pedestrians, other vehicles, bicycles, and traffic light status.

[0375] A "generative AI model" is an algorithm that uses artificial intelligence to analyze input data and generate output results.

[0376] "Notifications" are messages or alerts that convey specific information to the driver.

[0377] "Facial expressions" refer to facial movements and states that indicate a person's emotions and reactions.

[0378] Blinking is the act of closing and opening the eyes.

[0379] "Fatigue level" is an index that indicates the degree of fatigue experienced by a driver.

[0380] "Emotional state" refers to the psychological state or mood exhibited by the driver.

[0381] "Advice" means guidance or suggestions provided to a driver.

[0382] "Means for collecting information about the surrounding area" refers to devices and methods for acquiring data about the vehicle's external environment using on-board cameras, sensors, etc.

[0383] "Real-time analytical means" refers to devices or methods for processing collected data in real time and generating results.

[0384] A "smartphone" is a mobile phone equipped with internet connectivity and a camera.

[0385] "Smart glasses" are glasses-type devices that can display information in real time.

[0386] "Mobile devices" is a general term for portable electronic devices, including smartphones, tablets, and smart glasses.

[0387] "Relaxation music" is music that helps drivers relax.

[0388] "Real-time scene analysis" is a technology that instantly analyzes video data around the vehicle to grasp the current situation.

[0389] MODE FOR CARRYING OUT THE INVENTION

[0390] System Overview

[0391] This invention is a system that analyzes the driver's fatigue level and emotional state in real time and provides appropriate notifications and advice. The system components include:

[0392] 1. Device: A mobile device such as a smartphone or smart glasses that uses cameras and various sensors to collect data from the driver and the vehicle's surrounding environment.

[0393] 2. Server: Used to analyze collected data in real time, utilizing generative AI models and emotion engines for processing.

[0394] 3. Notification mechanism: Includes a display and voice assistant to notify the driver in a timely manner based on the analysis results.

[0395] Program processing

[0396] Data collection and transmission

[0397] The device uses a camera and various sensors to collect the driver's facial expressions, blinking, and information about the vehicle's surroundings (pedestrians, bicycles, other vehicles, etc.) in real time. The collected data is sent to a server in an appropriate format.

[0398] Data analysis and emotion recognition

[0399] The server uses a generative AI model to analyze the received data. It also uses an emotion engine to analyze facial expressions and voice data. This analysis identifies the driver's fatigue level and emotional state (e.g., stressed, relaxed, fatigued).

[0400] Generate and send notification information

[0401] The server generates notification data based on the analysis results. For example, if danger is imminent, a warning message such as "There is a pedestrian ahead. Please be careful" is generated. Advice based on the driver's emotional state (for example, "Please relax. We will play car music") is also generated. The generated notification data is sent to the device.

[0402] Notification implementation

[0403] The device then displays the received notification data on the display of a smartphone or smart glasses and notifies the driver via a voice assistant. The notification content includes information about nearby dangers and advice tailored to the driver's emotional state.

[0404] Hardware and software used

[0405] Camera: Cameras built into smartphones or smart glasses are used to collect data on the driver's facial expressions and the surrounding environment.

[0406] Generative AI models: Analyze the collected data using models built with machine learning libraries such as TensorFlow and Keras.

[0407] Emotion Engine: Uses the dlib library to detect facial landmarks and recognize emotions.

[0408] Audio notification: Uses the playsound library to provide audio alerts.

[0409] Specific examples

[0410] Scenario 1: Risk prediction notification

[0411] When a driver approaches an intersection ahead, the camera in the smart glasses detects pedestrians and the server notifies the driver, saying, "There is a pedestrian ahead. Please be careful." The warning is conveyed to the driver via the display and voice assistant, preventing accidents from occurring.

[0412] Scenario 2: Advice based on emotional state

[0413] If the driver shows an emotional state of "anger," the smartphone will notify the driver, "Relax. Car music will be played," and will automatically play car music.

[0414] Prompt Sentence Examples

[0415] Text format

[0416] I want to capture facial expressions from camera data on smart glasses and perform emotion recognition. If the result is "anger," I want to display "Please be careful" and play relaxation music.

[0417] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0418] Step 1:

[0419] The device uses a camera built into a smartphone or smart glasses to collect the driver's facial expressions, blinking, and information about the vehicle's surroundings (pedestrians, bicycles, other vehicles, etc.) in real time. This data includes images of the driver's facial expressions, video information about the surrounding environment, and audio data. The collected data is converted into an appropriate format as digital data and sent to a server in real time.

[0420] Step 2:

[0421] The server receives the data sent from the device. This data includes images of the driver's facial expressions and video information of the surrounding environment. To analyze the received data, the server preprocesses the data using a generative AI model. For example, it extracts the driver's facial area using a facial recognition algorithm and performs preprocessing to input the data into the emotion engine.

[0422] Step 3:

[0423] The server performs emotion recognition on the facial expression images using an emotion engine. The emotion engine uses the dlib library to detect facial landmarks (eyes, mouth, nose, etc.) and estimate the emotional state based on them. This process includes facial feature point detection, feature extraction, and emotion classification data computation. The output is the driver's emotional state (e.g., anger, sadness, joy, relaxation).

[0424] Step 4:

[0425] The server uses a generative AI model to analyze surrounding environment data and generate notification information for the driver. For example, it analyzes the location data of pedestrians, bicycles, and other vehicles, and generates a warning message such as "There is a pedestrian ahead. Please be careful" if a danger is imminent. It then performs appropriate data processing and calculations and creates a notification message as output.

[0426] Step 5:

[0427] The server generates notification data at the optimal timing for the driver based on the emotion recognition results and the analysis of the surrounding environment. For example, if the driver's emotional state is "anger," it generates an advice message such as "Please relax. We will play car music." This notification data is converted into an appropriate format and sent to the device.

[0428] Step 6:

[0429] The device displays the received notification data on the display of a smartphone or smart glasses and notifies the driver via a voice assistant. For example, a warning message such as "There is a pedestrian ahead. Please be careful" is displayed on the display and notified by voice. Relaxation music corresponding to the driver's emotional state is also automatically played on the smartphone.

[0430] Step 7:

[0431] The user takes appropriate driving actions based on notifications and advice from the device. For example, they may check for pedestrians ahead and slow down for safety. They may also listen to relaxation music to ensure a relaxed driving experience. In this way, the system supports the user's safe and comfortable driving.

[0432] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0433] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0434] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0435] [Second embodiment]

[0436] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0437] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0438] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0439] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0440] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0441] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0442] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0443] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0444] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0445] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0446] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0447] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0448] The present invention relates to a system that collects information about the driver and the vehicle's surroundings, analyzes it in real time, and notifies the driver. The system aims to improve driver safety and driving efficiency, and has functions such as judging the driver's fatigue level and encouraging them to take a break as necessary.

[0449] System Configuration

[0450] The system of the present invention consists of the following main components:

[0451] 1. Terminal: Installed in the vehicle, it collects information about the surrounding area using on-board cameras and various sensors.

[0452] 2. Server: Receives information sent from the device and analyzes it using a generative AI model.

[0453] 3. Notification mechanism: Display and voice assistant to notify the driver based on the analysis results.

[0454] Program processing

[0455] The program processing in this system is carried out in the following manner.

[0456] 1. Data Collection:

[0457] The device collects real-time data from on-board cameras and sensors, including the movement of pedestrians, motorcyclists, and cyclists, traffic light status, and the driver's facial expressions and eye blinks.

[0458] 2. Data transmission:

[0459] The terminals transmit the collected data to the server in real time in an appropriate format.

[0460] 3. Data Analysis:

[0461] The server receives the data sent from the device and analyzes it using a generative AI model. The analysis results include predictions of pedestrians running out into the road at a specific location, situations in which a motorbike is approaching suddenly, and the driver's fatigue level.

[0462] 4. Notification information generation:

[0463] Based on the analysis results, the server generates appropriate notification and advice messages for the driver, such as "There is a risk of a pedestrian jumping out into the road at the intersection 100 meters ahead" or "A motorcycle is rapidly approaching from behind."

[0464] 5. Notification Implementation:

[0465] The server then sends the created notification data to the device, which then displays it on the in-car display and notifies the driver via the voice assistant. For example, a warning may be displayed on the screen and a voice message may be heard saying, "There is a pedestrian ahead. Please be careful."

[0466] Specific examples

[0467] Scenario 1: Risk prediction notification

[0468] When a user approaches a pedestrian intersection ahead while driving, the in-car display suddenly displays the message, "A pedestrian may jump out into the road at the intersection 100 meters ahead," and at the same time, the voice assistant notifies the user, "A pedestrian may jump out into the road ahead. Please be careful," prompting the user to immediately slow down.

[0469] Scenario 2: Fatigue level assessment

[0470] If the user continues driving for a long time, the in-car camera will analyze the user's facial expressions and blinking to determine that they are fatigued. In this case, the display will show "You are fatigued. Please take a break" and a voice message will be given saying "We recommend taking a break," encouraging the user to decide to take a break.

[0471] Scenario 3: Route guidance taking traffic congestion into account

[0472] When the user voice-selects "a nearby cafe" as their destination, the server calculates the optimal route based on the latest traffic information and sends it to the device. The device then provides route guidance via voice prompts and a display: "Turn right at the next intersection, then turn left at the next traffic light."

[0473] In this way, the system of the present invention efficiently collects and analyzes information about the driver and the vehicle's surroundings, and provides notifications at the appropriate time, thereby supporting safe and comfortable driving.

[0474] The processing flow will be explained below.

[0475] Step 1:

[0476] The device collects real-time data from on-board cameras and sensors, including the movement of pedestrians, motorcyclists, and bicycles, the status of traffic lights, and the driver's facial expressions and blinking.

[0477] Step 2:

[0478] The terminal transmits the collected data to a server in real time via wireless communication in an appropriate format.

[0479] Step 3:

[0480] The server receives the data sent from the device and inputs it into the generative AI model to analyze the surrounding situation and the driver's state.

[0481] Step 4:

[0482] The server analyzes various factors, such as the approach of pedestrians, motorbikes, and bicycles, and the driver's fatigue level, to generate risk prediction data and data on the driver's condition, such as when there is a high possibility of a pedestrian jumping out into the intersection ahead or when the driver is highly fatigued.

[0483] Step 5:

[0484] The server generates notification data based on the analysis results, such as a warning message such as "A pedestrian may jump out into the street at the intersection 100 meters ahead" or a message encouraging the driver to take a break such as "You are fatigued. Please take a break."

[0485] Step 6:

[0486] The server sends the created notification data to the terminal. The data is designed to be sent quickly.

[0487] Step 7:

[0488] The device displays the received notification data on the in-car display and notifies the driver via the voice assistant. Specifically, the display will show "There is a pedestrian ahead. Please be careful," and at the same time a voice will say, "A pedestrian has jumped out ahead. Please be careful."

[0489] Step 8:

[0490] The device continuously analyzes the driver's facial expressions and blinking to monitor their fatigue level. If it determines that the driver is fatigued, it will notify the driver, saying, "You are fatigued. Please take a break."

[0491] Step 9:

[0492] The server obtains real-time traffic congestion information from external traffic information services and calculates the optimal route to the driver's destination and intermediate stops based on the information obtained.

[0493] Step 10:

[0494] The server sends the calculated route data to the device, which receives it and starts providing guidance to the driver via voice and display, providing specific instructions such as "turn right at the next intersection."

[0495] Through these steps, the system provides support to improve driver safety and efficiency.

[0496] Example 1

[0497] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0498] To improve driver safety and driving efficiency, it is important to collect and analyze information about the driver and the vehicle's surroundings in real time and provide notifications at the appropriate time. However, conventional systems have had problems with insufficient collection and analysis of information, making it difficult to provide notifications at the appropriate time. There are also few systems that can determine the driver's fatigue level in real time and prompt them to take appropriate breaks. Furthermore, there are also insufficient systems that allow the driver to set their destination via voice input or provide optimal routes based on the latest traffic congestion information.

[0499] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0500] In this invention, the server includes means for collecting information about the driver and the surroundings of the vehicle, means for using a generative AI model to analyze the information in real time, means for notifying the driver based on the analysis results, means for analyzing the driver's facial expressions and blinking to determine the level of fatigue, means for notifying the driver to take a break based on the level of fatigue, means for transmitting the collected data to the server in an appropriate format, means for generating a notification message based on the analysis results using the generative AI model, and means for communicating the notification message to the driver via an in-vehicle display or a voice assistant. This makes it possible to improve driver safety, increase driving efficiency, and reduce the burden on the driver.

[0501] "Driver" means a person who drives a vehicle.

[0502] "Vehicle" refers to land transportation such as automobiles and motorcycles.

[0503] "Information" refers to data about the vehicle and its surroundings, such as the driver's facial expressions, blinking, the movements of pedestrians, motorbikes and bicycles, and the status of traffic lights.

[0504] A "generative AI model" refers to an artificial intelligence model that analyzes collected information in real time and generates appropriate notifications and advice for drivers.

[0505] "Notification" refers to warnings and advice given to the driver based on the analysis results.

[0506] A "break notification" refers to the transmission of a message encouraging a driver to take a break when the driver's level of fatigue is judged to be high.

[0507] "Format" refers to the standard or format for properly organizing and converting data.

[0508] An "in-vehicle display" refers to a device installed inside a vehicle that displays various information on a screen.

[0509] "Voice assistant" refers to a system that uses voice to provide notifications and advice to drivers.

[0510] MODE FOR CARRYING OUT THE INVENTION

[0511] The present invention relates to a system that collects information about the driver and the vehicle's surroundings, analyzes it in real time, and notifies the driver. The system aims to improve driver safety and driving efficiency, and has the function of judging the driver's fatigue level and encouraging them to take a break as necessary.

[0512] System Configuration

[0513] The system consists of the following main components:

[0514] 1. Device:

[0515] - The device is installed in the vehicle and collects information about the surrounding area using on-board cameras and various sensors.

[0516] - The device transmits the collected data in the appropriate format to the server in real time.

[0517] - The device displays the notification message received from the server on the in-vehicle display and notifies the driver via the voice assistant.

[0518] 2. Server:

[0519] - The server receives the information sent from the device and analyzes it using the generative AI model.

[0520] - The server generates an appropriate notification message based on the analysis results.

[0521] 3. Notification mechanism:

[0522] - The notification mechanism consists of the device display and voice assistant.

[0523] - The notification mechanism provides real-time warnings and advice to the driver.

[0524] Hardware and software used

[0525] 1. Hardware:

[0526] - In-car camera: A camera that captures the driver's facial expressions, blinking, and surrounding pedestrians and vehicles.

[0527] - Various sensors: Sensors for detecting vehicle movement and surrounding information using accelerometers, gyro sensors, LIDAR, etc.

[0528] - Terminal: A terminal installed in a vehicle that collects and transmits data.

[0529] - Display and voice assistant: Output devices for notifying the driver.

[0530] 2. Software:

[0531] - Generative AI model: An artificial intelligence model for performing data analysis on the server.

[0532] - Data transmission software: Software for transmitting data from the terminal to the server.

[0533] - Notification generation software: Software for generating notification messages based on analysis results.

[0534] Specific examples

[0535] Scenario 1: Risk prediction notification

[0536] When a user approaches a pedestrian intersection while driving, the in-car display suddenly displays the message, "A pedestrian may jump out into the intersection 100 meters ahead," and at the same time, the voice assistant notifies the user, "A pedestrian has jumped out into the intersection ahead. Please be careful." This allows the user to immediately slow down.

[0537] Prompt Sentence: Generate a message to notify the driver of dangerous situations in advance while driving. For example, "A pedestrian may jump out at the intersection 100 meters ahead."

[0538] Scenario 2: Fatigue level assessment

[0539] If the user continues driving for a long time, the in-car camera analyzes facial expressions and blinking to determine that the user is highly fatigued. In this case, the display will show "You are fatigued. Please take a break" and a voice message will be played saying "We recommend taking a break." This will encourage the user to decide to take a break.

[0540] Prompt Sentence: Analyze the fatigue level of the user who drives for a long time and generate a message to prompt appropriate rest. For example, generate "You are tired. Please take a break."

[0541] Scenario 3: Route guidance taking traffic congestion into account

[0542] When the user voice-selects "a nearby cafe" as their destination, the server calculates the optimal route based on the latest traffic information and sends it to the device. The device then provides route guidance via voice prompts and a display: "Turn right at the next intersection, then turn left at the next traffic light."

[0543] Prompt: Calculate the optimal route to reach the specified destination and generate a message containing the instructions. For example, "Turn right at the next intersection, then left at the next traffic light."

[0544] In this way, the system of the present invention efficiently collects and analyzes information about the driver and vehicle's surroundings, and provides notifications at appropriate times to support safe and comfortable driving.

[0545] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0546] Step 1: Data collection

[0547] The device collects data in real time using onboard cameras and various sensors.

[0548] Input: Information captured by the in-vehicle camera, such as the driver's facial expressions and blinking, the movements of pedestrians, motorbikes, and bicycles, and the status of traffic lights. Data on the vehicle's movements and surrounding conditions from sensors.

[0549] Data processing / calculation: Each data point is temporarily stored in a buffer and converted into an easily readable format.

[0550] Output: A collection of recorded data.

[0551] Specific operation: The device uses the onboard camera to capture the driver's facial expressions and blinks, uses accelerometers and gyro sensors to detect vehicle movement, and uses LIDAR to scan the location of surrounding objects and pedestrians.

[0552] Step 2: Send data

[0553] The terminal transmits the collected data to the server in real time.

[0554] Input: Various data stored on the device.

[0555] Data processing / computation: Converting data into an appropriate format (e.g., JSON or XML) and encrypting it.

[0556] Output: The formatted encrypted data.

[0557] Specific operation: The device converts the collected data into an appropriate format, encrypts it for security purposes, and sends it to a server over the Internet.

[0558] Step 3: Data analysis

[0559] The server receives the data sent from the device and analyzes it using a generative AI model.

[0560] Input: Encrypted data sent from the device after format conversion.

[0561] Data processing / calculation: Data is input into a generative AI model to predict when pedestrians will suddenly run out into the road, when motorcycles will suddenly approach, and determine the driver's level of fatigue.

[0562] Output: Analysis results (e.g., prediction of pedestrians running out into the road, determination of driver fatigue level, etc.).

[0563] Specific operation: The server temporarily stores the received data in a database, extracts the necessary data from the database, inputs it into the generative AI model for analysis, and predicts the movements of pedestrians and the state of the driver as a result.

[0564] Step 4: Generate notification information

[0565] The server generates a message to notify the driver based on the analysis results.

[0566] Input: Analysis results from the generative AI model.

[0567] Data processing / calculation: Based on the analysis results, an appropriate notification message is generated for the driver.

[0568] Output: The generated notification message.

[0569] Specific operation: The server generates an appropriate notification message based on the analysis results obtained by the generative AI model. For example, it creates a message such as "A pedestrian may jump out into the street at the intersection 100 meters ahead."

[0570] Step 5: Notification

[0571] The server sends the created notification data to the terminal, which then notifies the driver via the in-car display and voice assistant.

[0572] Input: The generated notification message.

[0573] Data processing / calculation: Converts the notification message into an appropriate format and sends it to the terminal.

[0574] Output: Notification messages sent to the terminal.

[0575] Specific operation: The server sends the generated notification message to the device, and the device displays the received notification data on the in-car display and notifies the driver via the voice assistant. For example, the display may display "There is a pedestrian ahead. Please be careful" and a voice message may also be issued saying "There is a pedestrian ahead. Please be careful."

[0576] In this way, through the specific operations of each step, the system can provide a safe driving environment for the driver and improve driving efficiency.

[0577] (Application example 1)

[0578] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0579] In today's modern transportation society, there is a demand for improved driver safety and driving efficiency. In particular, the risk of accidents is increasing due to fatigue from long periods of driving and inattention to surrounding traffic conditions. Therefore, technology is needed to enable drivers to continue driving safely and efficiently. Another challenge is to monitor and analyze traffic conditions and driver status in real time and provide notifications at the appropriate time.

[0580] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0581] In this invention, the server includes means for collecting information about the driver and the surrounding area of ​​the vehicle, means for using a generative AI model to analyze the information in real time, means for notifying the driver based on the analysis results, means for analyzing the driver's facial expressions and blinking to determine the level of fatigue, means for issuing a notification encouraging the driver to take a break based on the level of fatigue, means for monitoring the movements of nearby pedestrians, motorbikes, and bicycles and the status of traffic lights to determine the level of danger, and means for issuing notifications via an in-vehicle display or voice assistant. This enables safe and efficient driving by analyzing the driver's level of fatigue and traffic conditions in real time and accurately notifying the driver of necessary information.

[0582] A "driver" is a person who drives a vehicle and is the entity that performs driving duties.

[0583] "Vehicle" means a means of transportation such as an automobile or motorcycle designed and manufactured for travel on land.

[0584] "Means of collecting information" refers to devices and technologies that use on-board cameras and various sensors to detect the situation around the driver and vehicle.

[0585] A "generative AI model" is an artificial intelligence model that uses algorithms learned from large amounts of data to solve specific problems.

[0586] "Means of notification" refers to devices and technologies such as displays and voice assistants that convey analysis results to the driver.

[0587] "Means for analyzing facial expressions and blinking" refers to technology that captures the driver's face in real time using an in-car camera or other device, and analyzes facial expressions and blink frequency using specific algorithms.

[0588] The "means for determining fatigue level" refers to technology and equipment for estimating a driver's fatigue level based on analyzed facial expressions and blinking data.

[0589] "Means for notifying drivers to take a break" refers to display or audio notification technology that advises drivers to take a break when it is determined that the driver is highly fatigued.

[0590] "Means for monitoring the movement of pedestrians, motorbikes, and bicycles" refers to devices and technologies that use on-board sensors and cameras to detect the position and movement of objects in real time.

[0591] "Means for monitoring the status of traffic lights" refers to technology that uses in-vehicle cameras and sensors to detect the color and lighting status of traffic lights in real time.

[0592] "Means for determining the level of danger" refers to technology and devices that use generative AI models to analyze surrounding information and the driver's condition and estimate potential danger.

[0593] An "in-vehicle display" is a display device installed inside a vehicle that displays driving information and warnings.

[0594] A "voice assistant" is a device equipped with voice recognition and voice synthesis technology that interacts with the driver based on voice input and provides necessary notifications and information.

[0595] The present invention relates to a system that collects information about the driver and the vehicle's surroundings, analyzes it in real time, and notifies the driver. The system aims to improve driver safety and driving efficiency, and has functions such as judging the driver's fatigue level and encouraging them to take a break as necessary.

[0596] System Configuration

[0597] The system of the present invention consists of the following main components:

[0598] 1. Device:

[0599] The car uses on-board cameras and various sensors to collect information about the surrounding area. Specifically, the cameras capture the driver's facial expressions and blinks, while the sensors monitor nearby pedestrians, motorbikes, bicycles, and traffic lights.

[0600] 2. Server:

[0601] The system receives information sent from the device and analyzes it using a generative AI model. Specifically, the system analyzes the data using a trained AI model (using frameworks such as Google TensorFlow or PyTorch). The analysis results include predictions of pedestrians jumping out at specific locations, situations in which a motorcycle is suddenly approaching, and the driver's fatigue level.

[0602] 3. Notification mechanism:

[0603] This includes a display and voice assistant to notify the driver based on the analysis results. For example, necessary information can be immediately conveyed to the driver by displaying a warning on the in-car display or using a voice assistant (general name).

[0604] Program processing

[0605] Hardware:

[0606] In-vehicle camera: Captures the driver's facial expressions and the surrounding environment in real time.

[0607] Sensors: Capture surrounding activity (e.g. LIDAR, radar).

[0608] In-car display: Show notifications.

[0609] Voice assistant: Provides voice notifications.

[0610] software:

[0611] Generative AI models: Use frameworks such as Google TensorFlow and PyTorch.

[0612] Data analysis platform: Apache Kafka (data streaming), Apache Hadoop (data storage).

[0613] Specific examples

[0614] Scenario 1: Fatigue assessment while driving

[0615] The device captures the driver's facial expressions and blinks in real time, and the server analyzes them using a generative AI model. If the driver is judged to be fatigued, a notification will be sent saying, "You are fatigued. Please take a break."

[0616] Scenario 2: Risk prediction notification

[0617] The device collects information on the movements of pedestrians, motorbikes, and bicycles in the vicinity, as well as the status of traffic lights, which the server analyzes using a generative AI model. If a dangerous situation is predicted, a notification will be sent to the user, stating, "There is a possibility that a pedestrian may suddenly jump out at the intersection 100 meters ahead."

[0618] Scenario 3: Route guidance taking traffic congestion into account

[0619] When the user voice-selects a destination as "a nearby cafe," the server calculates the optimal route based on the latest traffic information, and the device then provides route guidance through voice prompts and a display, such as "Turn right at the next intersection and left at the next traffic light."

[0620] Prompt Sentence Examples

[0621] "The system analyzes the driver's real-time facial expressions and blinking data to determine their fatigue level. If the level of fatigue is high, the system will notify the driver by voice, saying, 'We recommend you take a break.'"

[0622] As described above, the present invention is a system that efficiently collects and analyzes information about the driver and the vehicle's surroundings using an in-vehicle camera and various sensors, thereby improving the driver's safety and driving efficiency.

[0623] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0624] Step 1: Data collection

[0625] The device uses on-board cameras and various sensors to capture the driver's facial expressions and eye blinks, as well as the status of surrounding pedestrians, motorcyclists, bicycles, and traffic lights in real time. Input data includes video feeds and sensor data. As an output, this data is pre-processed and converted into a format for analysis.

[0626] Step 2: Send data

[0627] The device sends pre-processed data to the server in real time. The input data includes captured video feeds and sensor data. These data are converted into an appropriate data format (e.g., JSON, Protobuf) and sent to the server. The output is the data packets arriving at the server.

[0628] Step 3: Data analysis

[0629] The server analyzes the received data using a generative AI model. Input data includes the video feed and sensor data sent from the device. The analysis results include the driver's fatigue level, the movement of nearby pedestrians, the approach of motorcycles and bicycles, and the status of traffic lights. The output is the analyzed results data.

[0630] Step 4: Generate notification information

[0631] The server generates notification and advice messages for the driver based on the analysis results. The input data includes the analysis results data. The generated notification messages include warning information such as "There is a risk of a pedestrian jumping out at the intersection 100 meters ahead" or "You are fatigued. Please take a break." The output is a specific notification message.

[0632] Step 5: Notification

[0633] The server sends the created notification data to the terminal, which displays it on the in-car display and notifies the driver via the voice assistant. The input data includes the notification message sent from the server. Specific actions include displaying a warning on the display and notifying the driver by voice, "There is a pedestrian ahead. Please be careful." The output is the notification being conveyed to the driver.

[0634] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0635] This invention combines a system that collects information about the driver and vehicle's surroundings and analyzes it in real time with an emotion engine that recognizes the driver's emotions. This system aims to improve driver safety and driving efficiency, and has the function of analyzing the driver's emotional state and providing appropriate advice and notifications accordingly.

[0636] System Configuration

[0637] The system of the present invention consists of the following main components:

[0638] 1. Terminal: Installed in the vehicle, it uses onboard cameras and various sensors to collect information about the surrounding area, as well as the driver's facial expressions and voice data.

[0639] 2. Server: Receives information sent from the device and analyzes it using the generative AI model and emotion engine.

[0640] 3. Notification mechanism: Display and voice assistant to notify the driver based on the analysis results.

[0641] 4. Emotion engine: Analyzes the driver's facial expressions and voice data to recognize emotions.

[0642] Program processing

[0643] The program processing in this system is carried out in the following manner.

[0644] Data collection and transmission

[0645] The device collects data in real time from on-board cameras and sensors. The collected data includes the movement of pedestrians, motorbikes, and bicycles, the status of traffic lights, and the driver's facial expressions and voice data. The device then transmits the collected data to a server. The data is transmitted in real time in an appropriate format.

[0646] Data analysis and emotion recognition

[0647] The server receives the data sent from the device and uses the generative AI model to analyze the surrounding situation and the driver's state. Furthermore, the emotion engine analyzes the driver's facial expressions and voice data to recognize the driver's emotional state. For example, it determines whether the driver is stressed or relaxed.

[0648] Generate and send notification information

[0649] The server generates notification data based on the analysis results and emotion recognition results. For example, it creates a warning message such as "A pedestrian may jump out into the intersection 100 meters ahead" or an advice message based on the driver's emotion, such as "Please concentrate on driving." The server then sends the created notification data to the device.

[0650] Notification implementation

[0651] The device displays the received notification data on the in-car display and notifies the driver via the voice assistant. Specifically, the display will show "There is a pedestrian ahead. Please be careful," and at the same time a voice message will say, "A pedestrian has jumped out ahead. Please be careful." Advice based on the driver's emotions will also be displayed and spoken.

[0652] Specific examples

[0653] Scenario 1: Risk prediction notification

[0654] When a user approaches a pedestrian intersection ahead while driving, the in-car display suddenly displays the message, "A pedestrian may jump out at the intersection 100 meters ahead," and at the same time, the voice assistant notifies the user, "A pedestrian has jumped out ahead. Please be careful." This allows the user to immediately slow down and prevent an accident.

[0655] Scenario 2: Fatigue and Emotion Recognition

[0656] If the user continues driving for a long time, the in-car camera analyzes facial expressions and blinking to determine a high level of fatigue, and the emotion engine determines the driver's stress level. In this case, the system displays a message saying "You are fatigued. Please take a break" and also provides a voice message saying "We recommend taking a break." If the stress level is high, the system will also provide a notification suggesting relaxation methods.

[0657] Scenario 3: Emotional Advice

[0658] If the emotion engine determines that the user is becoming irritated while driving, the server will detect this and provide visual and audio advice such as "Please relax while driving." At the same time, it is also possible to provide car music and relaxation guides.

[0659] In this way, the system of the present invention supports safe and comfortable driving by efficiently collecting and analyzing information about the driver and the vehicle's surroundings and notifying them at the appropriate time. It also aims to achieve even safer driving by recognizing the driver's emotional state and providing advice and support accordingly.

[0660] The processing flow will be explained below.

[0661] Step 1:

[0662] The device collects real-time data from onboard cameras and sensors, including the movement of pedestrians, motorcyclists, and bicycles, the status of traffic lights, and the driver's facial expressions and voice data.

[0663] Step 2:

[0664] The terminal transmits the collected data to a server in real time via wireless communication in an appropriate format.

[0665] Step 3:

[0666] The server receives the data sent from the device and inputs it into the generative AI model to analyze the surrounding situation and the driver's state.

[0667] Step 4:

[0668] The server analyzes various factors, such as the approach of pedestrians, motorbikes, and bicycles, and the driver's fatigue level. For example, it analyzes the prediction of pedestrians jumping out at a specific location, the situation of a motorbike suddenly approaching, the driver's facial expressions, and the frequency of blinking to assess the driver's fatigue level.

[0669] Step 5:

[0670] The server generates risk prediction data from the analysis results. For example, if there is a high possibility that a pedestrian will suddenly jump out into the intersection ahead, it will generate the necessary warning message.

[0671] Step 6:

[0672] The server uses facial expressions and voice data to recognize the driver's emotions through an emotion engine, for example, determining whether the driver is stressed or relaxed.

[0673] Step 7:

[0674] The server generates appropriate advice and notification messages for the driver based on the analysis and emotion recognition results, such as "A pedestrian may jump out into the road at the intersection 100 meters ahead" or "Please concentrate on driving."

[0675] Step 8:

[0676] The server sends the created notification data to the terminal. The data is designed to be sent quickly.

[0677] Step 9:

[0678] The device displays the received notification data on the in-car display and notifies the driver via the voice assistant. Specifically, the display will show "There is a pedestrian ahead. Please be careful," and at the same time a voice will say, "A pedestrian has jumped out ahead. Please be careful."

[0679] Step 10:

[0680] The device continuously monitors the driver's facial expressions and blinking, and evaluates the driver's emotional state using an emotion engine. If the device determines that the driver is fatigued, it will notify the driver, saying, "You are fatigued. Please take a break." If the driver's stress level is high, it will also notify the driver and suggest relaxation methods.

[0681] Step 11:

[0682] The server obtains real-time traffic congestion information from external traffic information services and calculates the optimal route to the driver's destination and intermediate stops based on the information obtained.

[0683] Step 12:

[0684] The server sends the calculated route data to the device, which receives it and starts providing guidance to the driver via voice and display, providing specific instructions such as "turn right at the next intersection."

[0685] In this way, the system of the present invention analyzes various information in real time and notifies drivers at the appropriate time to improve driver safety and efficiency. In addition, by combining it with an emotion engine, it provides advice according to the driver's emotional state, aiming to further safer driving and reduce driver stress.

[0686] Example 2

[0687] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0688] In modern society, ensuring safe and comfortable driving requires accurate understanding of the driver's situation and emotional state, and providing appropriate advice and notifications accordingly. Conventional systems lack the ability to accurately recognize the driver's emotions and provide notifications in real time, creating the risk that driver stress and fatigue could impede safe driving. Furthermore, technology for quickly analyzing surrounding information and providing appropriate advice and warnings is also not yet mature. This makes it difficult to ensure the safety of the driver and other road users.

[0689] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0690] In this invention, the server includes a means for collecting information about the driver and the vehicle's surroundings, a means for analyzing the information in real time using a generative AI model, a means for analyzing the driver's facial expressions and voice data to recognize the driver's emotional state, a means for providing advice or notification based on the emotional state, and a means for notifying the driver based on the analysis results. This allows the server to accurately recognize the driver's emotional state and provide appropriate advice or warnings accordingly. It also efficiently analyzes information about the surrounding environment to support safe driving.

[0691] "Means for collecting information about the driver and the vehicle's surroundings" refers to devices and functions that use on-board cameras, microphones, and various sensors to collect data on the driver's condition and the vehicle's surrounding environment in real time.

[0692] "Means using generative AI models for real-time analysis" refers to technologies that use generative AI models to instantly analyze collected data and understand the surrounding environment and the driver's situation.

[0693] "Means for notifying the driver based on the analysis results" refers to devices or functions for providing appropriate notifications and advice to the driver based on the analysis results of the generative AI model.

[0694] "Means for analyzing the driver's facial expressions and voice data to recognize the emotional state" refers to technology or devices that analyze the driver's facial expressions and voice data and recognize the driver's emotional state (e.g., stress, relaxation, etc.) from that information.

[0695] "Means for providing advice or notifications based on emotional state" refers to a function or device that provides appropriate advice or warnings to the driver in real time based on the driver's recognized emotional state.

[0696] "Means for setting the driver's destination and intermediate points by voice" refers to technology or devices that allow the driver to set the destination and intermediate points using voice commands.

[0697] "Means of obtaining real-time traffic congestion information and calculating the optimal route" refers to technology or devices that obtain current traffic conditions and traffic congestion information in real time and calculate the optimal driving route based on that information.

[0698] "Means for providing audio and visual guidance to the driver of the calculated route" refers to a function or device that provides audio and visual guidance to the driver of the calculated driving route.

[0699] "Means including pedestrian, motorbike, and bicycle information" refers to technology or devices that can detect the presence of and collect information about pedestrians, motorbikes, and bicycles moving around the vehicle.

[0700] "Means for notifying the driver of warnings in real time via an in-vehicle display and voice assistant" refers to a function or device that notifies the driver of warning messages in real time using an in-vehicle display and voice assistant.

[0701] The present invention is a system that efficiently collects and analyzes information about the driver and the vehicle's surroundings, and recognizes the driver's emotional state to support safe and comfortable driving. Specific embodiments of the present invention are described below.

[0702] System Configuration

[0703] The system of the present invention consists of the following main components:

[0704] 1. Terminal

[0705] 2. Server

[0706] 3. Means of notification

[0707] 4. Emotion Engine

[0708] Terminal

[0709] The device is installed in the vehicle and uses an onboard camera, microphone, and various sensors to collect information about the surrounding area, as well as the driver's facial expression and voice data. The collected data includes the movements of pedestrians, motorbikes, and bicycles, the status of traffic lights, and the driver's facial expression and voice data. The device transmits this data to a server in an appropriate format (e.g., JSON format).

[0710] server

[0711] The server receives the data sent from the device and uses a generative AI model to analyze the surrounding situation and the driver's state. This analysis includes object recognition technology and traffic light status analysis. It also uses an emotion engine to analyze the driver's facial expressions and voice data to recognize the driver's emotional state. For example, it determines whether the driver is stressed or relaxed.

[0712] Notification means

[0713] The notification means include an in-vehicle display and a voice assistant. The server generates notification information based on the analysis results and emotion recognition results and sends it to the terminal. The terminal displays the received notification information on the in-vehicle display and notifies the driver via the voice assistant.

[0714] Emotion Engine

[0715] The emotion engine analyzes the driver's facial expressions and voice data to recognize their emotional state, for example, determining their stress level and fatigue level from eye movements, blinking frequency, and tone of voice.

[0716] Specific examples

[0717] Scenario 1: Risk prediction notification

[0718] When a user approaches a pedestrian intersection ahead while driving, the device sends video footage captured by the onboard camera to a server. The server analyzes this data and detects the possibility of a pedestrian jumping out at the intersection 100 meters ahead. The analysis results are generated as notification information and sent to the device. The device receives this information and displays "A pedestrian may jump out at the intersection 100 meters ahead" on the in-car display, and the voice assistant also notifies the user. This allows the user to immediately slow down and prevent an accident.

[0719] Scenario 2: Fatigue and Emotion Recognition

[0720] When a user continues driving for a long period of time, the in-car camera captures the user's facial expressions and blinking in real time and sends the data to a server. The server analyzes this data and determines that the level of fatigue is high. At the same time, the emotion engine determines the driver's stress level. Based on this result, the server generates a message saying, "You are fatigued. Please take a break," and sends it to the device. The device then displays a notification on the in-car display and also issues a voice message saying, "We recommend that you take a break."

[0721] Scenario 3: Emotional Advice

[0722] If the emotion engine determines that the user is becoming irritated while driving, the server detects this and generates advice such as "Please relax while driving." At the same time, it also creates relaxing car music and a relaxation guide and sends these to the device. The device then notifies the user of this information via the display and voice assistant.

[0723] Prompt Sentence Examples

[0724] Please explain the specific processing flow of a safety drive system that analyzes the driver's emotions and provides appropriate advice.

[0725] In this way, the system of the present invention supports safe and comfortable driving by efficiently collecting and analyzing information about the driver and vehicle's surroundings and notifying them at the appropriate time. Furthermore, it aims to achieve even safer driving by recognizing the driver's emotional state and providing advice and support accordingly.

[0726] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0727] Step 1: Data collection

[0728] The device uses onboard cameras, microphones, and various sensors to collect real-time information about the driver and the vehicle's surroundings.

[0729] Input: Driver's facial expression, voice data, video footage of the vehicle's surroundings, sensor data (vehicle speed, brake status, etc.)

[0730] Data processing: Capture camera footage frame by frame, record audio as sample data, and log sensor data by time.

[0731] Output: Collected real-time data (camera footage, audio data, sensor data)

[0732] Step 2: Send data

[0733] The device sends the collected data to the server in an appropriate format (e.g., JSON format).

[0734] Input: Collected real-time data (camera footage, audio data, sensor data)

[0735] Data processing: Convert the data into JSON format, split it into packets, and prepare it for transmission.

[0736] Output: JSON formatted data packet

[0737] Step 3: Receiving data

[0738] The server receives the data packets sent from the terminal.

[0739] Input: JSON formatted data packet

[0740] Data processing: Reconstructing received packets and restoring them to their original data format.

[0741] Output: Recovered real-time data (camera video, audio data, sensor data)

[0742] Step 4: Data analysis

[0743] The server uses a generative AI model to analyze the data it receives, specifically using object recognition technology to detect the presence of pedestrians, motorbikes, and bicycles, as well as analyze the status of traffic lights.

[0744] Input: Recovered real-time data (camera video, audio data, sensor data)

[0745] Data Computation: Uses generative AI models to perform object recognition and analyze traffic light status.

[0746] Output: Analysis results (presence of pedestrians, traffic light status, etc.)

[0747] Step 5: Emotion Recognition

[0748] The server uses an emotion engine to analyze the driver's facial expressions and voice data to recognize their emotional state.

[0749] Input: Driver's facial expression data and voice data

[0750] Data Computation: Using the emotion engine, facial and vocal features are extracted to classify emotional states.

[0751] Output: Emotion recognition results (e.g., stress, relaxation, fatigue, etc.)

[0752] Step 6: Generate notification information

[0753] The server generates notification information based on the analysis and emotion recognition results, such as a warning message such as "A pedestrian may jump out into the road at the intersection 100 meters ahead" or an advice message such as "Please concentrate on driving."

[0754] Input: Data analysis results, emotion recognition results

[0755] Data processing: Generate appropriate messages based on analysis results and emotion recognition results

[0756] Output: Notification information (warning messages, advice messages, etc.)

[0757] Step 7: Send notification data

[0758] The server transmits the generated notification information to the terminal.

[0759] Input: Notification information

[0760] Data processing: The notification message is converted into a data packet and prepared for transmission.

[0761] Output: Notification data packet

[0762] Step 8: Notification

[0763] The device displays the received notification information on the in-car display and notifies the driver via the voice assistant.

[0764] Input: Notification data packet

[0765] Data processing: Analyzes the notification data packet and converts it into a format for display and audio output.

[0766] Output: Warning message on the display, notification via voice assistant

[0767] Specifically, when a driver approaches a pedestrian intersection ahead, the camera image is sent to the server, which then generates an analysis result as notification information and sends it to the device. The device receives this information and issues a warning on the display and via the voice assistant. This series of steps allows the user to take appropriate action.

[0768] (Application example 2)

[0769] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0770] In recent years, as autonomous vehicles have become more widely used, there has been a demand for understanding the driver's fatigue level and emotional state to provide a safe and comfortable driving experience. However, conventional technologies lack the means to accurately analyze the driver's emotional state in real time and provide appropriate notifications and advice accordingly. Furthermore, systems that can instantly notify the driver using mobile devices such as smartphones and smart glasses have not been fully established. As a result, there have been concerns about reduced driver safety and driving efficiency.

[0771] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0772] In this invention, the server includes: means for collecting information about the driver and the surroundings of the vehicle; means for using a generative AI model to analyze the information in real time; means for notifying the driver based on the analysis results; means for analyzing the driver's facial expressions and blinking to determine the driver's fatigue level; means for notifying the driver to take a break based on the fatigue level; means for recognizing the driver's emotional state and providing appropriate advice or notification based on the emotional state; and means for providing notifications to the driver using a mobile device such as a smartphone or smart glasses. This makes it possible to analyze the driver's emotional state in real time and notify the driver at the appropriate time, thereby supporting safe and smooth driving.

[0773] "Driver" means a person who operates a vehicle.

[0774] "Vehicle" means a moving object designed to travel on roads.

[0775] "Surrounding information" refers to all data about the vehicle's external environment, including pedestrians, other vehicles, bicycles, and traffic light status.

[0776] A "generative AI model" is an algorithm that uses artificial intelligence to analyze input data and generate output results.

[0777] "Notifications" are messages or alerts that convey specific information to the driver.

[0778] "Facial expressions" refer to facial movements and states that indicate a person's emotions and reactions.

[0779] Blinking is the act of closing and opening the eyes.

[0780] "Fatigue level" is an index that indicates the degree of fatigue experienced by a driver.

[0781] "Emotional state" refers to the psychological state or mood exhibited by the driver.

[0782] "Advice" means guidance or suggestions provided to a driver.

[0783] "Means for collecting information about the surrounding area" refers to devices and methods for acquiring data about the vehicle's external environment using on-board cameras, sensors, etc.

[0784] "Real-time analytical means" refers to devices or methods for processing collected data in real time and generating results.

[0785] A "smartphone" is a mobile phone equipped with internet connectivity and a camera.

[0786] "Smart glasses" are glasses-type devices that can display information in real time.

[0787] "Mobile devices" is a general term for portable electronic devices, including smartphones, tablets, and smart glasses.

[0788] "Relaxation music" is music that helps drivers relax.

[0789] "Real-time scene analysis" is a technology that instantly analyzes video data around the vehicle to grasp the current situation.

[0790] MODE FOR CARRYING OUT THE INVENTION

[0791] System Overview

[0792] This invention is a system that analyzes the driver's fatigue level and emotional state in real time and provides appropriate notifications and advice. The system components include:

[0793] 1. Device: A mobile device such as a smartphone or smart glasses that uses cameras and various sensors to collect data from the driver and the vehicle's surrounding environment.

[0794] 2. Server: Used to analyze collected data in real time, utilizing generative AI models and emotion engines for processing.

[0795] 3. Notification mechanism: Includes a display and voice assistant to notify the driver in a timely manner based on the analysis results.

[0796] Program processing

[0797] Data collection and transmission

[0798] The device uses a camera and various sensors to collect the driver's facial expressions, blinking, and information about the vehicle's surroundings (pedestrians, bicycles, other vehicles, etc.) in real time. The collected data is sent to a server in an appropriate format.

[0799] Data analysis and emotion recognition

[0800] The server uses a generative AI model to analyze the received data. It also uses an emotion engine to analyze facial expressions and voice data. This analysis identifies the driver's fatigue level and emotional state (e.g., stressed, relaxed, fatigued).

[0801] Generate and send notification information

[0802] The server generates notification data based on the analysis results. For example, if danger is imminent, a warning message such as "There is a pedestrian ahead. Please be careful" is generated. Advice based on the driver's emotional state (for example, "Please relax. We will play car music") is also generated. The generated notification data is sent to the device.

[0803] Notification implementation

[0804] The device then displays the received notification data on the display of a smartphone or smart glasses and notifies the driver via a voice assistant. The notification content includes information about nearby dangers and advice tailored to the driver's emotional state.

[0805] Hardware and software used

[0806] Camera: Cameras built into smartphones or smart glasses are used to collect data on the driver's facial expressions and the surrounding environment.

[0807] Generative AI models: Analyze the collected data using models built with machine learning libraries such as TensorFlow and Keras.

[0808] Emotion Engine: Uses the dlib library to detect facial landmarks and recognize emotions.

[0809] Audio notification: Uses the playsound library to provide audio alerts.

[0810] Specific examples

[0811] Scenario 1: Risk prediction notification

[0812] When a driver approaches an intersection ahead, the camera in the smart glasses detects pedestrians and the server notifies the driver, saying, "There is a pedestrian ahead. Please be careful." The warning is conveyed to the driver via the display and voice assistant, preventing accidents from occurring.

[0813] Scenario 2: Advice based on emotional state

[0814] If the driver shows an emotional state of "anger," the smartphone will notify the driver, "Relax. Car music will be played," and will automatically play car music.

[0815] Prompt Sentence Examples

[0816] Text format

[0817] I want to capture facial expressions from camera data on smart glasses and perform emotion recognition. If the result is "anger," I want to display "Please be careful" and play relaxation music.

[0818] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0819] Step 1:

[0820] The device uses a camera built into a smartphone or smart glasses to collect the driver's facial expressions, blinking, and information about the vehicle's surroundings (pedestrians, bicycles, other vehicles, etc.) in real time. This data includes images of the driver's facial expressions, video information about the surrounding environment, and audio data. The collected data is converted into an appropriate format as digital data and sent to a server in real time.

[0821] Step 2:

[0822] The server receives the data sent from the device. This data includes images of the driver's facial expressions and video information of the surrounding environment. To analyze the received data, the server preprocesses the data using a generative AI model. For example, it extracts the driver's facial area using a facial recognition algorithm and performs preprocessing to input the data into the emotion engine.

[0823] Step 3:

[0824] The server performs emotion recognition on the facial expression images using an emotion engine. The emotion engine uses the dlib library to detect facial landmarks (eyes, mouth, nose, etc.) and estimate the emotional state based on them. This process includes facial feature point detection, feature extraction, and emotion classification data computation. The output is the driver's emotional state (e.g., anger, sadness, joy, relaxation).

[0825] Step 4:

[0826] The server uses a generative AI model to analyze surrounding environment data and generate notification information for the driver. For example, it analyzes the location data of pedestrians, bicycles, and other vehicles, and generates a warning message such as "There is a pedestrian ahead. Please be careful" if a danger is imminent. It then performs appropriate data processing and calculations and creates a notification message as output.

[0827] Step 5:

[0828] The server generates notification data at the optimal timing for the driver based on the emotion recognition results and the analysis of the surrounding environment. For example, if the driver's emotional state is "anger," it generates an advice message such as "Please relax. We will play car music." This notification data is converted into an appropriate format and sent to the device.

[0829] Step 6:

[0830] The device displays the received notification data on the display of a smartphone or smart glasses and notifies the driver via a voice assistant. For example, a warning message such as "There is a pedestrian ahead. Please be careful" is displayed on the display and notified by voice. Relaxation music corresponding to the driver's emotional state is also automatically played on the smartphone.

[0831] Step 7:

[0832] The user takes appropriate driving actions based on notifications and advice from the device. For example, they may check for pedestrians ahead and slow down for safety. They may also listen to relaxation music to ensure a relaxed driving experience. In this way, the system supports the user's safe and comfortable driving.

[0833] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0834] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0835] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0836] [Third embodiment]

[0837] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0838] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0839] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0840] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0841] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0842] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0843] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0844] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0845] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0846] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0847] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0848] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0849] The present invention relates to a system that collects information about the driver and the vehicle's surroundings, analyzes it in real time, and notifies the driver. The system aims to improve driver safety and driving efficiency, and has functions such as judging the driver's fatigue level and encouraging them to take a break as necessary.

[0850] System Configuration

[0851] The system of the present invention consists of the following main components:

[0852] 1. Terminal: Installed in the vehicle, it collects information about the surrounding area using on-board cameras and various sensors.

[0853] 2. Server: Receives information sent from the device and analyzes it using a generative AI model.

[0854] 3. Notification mechanism: Display and voice assistant to notify the driver based on the analysis results.

[0855] Program processing

[0856] The program processing in this system is carried out in the following manner.

[0857] 1. Data Collection:

[0858] The device collects real-time data from on-board cameras and sensors, including the movement of pedestrians, motorcyclists, and cyclists, traffic light status, and the driver's facial expressions and eye blinks.

[0859] 2. Data transmission:

[0860] The terminals transmit the collected data to the server in real time in an appropriate format.

[0861] 3. Data Analysis:

[0862] The server receives the data sent from the device and analyzes it using a generative AI model. The analysis results include predictions of pedestrians running out into the road at a specific location, situations in which a motorbike is approaching suddenly, and the driver's fatigue level.

[0863] 4. Notification information generation:

[0864] Based on the analysis results, the server generates appropriate notification and advice messages for the driver, such as "There is a risk of a pedestrian jumping out into the road at the intersection 100 meters ahead" or "A motorcycle is rapidly approaching from behind."

[0865] 5. Notification Implementation:

[0866] The server then sends the created notification data to the device, which then displays it on the in-car display and notifies the driver via the voice assistant. For example, a warning may be displayed on the screen and a voice message may be heard saying, "There is a pedestrian ahead. Please be careful."

[0867] Specific examples

[0868] Scenario 1: Risk prediction notification

[0869] When a user approaches a pedestrian intersection ahead while driving, the in-car display suddenly displays the message, "A pedestrian may jump out into the road at the intersection 100 meters ahead," and at the same time, the voice assistant notifies the user, "A pedestrian may jump out into the road ahead. Please be careful," prompting the user to immediately slow down.

[0870] Scenario 2: Fatigue level assessment

[0871] If the user continues driving for a long time, the in-car camera will analyze the user's facial expressions and blinking to determine that they are fatigued. In this case, the display will show "You are fatigued. Please take a break" and a voice message will be given saying "We recommend taking a break," encouraging the user to decide to take a break.

[0872] Scenario 3: Route guidance taking traffic congestion into account

[0873] When the user voice-selects "a nearby cafe" as their destination, the server calculates the optimal route based on the latest traffic information and sends it to the device. The device then provides route guidance via voice prompts and a display: "Turn right at the next intersection, then turn left at the next traffic light."

[0874] In this way, the system of the present invention efficiently collects and analyzes information about the driver and the vehicle's surroundings, and provides notifications at the appropriate time, thereby supporting safe and comfortable driving.

[0875] The processing flow will be explained below.

[0876] Step 1:

[0877] The device collects real-time data from on-board cameras and sensors, including the movement of pedestrians, motorcyclists, and bicycles, the status of traffic lights, and the driver's facial expressions and blinking.

[0878] Step 2:

[0879] The terminal transmits the collected data to a server in real time via wireless communication in an appropriate format.

[0880] Step 3:

[0881] The server receives the data sent from the device and inputs it into the generative AI model to analyze the surrounding situation and the driver's state.

[0882] Step 4:

[0883] The server analyzes various factors, such as the approach of pedestrians, motorbikes, and bicycles, and the driver's fatigue level, to generate risk prediction data and data on the driver's condition, such as when there is a high possibility of a pedestrian jumping out into the intersection ahead or when the driver is highly fatigued.

[0884] Step 5:

[0885] The server generates notification data based on the analysis results, such as a warning message such as "A pedestrian may jump out into the street at the intersection 100 meters ahead" or a message encouraging the driver to take a break such as "You are fatigued. Please take a break."

[0886] Step 6:

[0887] The server sends the created notification data to the terminal. The data is designed to be sent quickly.

[0888] Step 7:

[0889] The device displays the received notification data on the in-car display and notifies the driver via the voice assistant. Specifically, the display will show "There is a pedestrian ahead. Please be careful," and at the same time a voice will say, "A pedestrian has jumped out ahead. Please be careful."

[0890] Step 8:

[0891] The device continuously analyzes the driver's facial expressions and blinking to monitor their fatigue level. If it determines that the driver is fatigued, it will notify the driver, saying, "You are fatigued. Please take a break."

[0892] Step 9:

[0893] The server obtains real-time traffic congestion information from external traffic information services and calculates the optimal route to the driver's destination and intermediate stops based on the information obtained.

[0894] Step 10:

[0895] The server sends the calculated route data to the device, which receives it and starts providing guidance to the driver via voice and display, providing specific instructions such as "turn right at the next intersection."

[0896] Through these steps, the system provides support to improve driver safety and efficiency.

[0897] Example 1

[0898] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0899] To improve driver safety and driving efficiency, it is important to collect and analyze information about the driver and the vehicle's surroundings in real time and provide notifications at the appropriate time. However, conventional systems have had problems with insufficient collection and analysis of information, making it difficult to provide notifications at the appropriate time. There are also few systems that can determine the driver's fatigue level in real time and prompt them to take appropriate breaks. Furthermore, there are also insufficient systems that allow the driver to set their destination via voice input or provide optimal routes based on the latest traffic congestion information.

[0900] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0901] In this invention, the server includes means for collecting information about the driver and the surroundings of the vehicle, means for using a generative AI model to analyze the information in real time, means for notifying the driver based on the analysis results, means for analyzing the driver's facial expressions and blinking to determine the level of fatigue, means for notifying the driver to take a break based on the level of fatigue, means for transmitting the collected data to the server in an appropriate format, means for generating a notification message based on the analysis results using the generative AI model, and means for communicating the notification message to the driver via an in-vehicle display or a voice assistant. This makes it possible to improve driver safety, increase driving efficiency, and reduce the burden on the driver.

[0902] "Driver" means a person who drives a vehicle.

[0903] "Vehicle" refers to land transportation such as automobiles and motorcycles.

[0904] "Information" refers to data about the vehicle and its surroundings, such as the driver's facial expressions, blinking, the movements of pedestrians, motorbikes and bicycles, and the status of traffic lights.

[0905] A "generative AI model" refers to an artificial intelligence model that analyzes collected information in real time and generates appropriate notifications and advice for drivers.

[0906] "Notification" refers to warnings and advice given to the driver based on the analysis results.

[0907] A "break notification" refers to the transmission of a message encouraging a driver to take a break when the driver's level of fatigue is judged to be high.

[0908] "Format" refers to the standard or format for properly organizing and converting data.

[0909] An "in-vehicle display" refers to a device installed inside a vehicle that displays various information on a screen.

[0910] "Voice assistant" refers to a system that uses voice to provide notifications and advice to drivers.

[0911] MODE FOR CARRYING OUT THE INVENTION

[0912] The present invention relates to a system that collects information about the driver and the vehicle's surroundings, analyzes it in real time, and notifies the driver. The system aims to improve driver safety and driving efficiency, and has the function of judging the driver's fatigue level and encouraging them to take a break as necessary.

[0913] System Configuration

[0914] The system consists of the following main components:

[0915] 1. Device:

[0916] - The device is installed in the vehicle and collects information about the surrounding area using on-board cameras and various sensors.

[0917] - The device transmits the collected data in the appropriate format to the server in real time.

[0918] - The device displays the notification message received from the server on the in-vehicle display and notifies the driver via the voice assistant.

[0919] 2. Server:

[0920] - The server receives the information sent from the device and analyzes it using the generative AI model.

[0921] - The server generates an appropriate notification message based on the analysis results.

[0922] 3. Notification mechanism:

[0923] - The notification mechanism consists of the device display and voice assistant.

[0924] - The notification mechanism provides real-time warnings and advice to the driver.

[0925] Hardware and software used

[0926] 1. Hardware:

[0927] - In-car camera: A camera that captures the driver's facial expressions, blinking, and surrounding pedestrians and vehicles.

[0928] - Various sensors: Sensors for detecting vehicle movement and surrounding information using accelerometers, gyro sensors, LIDAR, etc.

[0929] - Terminal: A terminal installed in a vehicle that collects and transmits data.

[0930] - Display and voice assistant: Output devices for notifying the driver.

[0931] 2. Software:

[0932] - Generative AI model: An artificial intelligence model for performing data analysis on the server.

[0933] - Data transmission software: Software for transmitting data from the terminal to the server.

[0934] - Notification generation software: Software for generating notification messages based on analysis results.

[0935] Specific examples

[0936] Scenario 1: Risk prediction notification

[0937] When a user approaches a pedestrian intersection while driving, the in-car display suddenly displays the message, "A pedestrian may jump out into the intersection 100 meters ahead," and at the same time, the voice assistant notifies the user, "A pedestrian has jumped out into the intersection ahead. Please be careful." This allows the user to immediately slow down.

[0938] Prompt Sentence: Generate a message to notify the driver of dangerous situations in advance while driving. For example, "A pedestrian may jump out at the intersection 100 meters ahead."

[0939] Scenario 2: Fatigue level assessment

[0940] If the user continues driving for a long time, the in-car camera analyzes facial expressions and blinking to determine that the user is highly fatigued. In this case, the display will show "You are fatigued. Please take a break" and a voice message will be played saying "We recommend taking a break." This will encourage the user to decide to take a break.

[0941] Prompt Sentence: Analyze the fatigue level of the user who drives for a long time and generate a message to prompt appropriate rest. For example, generate "You are tired. Please take a break."

[0942] Scenario 3: Route guidance taking traffic congestion into account

[0943] When the user voice-selects "a nearby cafe" as their destination, the server calculates the optimal route based on the latest traffic information and sends it to the device. The device then provides route guidance via voice prompts and a display: "Turn right at the next intersection, then turn left at the next traffic light."

[0944] Prompt: Calculate the optimal route to reach the specified destination and generate a message containing the instructions. For example, "Turn right at the next intersection, then left at the next traffic light."

[0945] In this way, the system of the present invention efficiently collects and analyzes information about the driver and vehicle's surroundings, and provides notifications at appropriate times to support safe and comfortable driving.

[0946] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0947] Step 1: Data collection

[0948] The device collects data in real time using onboard cameras and various sensors.

[0949] Input: Information captured by the in-vehicle camera, such as the driver's facial expressions and blinking, the movements of pedestrians, motorbikes, and bicycles, and the status of traffic lights. Data on the vehicle's movements and surrounding conditions from sensors.

[0950] Data processing / calculation: Each data point is temporarily stored in a buffer and converted into an easily readable format.

[0951] Output: A collection of recorded data.

[0952] Specific operation: The device uses the onboard camera to capture the driver's facial expressions and blinks, uses accelerometers and gyro sensors to detect vehicle movement, and uses LIDAR to scan the location of surrounding objects and pedestrians.

[0953] Step 2: Send data

[0954] The terminal transmits the collected data to the server in real time.

[0955] Input: Various data stored on the device.

[0956] Data processing / computation: Converting data into an appropriate format (e.g., JSON or XML) and encrypting it.

[0957] Output: The formatted encrypted data.

[0958] Specific operation: The device converts the collected data into an appropriate format, encrypts it for security purposes, and sends it to a server over the Internet.

[0959] Step 3: Data analysis

[0960] The server receives the data sent from the device and analyzes it using a generative AI model.

[0961] Input: Encrypted data sent from the device after format conversion.

[0962] Data processing / calculation: Data is input into a generative AI model to predict when pedestrians will suddenly run out into the road, when motorcycles will suddenly approach, and determine the driver's level of fatigue.

[0963] Output: Analysis results (e.g., prediction of pedestrians running out into the road, determination of driver fatigue level, etc.).

[0964] Specific operation: The server temporarily stores the received data in a database, extracts the necessary data from the database, inputs it into the generative AI model for analysis, and predicts the movements of pedestrians and the state of the driver as a result.

[0965] Step 4: Generate notification information

[0966] The server generates a message to notify the driver based on the analysis results.

[0967] Input: Analysis results from the generative AI model.

[0968] Data processing / calculation: Based on the analysis results, an appropriate notification message is generated for the driver.

[0969] Output: The generated notification message.

[0970] Specific operation: The server generates an appropriate notification message based on the analysis results obtained by the generative AI model. For example, it creates a message such as "A pedestrian may jump out into the street at the intersection 100 meters ahead."

[0971] Step 5: Notification

[0972] The server sends the created notification data to the terminal, which then notifies the driver via the in-car display and voice assistant.

[0973] Input: The generated notification message.

[0974] Data processing / calculation: Converts the notification message into an appropriate format and sends it to the terminal.

[0975] Output: Notification messages sent to the terminal.

[0976] Specific operation: The server sends the generated notification message to the device, and the device displays the received notification data on the in-car display and notifies the driver via the voice assistant. For example, the display may display "There is a pedestrian ahead. Please be careful" and a voice message may also be issued saying "There is a pedestrian ahead. Please be careful."

[0977] In this way, through the specific operations of each step, the system can provide a safe driving environment for the driver and improve driving efficiency.

[0978] (Application example 1)

[0979] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0980] In today's modern transportation society, there is a demand for improved driver safety and driving efficiency. In particular, the risk of accidents is increasing due to fatigue from long periods of driving and inattention to surrounding traffic conditions. Therefore, technology is needed to enable drivers to continue driving safely and efficiently. Another challenge is to monitor and analyze traffic conditions and driver status in real time and provide notifications at the appropriate time.

[0981] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0982] In this invention, the server includes means for collecting information about the driver and the surrounding area of ​​the vehicle, means for using a generative AI model to analyze the information in real time, means for notifying the driver based on the analysis results, means for analyzing the driver's facial expressions and blinking to determine the level of fatigue, means for issuing a notification encouraging the driver to take a break based on the level of fatigue, means for monitoring the movements of nearby pedestrians, motorbikes, and bicycles and the status of traffic lights to determine the level of danger, and means for issuing notifications via an in-vehicle display or voice assistant. This enables safe and efficient driving by analyzing the driver's level of fatigue and traffic conditions in real time and accurately notifying the driver of necessary information.

[0983] A "driver" is a person who drives a vehicle and is the entity that performs driving duties.

[0984] "Vehicle" means a means of transportation such as an automobile or motorcycle designed and manufactured for travel on land.

[0985] "Means of collecting information" refers to devices and technologies that use on-board cameras and various sensors to detect the situation around the driver and vehicle.

[0986] A "generative AI model" is an artificial intelligence model that uses algorithms learned from large amounts of data to solve specific problems.

[0987] "Means of notification" refers to devices and technologies such as displays and voice assistants that convey analysis results to the driver.

[0988] "Means for analyzing facial expressions and blinking" refers to technology that captures the driver's face in real time using an in-car camera or other device, and analyzes facial expressions and blink frequency using specific algorithms.

[0989] The "means for determining fatigue level" refers to technology and equipment for estimating a driver's fatigue level based on analyzed facial expressions and blinking data.

[0990] "Means for notifying drivers to take a break" refers to display or audio notification technology that advises drivers to take a break when it is determined that the driver is highly fatigued.

[0991] "Means for monitoring the movement of pedestrians, motorbikes, and bicycles" refers to devices and technologies that use on-board sensors and cameras to detect the position and movement of objects in real time.

[0992] "Means for monitoring the status of traffic lights" refers to technology that uses in-vehicle cameras and sensors to detect the color and lighting status of traffic lights in real time.

[0993] "Means for determining the level of danger" refers to technology and devices that use generative AI models to analyze surrounding information and the driver's condition and estimate potential danger.

[0994] An "in-vehicle display" is a display device installed inside a vehicle that displays driving information and warnings.

[0995] A "voice assistant" is a device equipped with voice recognition and voice synthesis technology that interacts with the driver based on voice input and provides necessary notifications and information.

[0996] The present invention relates to a system that collects information about the driver and the vehicle's surroundings, analyzes it in real time, and notifies the driver. The system aims to improve driver safety and driving efficiency, and has functions such as judging the driver's fatigue level and encouraging them to take a break as necessary.

[0997] System Configuration

[0998] The system of the present invention consists of the following main components:

[0999] 1. Device:

[1000] The car uses on-board cameras and various sensors to collect information about the surrounding area. Specifically, the cameras capture the driver's facial expressions and blinks, while the sensors monitor nearby pedestrians, motorbikes, bicycles, and traffic lights.

[1001] 2. Server:

[1002] The system receives information sent from the device and analyzes it using a generative AI model. Specifically, the system analyzes the data using a trained AI model (using frameworks such as Google TensorFlow or PyTorch). The analysis results include predictions of pedestrians jumping out at specific locations, situations in which a motorcycle is suddenly approaching, and the driver's fatigue level.

[1003] 3. Notification mechanism:

[1004] This includes a display and voice assistant to notify the driver based on the analysis results. For example, necessary information can be immediately conveyed to the driver by displaying a warning on the in-car display or using a voice assistant (general name).

[1005] Program processing

[1006] Hardware:

[1007] In-vehicle camera: Captures the driver's facial expressions and the surrounding environment in real time.

[1008] Sensors: Capture surrounding activity (e.g. LIDAR, radar).

[1009] In-car display: Show notifications.

[1010] Voice assistant: Provides voice notifications.

[1011] software:

[1012] Generative AI models: Use frameworks such as Google TensorFlow and PyTorch.

[1013] Data analysis platform: Apache Kafka (data streaming), Apache Hadoop (data storage).

[1014] Specific examples

[1015] Scenario 1: Fatigue assessment while driving

[1016] The device captures the driver's facial expressions and blinks in real time, and the server analyzes them using a generative AI model. If the driver is judged to be fatigued, a notification will be sent saying, "You are fatigued. Please take a break."

[1017] Scenario 2: Risk prediction notification

[1018] The device collects information on the movements of pedestrians, motorbikes, and bicycles in the vicinity, as well as the status of traffic lights, which the server analyzes using a generative AI model. If a dangerous situation is predicted, a notification will be sent to the user, stating, "There is a possibility that a pedestrian may suddenly jump out at the intersection 100 meters ahead."

[1019] Scenario 3: Route guidance taking traffic congestion into account

[1020] When the user voice-selects a destination as "a nearby cafe," the server calculates the optimal route based on the latest traffic information, and the device then provides route guidance through voice prompts and a display, such as "Turn right at the next intersection and left at the next traffic light."

[1021] Prompt Sentence Examples

[1022] "The system analyzes the driver's real-time facial expressions and blinking data to determine their fatigue level. If the level of fatigue is high, the system will notify the driver by voice, saying, 'We recommend you take a break.'"

[1023] As described above, the present invention is a system that efficiently collects and analyzes information about the driver and the vehicle's surroundings using an in-vehicle camera and various sensors, thereby improving the driver's safety and driving efficiency.

[1024] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1025] Step 1: Data collection

[1026] The device uses on-board cameras and various sensors to capture the driver's facial expressions and eye blinks, as well as the status of surrounding pedestrians, motorcyclists, bicycles, and traffic lights in real time. Input data includes video feeds and sensor data. As an output, this data is pre-processed and converted into a format for analysis.

[1027] Step 2: Send data

[1028] The device sends pre-processed data to the server in real time. The input data includes captured video feeds and sensor data. These data are converted into an appropriate data format (e.g., JSON, Protobuf) and sent to the server. The output is the data packets arriving at the server.

[1029] Step 3: Data analysis

[1030] The server analyzes the received data using a generative AI model. Input data includes the video feed and sensor data sent from the device. The analysis results include the driver's fatigue level, the movement of nearby pedestrians, the approach of motorcycles and bicycles, and the status of traffic lights. The output is the analyzed results data.

[1031] Step 4: Generate notification information

[1032] The server generates notification and advice messages for the driver based on the analysis results. The input data includes the analysis results data. The generated notification messages include warning information such as "There is a risk of a pedestrian jumping out at the intersection 100 meters ahead" or "You are fatigued. Please take a break." The output is a specific notification message.

[1033] Step 5: Notification

[1034] The server sends the created notification data to the terminal, which displays it on the in-car display and notifies the driver via the voice assistant. The input data includes the notification message sent from the server. Specific actions include displaying a warning on the display and notifying the driver by voice, "There is a pedestrian ahead. Please be careful." The output is the notification being conveyed to the driver.

[1035] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1036] This invention combines a system that collects information about the driver and vehicle's surroundings and analyzes it in real time with an emotion engine that recognizes the driver's emotions. This system aims to improve driver safety and driving efficiency, and has the function of analyzing the driver's emotional state and providing appropriate advice and notifications accordingly.

[1037] System Configuration

[1038] The system of the present invention consists of the following main components:

[1039] 1. Terminal: Installed in the vehicle, it uses onboard cameras and various sensors to collect information about the surrounding area, as well as the driver's facial expressions and voice data.

[1040] 2. Server: Receives information sent from the device and analyzes it using the generative AI model and emotion engine.

[1041] 3. Notification mechanism: Display and voice assistant to notify the driver based on the analysis results.

[1042] 4. Emotion engine: Analyzes the driver's facial expressions and voice data to recognize emotions.

[1043] Program processing

[1044] The program processing in this system is carried out in the following manner.

[1045] Data collection and transmission

[1046] The device collects data in real time from on-board cameras and sensors. The collected data includes the movement of pedestrians, motorbikes, and bicycles, the status of traffic lights, and the driver's facial expressions and voice data. The device then transmits the collected data to a server. The data is transmitted in real time in an appropriate format.

[1047] Data analysis and emotion recognition

[1048] The server receives the data sent from the device and uses the generative AI model to analyze the surrounding situation and the driver's state. Furthermore, the emotion engine analyzes the driver's facial expressions and voice data to recognize the driver's emotional state. For example, it determines whether the driver is stressed or relaxed.

[1049] Generate and send notification information

[1050] The server generates notification data based on the analysis results and emotion recognition results. For example, it creates a warning message such as "A pedestrian may jump out into the intersection 100 meters ahead" or an advice message based on the driver's emotion, such as "Please concentrate on driving." The server then sends the created notification data to the device.

[1051] Notification implementation

[1052] The device displays the received notification data on the in-car display and notifies the driver via the voice assistant. Specifically, the display will show "There is a pedestrian ahead. Please be careful," and at the same time a voice message will say, "A pedestrian has jumped out ahead. Please be careful." Advice based on the driver's emotions will also be displayed and spoken.

[1053] Specific examples

[1054] Scenario 1: Risk prediction notification

[1055] When a user approaches a pedestrian intersection ahead while driving, the in-car display suddenly displays the message, "A pedestrian may jump out at the intersection 100 meters ahead," and at the same time, the voice assistant notifies the user, "A pedestrian has jumped out ahead. Please be careful." This allows the user to immediately slow down and prevent an accident.

[1056] Scenario 2: Fatigue and Emotion Recognition

[1057] If the user continues driving for a long time, the in-car camera analyzes facial expressions and blinking to determine a high level of fatigue, and the emotion engine determines the driver's stress level. In this case, the system displays a message saying "You are fatigued. Please take a break" and also provides a voice message saying "We recommend taking a break." If the stress level is high, the system will also provide a notification suggesting relaxation methods.

[1058] Scenario 3: Emotional Advice

[1059] If the emotion engine determines that the user is becoming irritated while driving, the server will detect this and provide visual and audio advice such as "Please relax while driving." At the same time, it is also possible to provide car music and relaxation guides.

[1060] In this way, the system of the present invention supports safe and comfortable driving by efficiently collecting and analyzing information about the driver and the vehicle's surroundings and notifying them at the appropriate time. It also aims to achieve even safer driving by recognizing the driver's emotional state and providing advice and support accordingly.

[1061] The processing flow will be explained below.

[1062] Step 1:

[1063] The device collects real-time data from onboard cameras and sensors, including the movement of pedestrians, motorcyclists, and bicycles, the status of traffic lights, and the driver's facial expressions and voice data.

[1064] Step 2:

[1065] The terminal transmits the collected data to a server in real time via wireless communication in an appropriate format.

[1066] Step 3:

[1067] The server receives the data sent from the device and inputs it into the generative AI model to analyze the surrounding situation and the driver's state.

[1068] Step 4:

[1069] The server analyzes various factors, such as the approach of pedestrians, motorbikes, and bicycles, and the driver's fatigue level. For example, it analyzes the prediction of pedestrians jumping out at a specific location, the situation of a motorbike suddenly approaching, the driver's facial expressions, and the frequency of blinking to assess the driver's fatigue level.

[1070] Step 5:

[1071] The server generates risk prediction data from the analysis results. For example, if there is a high possibility that a pedestrian will suddenly jump out into the intersection ahead, it will generate the necessary warning message.

[1072] Step 6:

[1073] The server uses facial expressions and voice data to recognize the driver's emotions through an emotion engine, for example, determining whether the driver is stressed or relaxed.

[1074] Step 7:

[1075] The server generates appropriate advice and notification messages for the driver based on the analysis and emotion recognition results, such as "A pedestrian may jump out into the road at the intersection 100 meters ahead" or "Please concentrate on driving."

[1076] Step 8:

[1077] The server sends the created notification data to the terminal. The data is designed to be sent quickly.

[1078] Step 9:

[1079] The device displays the received notification data on the in-car display and notifies the driver via the voice assistant. Specifically, the display will show "There is a pedestrian ahead. Please be careful," and at the same time a voice will say, "A pedestrian has jumped out ahead. Please be careful."

[1080] Step 10:

[1081] The device continuously monitors the driver's facial expressions and blinking, and evaluates the driver's emotional state using an emotion engine. If the device determines that the driver is fatigued, it will notify the driver, saying, "You are fatigued. Please take a break." If the driver's stress level is high, it will also notify the driver and suggest relaxation methods.

[1082] Step 11:

[1083] The server obtains real-time traffic congestion information from external traffic information services and calculates the optimal route to the driver's destination and intermediate stops based on the information obtained.

[1084] Step 12:

[1085] The server sends the calculated route data to the device, which receives it and starts providing guidance to the driver via voice and display, providing specific instructions such as "turn right at the next intersection."

[1086] In this way, the system of the present invention analyzes various information in real time and notifies drivers at the appropriate time to improve driver safety and efficiency. In addition, by combining it with an emotion engine, it provides advice according to the driver's emotional state, aiming to further safer driving and reduce driver stress.

[1087] Example 2

[1088] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1089] In modern society, ensuring safe and comfortable driving requires accurate understanding of the driver's situation and emotional state, and providing appropriate advice and notifications accordingly. Conventional systems lack the ability to accurately recognize the driver's emotions and provide notifications in real time, creating the risk that driver stress and fatigue could impede safe driving. Furthermore, technology for quickly analyzing surrounding information and providing appropriate advice and warnings is also not yet mature. This makes it difficult to ensure the safety of the driver and other road users.

[1090] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1091] In this invention, the server includes a means for collecting information about the driver and the vehicle's surroundings, a means for analyzing the information in real time using a generative AI model, a means for analyzing the driver's facial expressions and voice data to recognize the driver's emotional state, a means for providing advice or notification based on the emotional state, and a means for notifying the driver based on the analysis results. This allows the server to accurately recognize the driver's emotional state and provide appropriate advice or warnings accordingly. It also efficiently analyzes information about the surrounding environment to support safe driving.

[1092] "Means for collecting information about the driver and the vehicle's surroundings" refers to devices and functions that use on-board cameras, microphones, and various sensors to collect data on the driver's condition and the vehicle's surrounding environment in real time.

[1093] "Means using generative AI models for real-time analysis" refers to technologies that use generative AI models to instantly analyze collected data and understand the surrounding environment and the driver's situation.

[1094] "Means for notifying the driver based on the analysis results" refers to devices or functions for providing appropriate notifications and advice to the driver based on the analysis results of the generative AI model.

[1095] "Means for analyzing the driver's facial expressions and voice data to recognize the emotional state" refers to technology or devices that analyze the driver's facial expressions and voice data and recognize the driver's emotional state (e.g., stress, relaxation, etc.) from that information.

[1096] "Means for providing advice or notifications based on emotional state" refers to a function or device that provides appropriate advice or warnings to the driver in real time based on the driver's recognized emotional state.

[1097] "Means for setting the driver's destination and intermediate points by voice" refers to technology or devices that allow the driver to set the destination and intermediate points using voice commands.

[1098] "Means of obtaining real-time traffic congestion information and calculating the optimal route" refers to technology or devices that obtain current traffic conditions and traffic congestion information in real time and calculate the optimal driving route based on that information.

[1099] "Means for providing audio and visual guidance to the driver of the calculated route" refers to a function or device that provides audio and visual guidance to the driver of the calculated driving route.

[1100] "Means including pedestrian, motorbike, and bicycle information" refers to technology or devices that can detect the presence of and collect information about pedestrians, motorbikes, and bicycles moving around the vehicle.

[1101] "Means for notifying the driver of warnings in real time via an in-vehicle display and voice assistant" refers to a function or device that notifies the driver of warning messages in real time using an in-vehicle display and voice assistant.

[1102] The present invention is a system that efficiently collects and analyzes information about the driver and the vehicle's surroundings, and recognizes the driver's emotional state to support safe and comfortable driving. Specific embodiments of the present invention are described below.

[1103] System Configuration

[1104] The system of the present invention consists of the following main components:

[1105] 1. Terminal

[1106] 2. Server

[1107] 3. Means of notification

[1108] 4. Emotion Engine

[1109] Terminal

[1110] The device is installed in the vehicle and uses an onboard camera, microphone, and various sensors to collect information about the surrounding area, as well as the driver's facial expression and voice data. The collected data includes the movements of pedestrians, motorbikes, and bicycles, the status of traffic lights, and the driver's facial expression and voice data. The device transmits this data to a server in an appropriate format (e.g., JSON format).

[1111] server

[1112] The server receives the data sent from the device and uses a generative AI model to analyze the surrounding situation and the driver's state. This analysis includes object recognition technology and traffic light status analysis. It also uses an emotion engine to analyze the driver's facial expressions and voice data to recognize the driver's emotional state. For example, it determines whether the driver is stressed or relaxed.

[1113] Notification means

[1114] The notification means include an in-vehicle display and a voice assistant. The server generates notification information based on the analysis results and emotion recognition results and sends it to the terminal. The terminal displays the received notification information on the in-vehicle display and notifies the driver via the voice assistant.

[1115] Emotion Engine

[1116] The emotion engine analyzes the driver's facial expressions and voice data to recognize their emotional state, for example, determining their stress level and fatigue level from eye movements, blinking frequency, and tone of voice.

[1117] Specific examples

[1118] Scenario 1: Risk prediction notification

[1119] When a user approaches a pedestrian intersection ahead while driving, the device sends video footage captured by the onboard camera to a server. The server analyzes this data and detects the possibility of a pedestrian jumping out at the intersection 100 meters ahead. The analysis results are generated as notification information and sent to the device. The device receives this information and displays "A pedestrian may jump out at the intersection 100 meters ahead" on the in-car display, and the voice assistant also notifies the user. This allows the user to immediately slow down and prevent an accident.

[1120] Scenario 2: Fatigue and Emotion Recognition

[1121] When a user continues driving for a long period of time, the in-car camera captures the user's facial expressions and blinking in real time and sends the data to a server. The server analyzes this data and determines that the level of fatigue is high. At the same time, the emotion engine determines the driver's stress level. Based on this result, the server generates a message saying, "You are fatigued. Please take a break," and sends it to the device. The device then displays a notification on the in-car display and also issues a voice message saying, "We recommend that you take a break."

[1122] Scenario 3: Emotional Advice

[1123] If the emotion engine determines that the user is becoming irritated while driving, the server detects this and generates advice such as "Please relax while driving." At the same time, it also creates relaxing car music and a relaxation guide and sends these to the device. The device then notifies the user of this information via the display and voice assistant.

[1124] Prompt Sentence Examples

[1125] Please explain the specific processing flow of a safety drive system that analyzes the driver's emotions and provides appropriate advice.

[1126] In this way, the system of the present invention supports safe and comfortable driving by efficiently collecting and analyzing information about the driver and vehicle's surroundings and notifying them at the appropriate time. Furthermore, it aims to achieve even safer driving by recognizing the driver's emotional state and providing advice and support accordingly.

[1127] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1128] Step 1: Data collection

[1129] The device uses onboard cameras, microphones, and various sensors to collect real-time information about the driver and the vehicle's surroundings.

[1130] Input: Driver's facial expression, voice data, video footage of the vehicle's surroundings, sensor data (vehicle speed, brake status, etc.)

[1131] Data processing: Capture camera footage frame by frame, record audio as sample data, and log sensor data by time.

[1132] Output: Collected real-time data (camera footage, audio data, sensor data)

[1133] Step 2: Send data

[1134] The device sends the collected data to the server in an appropriate format (e.g., JSON format).

[1135] Input: Collected real-time data (camera footage, audio data, sensor data)

[1136] Data processing: Convert the data into JSON format, split it into packets, and prepare it for transmission.

[1137] Output: JSON formatted data packet

[1138] Step 3: Receiving data

[1139] The server receives the data packets sent from the terminal.

[1140] Input: JSON formatted data packet

[1141] Data processing: Reconstructing received packets and restoring them to their original data format.

[1142] Output: Recovered real-time data (camera video, audio data, sensor data)

[1143] Step 4: Data analysis

[1144] The server uses a generative AI model to analyze the data it receives, specifically using object recognition technology to detect the presence of pedestrians, motorbikes, and bicycles, as well as analyze the status of traffic lights.

[1145] Input: Recovered real-time data (camera video, audio data, sensor data)

[1146] Data Computation: Uses generative AI models to perform object recognition and analyze traffic light status.

[1147] Output: Analysis results (presence of pedestrians, traffic light status, etc.)

[1148] Step 5: Emotion Recognition

[1149] The server uses an emotion engine to analyze the driver's facial expressions and voice data to recognize their emotional state.

[1150] Input: Driver's facial expression data and voice data

[1151] Data Computation: Using the emotion engine, facial and vocal features are extracted to classify emotional states.

[1152] Output: Emotion recognition results (e.g., stress, relaxation, fatigue, etc.)

[1153] Step 6: Generate notification information

[1154] The server generates notification information based on the analysis and emotion recognition results, such as a warning message such as "A pedestrian may jump out into the road at the intersection 100 meters ahead" or an advice message such as "Please concentrate on driving."

[1155] Input: Data analysis results, emotion recognition results

[1156] Data processing: Generate appropriate messages based on analysis results and emotion recognition results

[1157] Output: Notification information (warning messages, advice messages, etc.)

[1158] Step 7: Send notification data

[1159] The server transmits the generated notification information to the terminal.

[1160] Input: Notification information

[1161] Data processing: The notification message is converted into a data packet and prepared for transmission.

[1162] Output: Notification data packet

[1163] Step 8: Notification

[1164] The device displays the received notification information on the in-car display and notifies the driver via the voice assistant.

[1165] Input: Notification data packet

[1166] Data processing: Analyzes the notification data packet and converts it into a format for display and audio output.

[1167] Output: Warning message on the display, notification via voice assistant

[1168] Specifically, when a driver approaches a pedestrian intersection ahead, the camera image is sent to the server, which then generates an analysis result as notification information and sends it to the device. The device receives this information and issues a warning on the display and via the voice assistant. This series of steps allows the user to take appropriate action.

[1169] (Application example 2)

[1170] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1171] In recent years, as autonomous vehicles have become more widely used, there has been a demand for understanding the driver's fatigue level and emotional state to provide a safe and comfortable driving experience. However, conventional technologies lack the means to accurately analyze the driver's emotional state in real time and provide appropriate notifications and advice accordingly. Furthermore, systems that can instantly notify the driver using mobile devices such as smartphones and smart glasses have not been fully established. As a result, there have been concerns about reduced driver safety and driving efficiency.

[1172] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1173] In this invention, the server includes: means for collecting information about the driver and the surroundings of the vehicle; means for using a generative AI model to analyze the information in real time; means for notifying the driver based on the analysis results; means for analyzing the driver's facial expressions and blinking to determine the driver's fatigue level; means for notifying the driver to take a break based on the fatigue level; means for recognizing the driver's emotional state and providing appropriate advice or notification based on the emotional state; and means for providing notifications to the driver using a mobile device such as a smartphone or smart glasses. This makes it possible to analyze the driver's emotional state in real time and notify the driver at the appropriate time, thereby supporting safe and smooth driving.

[1174] "Driver" means a person who operates a vehicle.

[1175] "Vehicle" means a moving object designed to travel on roads.

[1176] "Surrounding information" refers to all data about the vehicle's external environment, including pedestrians, other vehicles, bicycles, and traffic light status.

[1177] A "generative AI model" is an algorithm that uses artificial intelligence to analyze input data and generate output results.

[1178] "Notifications" are messages or alerts that convey specific information to the driver.

[1179] "Facial expressions" refer to facial movements and states that indicate a person's emotions and reactions.

[1180] Blinking is the act of closing and opening the eyes.

[1181] "Fatigue level" is an index that indicates the degree of fatigue experienced by a driver.

[1182] "Emotional state" refers to the psychological state or mood exhibited by the driver.

[1183] "Advice" means guidance or suggestions provided to a driver.

[1184] "Means for collecting information about the surrounding area" refers to devices and methods for acquiring data about the vehicle's external environment using on-board cameras, sensors, etc.

[1185] "Real-time analytical means" refers to devices or methods for processing collected data in real time and generating results.

[1186] A "smartphone" is a mobile phone equipped with internet connectivity and a camera.

[1187] "Smart glasses" are glasses-type devices that can display information in real time.

[1188] "Mobile devices" is a general term for portable electronic devices, including smartphones, tablets, and smart glasses.

[1189] "Relaxation music" is music that helps drivers relax.

[1190] "Real-time scene analysis" is a technology that instantly analyzes video data around the vehicle to grasp the current situation.

[1191] MODE FOR CARRYING OUT THE INVENTION

[1192] System Overview

[1193] This invention is a system that analyzes the driver's fatigue level and emotional state in real time and provides appropriate notifications and advice. The system components include:

[1194] 1. Device: A mobile device such as a smartphone or smart glasses that uses cameras and various sensors to collect data from the driver and the vehicle's surrounding environment.

[1195] 2. Server: Used to analyze collected data in real time, utilizing generative AI models and emotion engines for processing.

[1196] 3. Notification mechanism: Includes a display and voice assistant to notify the driver in a timely manner based on the analysis results.

[1197] Program processing

[1198] Data collection and transmission

[1199] The device uses a camera and various sensors to collect the driver's facial expressions, blinking, and information about the vehicle's surroundings (pedestrians, bicycles, other vehicles, etc.) in real time. The collected data is sent to a server in an appropriate format.

[1200] Data analysis and emotion recognition

[1201] The server uses a generative AI model to analyze the received data. It also uses an emotion engine to analyze facial expressions and voice data. This analysis identifies the driver's fatigue level and emotional state (e.g., stressed, relaxed, fatigued).

[1202] Generate and send notification information

[1203] The server generates notification data based on the analysis results. For example, if danger is imminent, a warning message such as "There is a pedestrian ahead. Please be careful" is generated. Advice based on the driver's emotional state (for example, "Please relax. We will play car music") is also generated. The generated notification data is sent to the device.

[1204] Notification implementation

[1205] The device then displays the received notification data on the display of a smartphone or smart glasses and notifies the driver via a voice assistant. The notification content includes information about nearby dangers and advice tailored to the driver's emotional state.

[1206] Hardware and software used

[1207] Camera: Cameras built into smartphones or smart glasses are used to collect data on the driver's facial expressions and the surrounding environment.

[1208] Generative AI models: Analyze the collected data using models built with machine learning libraries such as TensorFlow and Keras.

[1209] Emotion Engine: Uses the dlib library to detect facial landmarks and recognize emotions.

[1210] Audio notification: Uses the playsound library to provide audio alerts.

[1211] Specific examples

[1212] Scenario 1: Risk prediction notification

[1213] When a driver approaches an intersection ahead, the camera in the smart glasses detects pedestrians and the server notifies the driver, saying, "There is a pedestrian ahead. Please be careful." The warning is conveyed to the driver via the display and voice assistant, preventing accidents from occurring.

[1214] Scenario 2: Advice based on emotional state

[1215] If the driver shows an emotional state of "anger," the smartphone will notify the driver, "Relax. Car music will be played," and will automatically play car music.

[1216] Prompt Sentence Examples

[1217] Text format

[1218] I want to capture facial expressions from camera data on smart glasses and perform emotion recognition. If the result is "anger," I want to display "Please be careful" and play relaxation music.

[1219] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1220] Step 1:

[1221] The device uses a camera built into a smartphone or smart glasses to collect the driver's facial expressions, blinking, and information about the vehicle's surroundings (pedestrians, bicycles, other vehicles, etc.) in real time. This data includes images of the driver's facial expressions, video information about the surrounding environment, and audio data. The collected data is converted into an appropriate format as digital data and sent to a server in real time.

[1222] Step 2:

[1223] The server receives the data sent from the device. This data includes images of the driver's facial expressions and video information of the surrounding environment. To analyze the received data, the server preprocesses the data using a generative AI model. For example, it extracts the driver's facial area using a facial recognition algorithm and performs preprocessing to input the data into the emotion engine.

[1224] Step 3:

[1225] The server performs emotion recognition on the facial expression images using an emotion engine. The emotion engine uses the dlib library to detect facial landmarks (eyes, mouth, nose, etc.) and estimate the emotional state based on them. This process includes facial feature point detection, feature extraction, and emotion classification data computation. The output is the driver's emotional state (e.g., anger, sadness, joy, relaxation).

[1226] Step 4:

[1227] The server uses a generative AI model to analyze surrounding environment data and generate notification information for the driver. For example, it analyzes the location data of pedestrians, bicycles, and other vehicles, and generates a warning message such as "There is a pedestrian ahead. Please be careful" if a danger is imminent. It then performs appropriate data processing and calculations and creates a notification message as output.

[1228] Step 5:

[1229] The server generates notification data at the optimal timing for the driver based on the emotion recognition results and the analysis of the surrounding environment. For example, if the driver's emotional state is "anger," it generates an advice message such as "Please relax. We will play car music." This notification data is converted into an appropriate format and sent to the device.

[1230] Step 6:

[1231] The device displays the received notification data on the display of a smartphone or smart glasses and notifies the driver via a voice assistant. For example, a warning message such as "There is a pedestrian ahead. Please be careful" is displayed on the display and notified by voice. Relaxation music corresponding to the driver's emotional state is also automatically played on the smartphone.

[1232] Step 7:

[1233] The user takes appropriate driving actions based on notifications and advice from the device. For example, they may check for pedestrians ahead and slow down for safety. They may also listen to relaxation music to ensure a relaxed driving experience. In this way, the system supports the user's safe and comfortable driving.

[1234] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1235] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1236] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1237] [Fourth embodiment]

[1238] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1239] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1240] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1241] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1242] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1243] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1244] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1245] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1246] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1247] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1248] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1249] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1250] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1251] The present invention relates to a system that collects information about the driver and the vehicle's surroundings, analyzes it in real time, and notifies the driver. The system aims to improve driver safety and driving efficiency, and has functions such as judging the driver's fatigue level and encouraging them to take a break as necessary.

[1252] System Configuration

[1253] The system of the present invention consists of the following main components:

[1254] 1. Terminal: Installed in the vehicle, it collects information about the surrounding area using on-board cameras and various sensors.

[1255] 2. Server: Receives information sent from the device and analyzes it using a generative AI model.

[1256] 3. Notification mechanism: Display and voice assistant to notify the driver based on the analysis results.

[1257] Program processing

[1258] The program processing in this system is carried out in the following manner.

[1259] 1. Data Collection:

[1260] The device collects real-time data from on-board cameras and sensors, including the movement of pedestrians, motorcyclists, and cyclists, traffic light status, and the driver's facial expressions and eye blinks.

[1261] 2. Data transmission:

[1262] The terminals transmit the collected data to the server in real time in an appropriate format.

[1263] 3. Data Analysis:

[1264] The server receives the data sent from the device and analyzes it using a generative AI model. The analysis results include predictions of pedestrians running out into the road at a specific location, situations in which a motorbike is approaching suddenly, and the driver's fatigue level.

[1265] 4. Notification information generation:

[1266] Based on the analysis results, the server generates appropriate notification and advice messages for the driver, such as "There is a risk of a pedestrian jumping out into the road at the intersection 100 meters ahead" or "A motorcycle is rapidly approaching from behind."

[1267] 5. Notification Implementation:

[1268] The server then sends the created notification data to the device, which then displays it on the in-car display and notifies the driver via the voice assistant. For example, a warning may be displayed on the screen and a voice message may be heard saying, "There is a pedestrian ahead. Please be careful."

[1269] Specific examples

[1270] Scenario 1: Risk prediction notification

[1271] When a user approaches a pedestrian intersection ahead while driving, the in-car display suddenly displays the message, "A pedestrian may jump out into the road at the intersection 100 meters ahead," and at the same time, the voice assistant notifies the user, "A pedestrian may jump out into the road ahead. Please be careful," prompting the user to immediately slow down.

[1272] Scenario 2: Fatigue level assessment

[1273] If the user continues driving for a long time, the in-car camera will analyze the user's facial expressions and blinking to determine that they are fatigued. In this case, the display will show "You are fatigued. Please take a break" and a voice message will be given saying "We recommend taking a break," encouraging the user to decide to take a break.

[1274] Scenario 3: Route guidance taking traffic congestion into account

[1275] When the user voice-selects "a nearby cafe" as their destination, the server calculates the optimal route based on the latest traffic information and sends it to the device. The device then provides route guidance via voice prompts and a display: "Turn right at the next intersection, then turn left at the next traffic light."

[1276] In this way, the system of the present invention efficiently collects and analyzes information about the driver and the vehicle's surroundings, and provides notifications at the appropriate time, thereby supporting safe and comfortable driving.

[1277] The processing flow will be explained below.

[1278] Step 1:

[1279] The device collects real-time data from on-board cameras and sensors, including the movement of pedestrians, motorcyclists, and bicycles, the status of traffic lights, and the driver's facial expressions and blinking.

[1280] Step 2:

[1281] The terminal transmits the collected data to a server in real time via wireless communication in an appropriate format.

[1282] Step 3:

[1283] The server receives the data sent from the device and inputs it into the generative AI model to analyze the surrounding situation and the driver's state.

[1284] Step 4:

[1285] The server analyzes various factors, such as the approach of pedestrians, motorbikes, and bicycles, and the driver's fatigue level, to generate risk prediction data and data on the driver's condition, such as when there is a high possibility of a pedestrian jumping out into the intersection ahead or when the driver is highly fatigued.

[1286] Step 5:

[1287] The server generates notification data based on the analysis results, such as a warning message such as "A pedestrian may jump out into the street at the intersection 100 meters ahead" or a message encouraging the driver to take a break such as "You are fatigued. Please take a break."

[1288] Step 6:

[1289] The server sends the created notification data to the terminal. The data is designed to be sent quickly.

[1290] Step 7:

[1291] The device displays the received notification data on the in-car display and notifies the driver via the voice assistant. Specifically, the display will show "There is a pedestrian ahead. Please be careful," and at the same time a voice will say, "A pedestrian has jumped out ahead. Please be careful."

[1292] Step 8:

[1293] The device continuously analyzes the driver's facial expressions and blinking to monitor their fatigue level. If it determines that the driver is fatigued, it will notify the driver, saying, "You are fatigued. Please take a break."

[1294] Step 9:

[1295] The server obtains real-time traffic congestion information from external traffic information services and calculates the optimal route to the driver's destination and intermediate stops based on the information obtained.

[1296] Step 10:

[1297] The server sends the calculated route data to the device, which receives it and starts providing guidance to the driver via voice and display, providing specific instructions such as "turn right at the next intersection."

[1298] Through these steps, the system provides support to improve driver safety and efficiency.

[1299] Example 1

[1300] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1301] To improve driver safety and driving efficiency, it is important to collect and analyze information about the driver and the vehicle's surroundings in real time and provide notifications at the appropriate time. However, conventional systems have had problems with insufficient collection and analysis of information, making it difficult to provide notifications at the appropriate time. There are also few systems that can determine the driver's fatigue level in real time and prompt them to take appropriate breaks. Furthermore, there are also insufficient systems that allow the driver to set their destination via voice input or provide optimal routes based on the latest traffic congestion information.

[1302] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1303] In this invention, the server includes means for collecting information about the driver and the surroundings of the vehicle, means for using a generative AI model to analyze the information in real time, means for notifying the driver based on the analysis results, means for analyzing the driver's facial expressions and blinking to determine the level of fatigue, means for notifying the driver to take a break based on the level of fatigue, means for transmitting the collected data to the server in an appropriate format, means for generating a notification message based on the analysis results using the generative AI model, and means for communicating the notification message to the driver via an in-vehicle display or a voice assistant. This makes it possible to improve driver safety, increase driving efficiency, and reduce the burden on the driver.

[1304] "Driver" means a person who drives a vehicle.

[1305] "Vehicle" refers to land transportation such as automobiles and motorcycles.

[1306] "Information" refers to data about the vehicle and its surroundings, such as the driver's facial expressions, blinking, the movements of pedestrians, motorbikes and bicycles, and the status of traffic lights.

[1307] A "generative AI model" refers to an artificial intelligence model that analyzes collected information in real time and generates appropriate notifications and advice for drivers.

[1308] "Notification" refers to warnings and advice given to the driver based on the analysis results.

[1309] A "break notification" refers to the transmission of a message encouraging a driver to take a break when the driver's level of fatigue is judged to be high.

[1310] "Format" refers to the standard or format for properly organizing and converting data.

[1311] An "in-vehicle display" refers to a device installed inside a vehicle that displays various information on a screen.

[1312] "Voice assistant" refers to a system that uses voice to provide notifications and advice to drivers.

[1313] MODE FOR CARRYING OUT THE INVENTION

[1314] The present invention relates to a system that collects information about the driver and the vehicle's surroundings, analyzes it in real time, and notifies the driver. The system aims to improve driver safety and driving efficiency, and has the function of judging the driver's fatigue level and encouraging them to take a break as necessary.

[1315] System Configuration

[1316] The system consists of the following main components:

[1317] 1. Device:

[1318] - The device is installed in the vehicle and collects information about the surrounding area using on-board cameras and various sensors.

[1319] - The device transmits the collected data in the appropriate format to the server in real time.

[1320] - The device displays the notification message received from the server on the in-vehicle display and notifies the driver via the voice assistant.

[1321] 2. Server:

[1322] - The server receives the information sent from the device and analyzes it using the generative AI model.

[1323] - The server generates an appropriate notification message based on the analysis results.

[1324] 3. Notification mechanism:

[1325] - The notification mechanism consists of the device display and voice assistant.

[1326] - The notification mechanism provides real-time warnings and advice to the driver.

[1327] Hardware and software used

[1328] 1. Hardware:

[1329] - In-car camera: A camera that captures the driver's facial expressions, blinking, and surrounding pedestrians and vehicles.

[1330] - Various sensors: Sensors for detecting vehicle movement and surrounding information using accelerometers, gyro sensors, LIDAR, etc.

[1331] - Terminal: A terminal installed in a vehicle that collects and transmits data.

[1332] - Display and voice assistant: Output devices for notifying the driver.

[1333] 2. Software:

[1334] - Generative AI model: An artificial intelligence model for performing data analysis on the server.

[1335] - Data transmission software: Software for transmitting data from the terminal to the server.

[1336] - Notification generation software: Software for generating notification messages based on analysis results.

[1337] Specific examples

[1338] Scenario 1: Risk prediction notification

[1339] When a user approaches a pedestrian intersection while driving, the in-car display suddenly displays the message, "A pedestrian may jump out into the intersection 100 meters ahead," and at the same time, the voice assistant notifies the user, "A pedestrian has jumped out into the intersection ahead. Please be careful." This allows the user to immediately slow down.

[1340] Prompt Sentence: Generate a message to notify the driver of dangerous situations in advance while driving. For example, "A pedestrian may jump out at the intersection 100 meters ahead."

[1341] Scenario 2: Fatigue level assessment

[1342] If the user continues driving for a long time, the in-car camera analyzes facial expressions and blinking to determine that the user is highly fatigued. In this case, the display will show "You are fatigued. Please take a break" and a voice message will be played saying "We recommend taking a break." This will encourage the user to decide to take a break.

[1343] Prompt Sentence: Analyze the fatigue level of the user who drives for a long time and generate a message to prompt appropriate rest. For example, generate "You are tired. Please take a break."

[1344] Scenario 3: Route guidance taking traffic congestion into account

[1345] When the user voice-selects "a nearby cafe" as their destination, the server calculates the optimal route based on the latest traffic information and sends it to the device. The device then provides route guidance via voice prompts and a display: "Turn right at the next intersection, then turn left at the next traffic light."

[1346] Prompt: Calculate the optimal route to reach the specified destination and generate a message containing the instructions. For example, "Turn right at the next intersection, then left at the next traffic light."

[1347] In this way, the system of the present invention efficiently collects and analyzes information about the driver and vehicle's surroundings, and provides notifications at appropriate times to support safe and comfortable driving.

[1348] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1349] Step 1: Data collection

[1350] The device collects data in real time using onboard cameras and various sensors.

[1351] Input: Information captured by the in-vehicle camera, such as the driver's facial expressions and blinking, the movements of pedestrians, motorbikes, and bicycles, and the status of traffic lights. Data on the vehicle's movements and surrounding conditions from sensors.

[1352] Data processing / calculation: Each data point is temporarily stored in a buffer and converted into an easily readable format.

[1353] Output: A collection of recorded data.

[1354] Specific operation: The device uses the onboard camera to capture the driver's facial expressions and blinks, uses accelerometers and gyro sensors to detect vehicle movement, and uses LIDAR to scan the location of surrounding objects and pedestrians.

[1355] Step 2: Send data

[1356] The terminal transmits the collected data to the server in real time.

[1357] Input: Various data stored on the device.

[1358] Data processing / computation: Converting data into an appropriate format (e.g., JSON or XML) and encrypting it.

[1359] Output: The formatted encrypted data.

[1360] Specific operation: The device converts the collected data into an appropriate format, encrypts it for security purposes, and sends it to a server over the Internet.

[1361] Step 3: Data analysis

[1362] The server receives the data sent from the device and analyzes it using a generative AI model.

[1363] Input: Encrypted data sent from the device after format conversion.

[1364] Data processing / calculation: Data is input into a generative AI model to predict when pedestrians will suddenly run out into the road, when motorcycles will suddenly approach, and determine the driver's level of fatigue.

[1365] Output: Analysis results (e.g., prediction of pedestrians running out into the road, determination of driver fatigue level, etc.).

[1366] Specific operation: The server temporarily stores the received data in a database, extracts the necessary data from the database, inputs it into the generative AI model for analysis, and predicts the movements of pedestrians and the state of the driver as a result.

[1367] Step 4: Generate notification information

[1368] The server generates a message to notify the driver based on the analysis results.

[1369] Input: Analysis results from the generative AI model.

[1370] Data processing / calculation: Based on the analysis results, an appropriate notification message is generated for the driver.

[1371] Output: The generated notification message.

[1372] Specific operation: The server generates an appropriate notification message based on the analysis results obtained by the generative AI model. For example, it creates a message such as "A pedestrian may jump out into the street at the intersection 100 meters ahead."

[1373] Step 5: Notification

[1374] The server sends the created notification data to the terminal, which then notifies the driver via the in-car display and voice assistant.

[1375] Input: The generated notification message.

[1376] Data processing / calculation: Converts the notification message into an appropriate format and sends it to the terminal.

[1377] Output: Notification messages sent to the terminal.

[1378] Specific operation: The server sends the generated notification message to the device, and the device displays the received notification data on the in-car display and notifies the driver via the voice assistant. For example, the display may display "There is a pedestrian ahead. Please be careful" and a voice message may also be issued saying "There is a pedestrian ahead. Please be careful."

[1379] In this way, through the specific operations of each step, the system can provide a safe driving environment for the driver and improve driving efficiency.

[1380] (Application example 1)

[1381] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1382] In today's modern transportation society, there is a demand for improved driver safety and driving efficiency. In particular, the risk of accidents is increasing due to fatigue from long periods of driving and inattention to surrounding traffic conditions. Therefore, technology is needed to enable drivers to continue driving safely and efficiently. Another challenge is to monitor and analyze traffic conditions and driver status in real time and provide notifications at the appropriate time.

[1383] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1384] In this invention, the server includes means for collecting information about the driver and the surrounding area of ​​the vehicle, means for using a generative AI model to analyze the information in real time, means for notifying the driver based on the analysis results, means for analyzing the driver's facial expressions and blinking to determine the level of fatigue, means for issuing a notification encouraging the driver to take a break based on the level of fatigue, means for monitoring the movements of nearby pedestrians, motorbikes, and bicycles and the status of traffic lights to determine the level of danger, and means for issuing notifications via an in-vehicle display or voice assistant. This enables safe and efficient driving by analyzing the driver's level of fatigue and traffic conditions in real time and accurately notifying the driver of necessary information.

[1385] A "driver" is a person who drives a vehicle and is the entity that performs driving duties.

[1386] "Vehicle" means a means of transportation such as an automobile or motorcycle designed and manufactured for travel on land.

[1387] "Means of collecting information" refers to devices and technologies that use on-board cameras and various sensors to detect the situation around the driver and vehicle.

[1388] A "generative AI model" is an artificial intelligence model that uses algorithms learned from large amounts of data to solve specific problems.

[1389] "Means of notification" refers to devices and technologies such as displays and voice assistants that convey analysis results to the driver.

[1390] "Means for analyzing facial expressions and blinking" refers to technology that captures the driver's face in real time using an in-car camera or other device, and analyzes facial expressions and blink frequency using specific algorithms.

[1391] The "means for determining fatigue level" refers to technology and equipment for estimating a driver's fatigue level based on analyzed facial expressions and blinking data.

[1392] "Means for notifying drivers to take a break" refers to display or audio notification technology that advises drivers to take a break when it is determined that the driver is highly fatigued.

[1393] "Means for monitoring the movement of pedestrians, motorbikes, and bicycles" refers to devices and technologies that use on-board sensors and cameras to detect the position and movement of objects in real time.

[1394] "Means for monitoring the status of traffic lights" refers to technology that uses in-vehicle cameras and sensors to detect the color and lighting status of traffic lights in real time.

[1395] "Means for determining the level of danger" refers to technology and devices that use generative AI models to analyze surrounding information and the driver's condition and estimate potential danger.

[1396] An "in-vehicle display" is a display device installed inside a vehicle that displays driving information and warnings.

[1397] A "voice assistant" is a device equipped with voice recognition and voice synthesis technology that interacts with the driver based on voice input and provides necessary notifications and information.

[1398] The present invention relates to a system that collects information about the driver and the vehicle's surroundings, analyzes it in real time, and notifies the driver. The system aims to improve driver safety and driving efficiency, and has functions such as judging the driver's fatigue level and encouraging them to take a break as necessary.

[1399] System Configuration

[1400] The system of the present invention consists of the following main components:

[1401] 1. Device:

[1402] The car uses on-board cameras and various sensors to collect information about the surrounding area. Specifically, the cameras capture the driver's facial expressions and blinks, while the sensors monitor nearby pedestrians, motorbikes, bicycles, and traffic lights.

[1403] 2. Server:

[1404] The system receives information sent from the device and analyzes it using a generative AI model. Specifically, the system analyzes the data using a trained AI model (using frameworks such as Google TensorFlow or PyTorch). The analysis results include predictions of pedestrians jumping out at specific locations, situations in which a motorcycle is suddenly approaching, and the driver's fatigue level.

[1405] 3. Notification mechanism:

[1406] This includes a display and voice assistant to notify the driver based on the analysis results. For example, necessary information can be immediately conveyed to the driver by displaying a warning on the in-car display or using a voice assistant (general name).

[1407] Program processing

[1408] Hardware:

[1409] In-vehicle camera: Captures the driver's facial expressions and the surrounding environment in real time.

[1410] Sensors: Capture surrounding activity (e.g. LIDAR, radar).

[1411] In-car display: Show notifications.

[1412] Voice assistant: Provides voice notifications.

[1413] software:

[1414] Generative AI models: Use frameworks such as Google TensorFlow and PyTorch.

[1415] Data analysis platform: Apache Kafka (data streaming), Apache Hadoop (data storage).

[1416] Specific examples

[1417] Scenario 1: Fatigue assessment while driving

[1418] The device captures the driver's facial expressions and blinks in real time, and the server analyzes them using a generative AI model. If the driver is judged to be fatigued, a notification will be sent saying, "You are fatigued. Please take a break."

[1419] Scenario 2: Risk prediction notification

[1420] The device collects information on the movements of pedestrians, motorbikes, and bicycles in the vicinity, as well as the status of traffic lights, which the server analyzes using a generative AI model. If a dangerous situation is predicted, a notification will be sent to the user, stating, "There is a possibility that a pedestrian may suddenly jump out at the intersection 100 meters ahead."

[1421] Scenario 3: Route guidance taking traffic congestion into account

[1422] When the user voice-selects a destination as "a nearby cafe," the server calculates the optimal route based on the latest traffic information, and the device then provides route guidance through voice prompts and a display, such as "Turn right at the next intersection and left at the next traffic light."

[1423] Prompt Sentence Examples

[1424] "The system analyzes the driver's real-time facial expressions and blinking data to determine their fatigue level. If the level of fatigue is high, the system will notify the driver by voice, saying, 'We recommend you take a break.'"

[1425] As described above, the present invention is a system that efficiently collects and analyzes information about the driver and the vehicle's surroundings using an in-vehicle camera and various sensors, thereby improving the driver's safety and driving efficiency.

[1426] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1427] Step 1: Data collection

[1428] The device uses on-board cameras and various sensors to capture the driver's facial expressions and eye blinks, as well as the status of surrounding pedestrians, motorcyclists, bicycles, and traffic lights in real time. Input data includes video feeds and sensor data. As an output, this data is pre-processed and converted into a format for analysis.

[1429] Step 2: Send data

[1430] The device sends pre-processed data to the server in real time. The input data includes captured video feeds and sensor data. These data are converted into an appropriate data format (e.g., JSON, Protobuf) and sent to the server. The output is the data packets arriving at the server.

[1431] Step 3: Data analysis

[1432] The server analyzes the received data using a generative AI model. Input data includes the video feed and sensor data sent from the device. The analysis results include the driver's fatigue level, the movement of nearby pedestrians, the approach of motorcycles and bicycles, and the status of traffic lights. The output is the analyzed results data.

[1433] Step 4: Generate notification information

[1434] The server generates notification and advice messages for the driver based on the analysis results. The input data includes the analysis results data. The generated notification messages include warning information such as "There is a risk of a pedestrian jumping out at the intersection 100 meters ahead" or "You are fatigued. Please take a break." The output is a specific notification message.

[1435] Step 5: Notification

[1436] The server sends the created notification data to the terminal, which displays it on the in-car display and notifies the driver via the voice assistant. The input data includes the notification message sent from the server. Specific actions include displaying a warning on the display and notifying the driver by voice, "There is a pedestrian ahead. Please be careful." The output is the notification being conveyed to the driver.

[1437] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1438] This invention combines a system that collects information about the driver and vehicle's surroundings and analyzes it in real time with an emotion engine that recognizes the driver's emotions. This system aims to improve driver safety and driving efficiency, and has the function of analyzing the driver's emotional state and providing appropriate advice and notifications accordingly.

[1439] System Configuration

[1440] The system of the present invention consists of the following main components:

[1441] 1. Terminal: Installed in the vehicle, it uses onboard cameras and various sensors to collect information about the surrounding area, as well as the driver's facial expressions and voice data.

[1442] 2. Server: Receives information sent from the device and analyzes it using the generative AI model and emotion engine.

[1443] 3. Notification mechanism: Display and voice assistant to notify the driver based on the analysis results.

[1444] 4. Emotion engine: Analyzes the driver's facial expressions and voice data to recognize emotions.

[1445] Program processing

[1446] The program processing in this system is carried out in the following manner.

[1447] Data collection and transmission

[1448] The device collects data in real time from on-board cameras and sensors. The collected data includes the movement of pedestrians, motorbikes, and bicycles, the status of traffic lights, and the driver's facial expressions and voice data. The device then transmits the collected data to a server. The data is transmitted in real time in an appropriate format.

[1449] Data analysis and emotion recognition

[1450] The server receives the data sent from the device and uses the generative AI model to analyze the surrounding situation and the driver's state. Furthermore, the emotion engine analyzes the driver's facial expressions and voice data to recognize the driver's emotional state. For example, it determines whether the driver is stressed or relaxed.

[1451] Generate and send notification information

[1452] The server generates notification data based on the analysis results and emotion recognition results. For example, it creates a warning message such as "A pedestrian may jump out into the intersection 100 meters ahead" or an advice message based on the driver's emotion, such as "Please concentrate on driving." The server then sends the created notification data to the device.

[1453] Notification implementation

[1454] The device displays the received notification data on the in-car display and notifies the driver via the voice assistant. Specifically, the display will show "There is a pedestrian ahead. Please be careful," and at the same time a voice message will say, "A pedestrian has jumped out ahead. Please be careful." Advice based on the driver's emotions will also be displayed and spoken.

[1455] Specific examples

[1456] Scenario 1: Risk prediction notification

[1457] When a user approaches a pedestrian intersection ahead while driving, the in-car display suddenly displays the message, "A pedestrian may jump out at the intersection 100 meters ahead," and at the same time, the voice assistant notifies the user, "A pedestrian has jumped out ahead. Please be careful." This allows the user to immediately slow down and prevent an accident.

[1458] Scenario 2: Fatigue and Emotion Recognition

[1459] If the user continues driving for a long time, the in-car camera analyzes facial expressions and blinking to determine a high level of fatigue, and the emotion engine determines the driver's stress level. In this case, the system displays a message saying "You are fatigued. Please take a break" and also provides a voice message saying "We recommend taking a break." If the stress level is high, the system will also provide a notification suggesting relaxation methods.

[1460] Scenario 3: Emotional Advice

[1461] If the emotion engine determines that the user is becoming irritated while driving, the server will detect this and provide visual and audio advice such as "Please relax while driving." At the same time, it is also possible to provide car music and relaxation guides.

[1462] In this way, the system of the present invention supports safe and comfortable driving by efficiently collecting and analyzing information about the driver and the vehicle's surroundings and notifying them at the appropriate time. It also aims to achieve even safer driving by recognizing the driver's emotional state and providing advice and support accordingly.

[1463] The processing flow will be explained below.

[1464] Step 1:

[1465] The device collects real-time data from onboard cameras and sensors, including the movement of pedestrians, motorcyclists, and bicycles, the status of traffic lights, and the driver's facial expressions and voice data.

[1466] Step 2:

[1467] The terminal transmits the collected data to a server in real time via wireless communication in an appropriate format.

[1468] Step 3:

[1469] The server receives the data sent from the device and inputs it into the generative AI model to analyze the surrounding situation and the driver's state.

[1470] Step 4:

[1471] The server analyzes various factors, such as the approach of pedestrians, motorbikes, and bicycles, and the driver's fatigue level. For example, it analyzes the prediction of pedestrians jumping out at a specific location, the situation of a motorbike suddenly approaching, the driver's facial expressions, and the frequency of blinking to assess the driver's fatigue level.

[1472] Step 5:

[1473] The server generates risk prediction data from the analysis results. For example, if there is a high possibility that a pedestrian will suddenly jump out into the intersection ahead, it will generate the necessary warning message.

[1474] Step 6:

[1475] The server uses facial expressions and voice data to recognize the driver's emotions through an emotion engine, for example, determining whether the driver is stressed or relaxed.

[1476] Step 7:

[1477] The server generates appropriate advice and notification messages for the driver based on the analysis and emotion recognition results, such as "A pedestrian may jump out into the road at the intersection 100 meters ahead" or "Please concentrate on driving."

[1478] Step 8:

[1479] The server sends the created notification data to the terminal. The data is designed to be sent quickly.

[1480] Step 9:

[1481] The device displays the received notification data on the in-car display and notifies the driver via the voice assistant. Specifically, the display will show "There is a pedestrian ahead. Please be careful," and at the same time a voice will say, "A pedestrian has jumped out ahead. Please be careful."

[1482] Step 10:

[1483] The device continuously monitors the driver's facial expressions and blinking, and evaluates the driver's emotional state using an emotion engine. If the device determines that the driver is fatigued, it will notify the driver, saying, "You are fatigued. Please take a break." If the driver's stress level is high, it will also notify the driver and suggest relaxation methods.

[1484] Step 11:

[1485] The server obtains real-time traffic congestion information from external traffic information services and calculates the optimal route to the driver's destination and intermediate stops based on the information obtained.

[1486] Step 12:

[1487] The server sends the calculated route data to the device, which receives it and starts providing guidance to the driver via voice and display, providing specific instructions such as "turn right at the next intersection."

[1488] In this way, the system of the present invention analyzes various information in real time and notifies drivers at the appropriate time to improve driver safety and efficiency. In addition, by combining it with an emotion engine, it provides advice according to the driver's emotional state, aiming to further safer driving and reduce driver stress.

[1489] Example 2

[1490] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1491] In modern society, ensuring safe and comfortable driving requires accurate understanding of the driver's situation and emotional state, and providing appropriate advice and notifications accordingly. Conventional systems lack the ability to accurately recognize the driver's emotions and provide notifications in real time, creating the risk that driver stress and fatigue could impede safe driving. Furthermore, technology for quickly analyzing surrounding information and providing appropriate advice and warnings is also not yet mature. This makes it difficult to ensure the safety of the driver and other road users.

[1492] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1493] In this invention, the server includes a means for collecting information about the driver and the vehicle's surroundings, a means for analyzing the information in real time using a generative AI model, a means for analyzing the driver's facial expressions and voice data to recognize the driver's emotional state, a means for providing advice or notification based on the emotional state, and a means for notifying the driver based on the analysis results. This allows the server to accurately recognize the driver's emotional state and provide appropriate advice or warnings accordingly. It also efficiently analyzes information about the surrounding environment to support safe driving.

[1494] "Means for collecting information about the driver and the vehicle's surroundings" refers to devices and functions that use on-board cameras, microphones, and various sensors to collect data on the driver's condition and the vehicle's surrounding environment in real time.

[1495] "Means using generative AI models for real-time analysis" refers to technologies that use generative AI models to instantly analyze collected data and understand the surrounding environment and the driver's situation.

[1496] "Means for notifying the driver based on the analysis results" refers to devices or functions for providing appropriate notifications and advice to the driver based on the analysis results of the generative AI model.

[1497] "Means for analyzing the driver's facial expressions and voice data to recognize the emotional state" refers to technology or devices that analyze the driver's facial expressions and voice data and recognize the driver's emotional state (e.g., stress, relaxation, etc.) from that information.

[1498] "Means for providing advice or notifications based on emotional state" refers to a function or device that provides appropriate advice or warnings to the driver in real time based on the driver's recognized emotional state.

[1499] "Means for setting the driver's destination and intermediate points by voice" refers to technology or devices that allow the driver to set the destination and intermediate points using voice commands.

[1500] "Means of obtaining real-time traffic congestion information and calculating the optimal route" refers to technology or devices that obtain current traffic conditions and traffic congestion information in real time and calculate the optimal driving route based on that information.

[1501] "Means for providing audio and visual guidance to the driver of the calculated route" refers to a function or device that provides audio and visual guidance to the driver of the calculated driving route.

[1502] "Means including pedestrian, motorbike, and bicycle information" refers to technology or devices that can detect the presence of and collect information about pedestrians, motorbikes, and bicycles moving around the vehicle.

[1503] "Means for notifying the driver of warnings in real time via an in-vehicle display and voice assistant" refers to a function or device that notifies the driver of warning messages in real time using an in-vehicle display and voice assistant.

[1504] The present invention is a system that efficiently collects and analyzes information about the driver and the vehicle's surroundings, and recognizes the driver's emotional state to support safe and comfortable driving. Specific embodiments of the present invention are described below.

[1505] System Configuration

[1506] The system of the present invention consists of the following main components:

[1507] 1. Terminal

[1508] 2. Server

[1509] 3. Means of notification

[1510] 4. Emotion Engine

[1511] Terminal

[1512] The device is installed in the vehicle and uses an onboard camera, microphone, and various sensors to collect information about the surrounding area, as well as the driver's facial expression and voice data. The collected data includes the movements of pedestrians, motorbikes, and bicycles, the status of traffic lights, and the driver's facial expression and voice data. The device transmits this data to a server in an appropriate format (e.g., JSON format).

[1513] server

[1514] The server receives the data sent from the device and uses a generative AI model to analyze the surrounding situation and the driver's state. This analysis includes object recognition technology and traffic light status analysis. It also uses an emotion engine to analyze the driver's facial expressions and voice data to recognize the driver's emotional state. For example, it determines whether the driver is stressed or relaxed.

[1515] Notification means

[1516] The notification means include an in-vehicle display and a voice assistant. The server generates notification information based on the analysis results and emotion recognition results and sends it to the terminal. The terminal displays the received notification information on the in-vehicle display and notifies the driver via the voice assistant.

[1517] Emotion Engine

[1518] The emotion engine analyzes the driver's facial expressions and voice data to recognize their emotional state, for example, determining their stress level and fatigue level from eye movements, blinking frequency, and tone of voice.

[1519] Specific examples

[1520] Scenario 1: Risk prediction notification

[1521] When a user approaches a pedestrian intersection ahead while driving, the device sends video footage captured by the onboard camera to a server. The server analyzes this data and detects the possibility of a pedestrian jumping out at the intersection 100 meters ahead. The analysis results are generated as notification information and sent to the device. The device receives this information and displays "A pedestrian may jump out at the intersection 100 meters ahead" on the in-car display, and the voice assistant also notifies the user. This allows the user to immediately slow down and prevent an accident.

[1522] Scenario 2: Fatigue and Emotion Recognition

[1523] When a user continues driving for a long period of time, the in-car camera captures the user's facial expressions and blinking in real time and sends the data to a server. The server analyzes this data and determines that the level of fatigue is high. At the same time, the emotion engine determines the driver's stress level. Based on this result, the server generates a message saying, "You are fatigued. Please take a break," and sends it to the device. The device then displays a notification on the in-car display and also issues a voice message saying, "We recommend that you take a break."

[1524] Scenario 3: Emotional Advice

[1525] If the emotion engine determines that the user is becoming irritated while driving, the server detects this and generates advice such as "Please relax while driving." At the same time, it also creates relaxing car music and a relaxation guide and sends these to the device. The device then notifies the user of this information via the display and voice assistant.

[1526] Prompt Sentence Examples

[1527] Please explain the specific processing flow of a safety drive system that analyzes the driver's emotions and provides appropriate advice.

[1528] In this way, the system of the present invention supports safe and comfortable driving by efficiently collecting and analyzing information about the driver and vehicle's surroundings and notifying them at the appropriate time. Furthermore, it aims to achieve even safer driving by recognizing the driver's emotional state and providing advice and support accordingly.

[1529] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1530] Step 1: Data collection

[1531] The device uses onboard cameras, microphones, and various sensors to collect real-time information about the driver and the vehicle's surroundings.

[1532] Input: Driver's facial expression, voice data, video footage of the vehicle's surroundings, sensor data (vehicle speed, brake status, etc.)

[1533] Data processing: Capture camera footage frame by frame, record audio as sample data, and log sensor data by time.

[1534] Output: Collected real-time data (camera footage, audio data, sensor data)

[1535] Step 2: Send data

[1536] The device sends the collected data to the server in an appropriate format (e.g., JSON format).

[1537] Input: Collected real-time data (camera footage, audio data, sensor data)

[1538] Data processing: Convert the data into JSON format, split it into packets, and prepare it for transmission.

[1539] Output: JSON formatted data packet

[1540] Step 3: Receiving data

[1541] The server receives the data packets sent from the terminal.

[1542] Input: JSON formatted data packet

[1543] Data processing: Reconstructing received packets and restoring them to their original data format.

[1544] Output: Recovered real-time data (camera video, audio data, sensor data)

[1545] Step 4: Data analysis

[1546] The server uses a generative AI model to analyze the data it receives, specifically using object recognition technology to detect the presence of pedestrians, motorbikes, and bicycles, as well as analyze the status of traffic lights.

[1547] Input: Recovered real-time data (camera video, audio data, sensor data)

[1548] Data Computation: Uses generative AI models to perform object recognition and analyze traffic light status.

[1549] Output: Analysis results (presence of pedestrians, traffic light status, etc.)

[1550] Step 5: Emotion Recognition

[1551] The server uses an emotion engine to analyze the driver's facial expressions and voice data to recognize their emotional state.

[1552] Input: Driver's facial expression data and voice data

[1553] Data Computation: Using the emotion engine, facial and vocal features are extracted to classify emotional states.

[1554] Output: Emotion recognition results (e.g., stress, relaxation, fatigue, etc.)

[1555] Step 6: Generate notification information

[1556] The server generates notification information based on the analysis and emotion recognition results, such as a warning message such as "A pedestrian may jump out into the road at the intersection 100 meters ahead" or an advice message such as "Please concentrate on driving."

[1557] Input: Data analysis results, emotion recognition results

[1558] Data processing: Generate appropriate messages based on analysis results and emotion recognition results

[1559] Output: Notification information (warning messages, advice messages, etc.)

[1560] Step 7: Send notification data

[1561] The server transmits the generated notification information to the terminal.

[1562] Input: Notification information

[1563] Data processing: The notification message is converted into a data packet and prepared for transmission.

[1564] Output: Notification data packet

[1565] Step 8: Notification

[1566] The device displays the received notification information on the in-car display and notifies the driver via the voice assistant.

[1567] Input: Notification data packet

[1568] Data processing: Analyzes the notification data packet and converts it into a format for display and audio output.

[1569] Output: Warning message on the display, notification via voice assistant

[1570] Specifically, when a driver approaches a pedestrian intersection ahead, the camera image is sent to the server, which then generates an analysis result as notification information and sends it to the device. The device receives this information and issues a warning on the display and via the voice assistant. This series of steps allows the user to take appropriate action.

[1571] (Application example 2)

[1572] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1573] In recent years, as autonomous vehicles have become more widely used, there has been a demand for understanding the driver's fatigue level and emotional state to provide a safe and comfortable driving experience. However, conventional technologies lack the means to accurately analyze the driver's emotional state in real time and provide appropriate notifications and advice accordingly. Furthermore, systems that can instantly notify the driver using mobile devices such as smartphones and smart glasses have not been fully established. As a result, there have been concerns about reduced driver safety and driving efficiency.

[1574] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1575] In this invention, the server includes: means for collecting information about the driver and the surroundings of the vehicle; means for using a generative AI model to analyze the information in real time; means for notifying the driver based on the analysis results; means for analyzing the driver's facial expressions and blinking to determine the driver's fatigue level; means for notifying the driver to take a break based on the fatigue level; means for recognizing the driver's emotional state and providing appropriate advice or notification based on the emotional state; and means for providing notifications to the driver using a mobile device such as a smartphone or smart glasses. This makes it possible to analyze the driver's emotional state in real time and notify the driver at the appropriate time, thereby supporting safe and smooth driving.

[1576] "Driver" means a person who operates a vehicle.

[1577] "Vehicle" means a moving object designed to travel on roads.

[1578] "Surrounding information" refers to all data about the vehicle's external environment, including pedestrians, other vehicles, bicycles, and traffic light status.

[1579] A "generative AI model" is an algorithm that uses artificial intelligence to analyze input data and generate output results.

[1580] "Notifications" are messages or alerts that convey specific information to the driver.

[1581] "Facial expressions" refer to facial movements and states that indicate a person's emotions and reactions.

[1582] Blinking is the act of closing and opening the eyes.

[1583] "Fatigue level" is an index that indicates the degree of fatigue experienced by a driver.

[1584] "Emotional state" refers to the psychological state or mood exhibited by the driver.

[1585] "Advice" means guidance or suggestions provided to a driver.

[1586] "Means for collecting information about the surrounding area" refers to devices and methods for acquiring data about the vehicle's external environment using on-board cameras, sensors, etc.

[1587] "Real-time analytical means" refers to devices or methods for processing collected data in real time and generating results.

[1588] A "smartphone" is a mobile phone equipped with internet connectivity and a camera.

[1589] "Smart glasses" are glasses-type devices that can display information in real time.

[1590] "Mobile devices" is a general term for portable electronic devices, including smartphones, tablets, and smart glasses.

[1591] "Relaxation music" is music that helps drivers relax.

[1592] "Real-time scene analysis" is a technology that instantly analyzes video data around the vehicle to grasp the current situation.

[1593] MODE FOR CARRYING OUT THE INVENTION

[1594] System Overview

[1595] This invention is a system that analyzes the driver's fatigue level and emotional state in real time and provides appropriate notifications and advice. The system components include:

[1596] 1. Device: A mobile device such as a smartphone or smart glasses that uses cameras and various sensors to collect data from the driver and the vehicle's surrounding environment.

[1597] 2. Server: Used to analyze collected data in real time, utilizing generative AI models and emotion engines for processing.

[1598] 3. Notification mechanism: Includes a display and voice assistant to notify the driver in a timely manner based on the analysis results.

[1599] Program processing

[1600] Data collection and transmission

[1601] The device uses a camera and various sensors to collect the driver's facial expressions, blinking, and information about the vehicle's surroundings (pedestrians, bicycles, other vehicles, etc.) in real time. The collected data is sent to a server in an appropriate format.

[1602] Data analysis and emotion recognition

[1603] The server uses a generative AI model to analyze the received data. It also uses an emotion engine to analyze facial expressions and voice data. This analysis identifies the driver's fatigue level and emotional state (e.g., stressed, relaxed, fatigued).

[1604] Generate and send notification information

[1605] The server generates notification data based on the analysis results. For example, if danger is imminent, a warning message such as "There is a pedestrian ahead. Please be careful" is generated. Advice based on the driver's emotional state (for example, "Please relax. We will play car music") is also generated. The generated notification data is sent to the device.

[1606] Notification implementation

[1607] The device then displays the received notification data on the display of a smartphone or smart glasses and notifies the driver via a voice assistant. The notification content includes information about nearby dangers and advice tailored to the driver's emotional state.

[1608] Hardware and software used

[1609] Camera: Cameras built into smartphones or smart glasses are used to collect data on the driver's facial expressions and the surrounding environment.

[1610] Generative AI models: Analyze the collected data using models built with machine learning libraries such as TensorFlow and Keras.

[1611] Emotion Engine: Uses the dlib library to detect facial landmarks and recognize emotions.

[1612] Audio notification: Uses the playsound library to provide audio alerts.

[1613] Specific examples

[1614] Scenario 1: Risk prediction notification

[1615] When a driver approaches an intersection ahead, the camera in the smart glasses detects pedestrians and the server notifies the driver, saying, "There is a pedestrian ahead. Please be careful." The warning is conveyed to the driver via the display and voice assistant, preventing accidents from occurring.

[1616] Scenario 2: Advice based on emotional state

[1617] If the driver shows an emotional state of "anger," the smartphone will notify the driver, "Relax. Car music will be played," and will automatically play car music.

[1618] Prompt Sentence Examples

[1619] Text format

[1620] I want to capture facial expressions from camera data on smart glasses and perform emotion recognition. If the result is "anger," I want to display "Please be careful" and play relaxation music.

[1621] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1622] Step 1:

[1623] The device uses a camera built into a smartphone or smart glasses to collect the driver's facial expressions, blinking, and information about the vehicle's surroundings (pedestrians, bicycles, other vehicles, etc.) in real time. This data includes images of the driver's facial expressions, video information about the surrounding environment, and audio data. The collected data is converted into an appropriate format as digital data and sent to a server in real time.

[1624] Step 2:

[1625] The server receives the data sent from the device. This data includes images of the driver's facial expressions and video information of the surrounding environment. To analyze the received data, the server preprocesses the data using a generative AI model. For example, it extracts the driver's facial area using a facial recognition algorithm and performs preprocessing to input the data into the emotion engine.

[1626] Step 3:

[1627] The server performs emotion recognition on the facial expression images using an emotion engine. The emotion engine uses the dlib library to detect facial landmarks (eyes, mouth, nose, etc.) and estimate the emotional state based on them. This process includes facial feature point detection, feature extraction, and emotion classification data computation. The output is the driver's emotional state (e.g., anger, sadness, joy, relaxation).

[1628] Step 4:

[1629] The server uses a generative AI model to analyze surrounding environment data and generate notification information for the driver. For example, it analyzes the location data of pedestrians, bicycles, and other vehicles, and generates a warning message such as "There is a pedestrian ahead. Please be careful" if a danger is imminent. It then performs appropriate data processing and calculations and creates a notification message as output.

[1630] Step 5:

[1631] The server generates notification data at the optimal timing for the driver based on the emotion recognition results and the analysis of the surrounding environment. For example, if the driver's emotional state is "anger," it generates an advice message such as "Please relax. We will play car music." This notification data is converted into an appropriate format and sent to the device.

[1632] Step 6:

[1633] The device displays the received notification data on the display of a smartphone or smart glasses and notifies the driver via a voice assistant. For example, a warning message such as "There is a pedestrian ahead. Please be careful" is displayed on the display and notified by voice. Relaxation music corresponding to the driver's emotional state is also automatically played on the smartphone.

[1634] Step 7:

[1635] The user takes appropriate driving actions based on notifications and advice from the device. For example, they may check for pedestrians ahead and slow down for safety. They may also listen to relaxation music to ensure a relaxed driving experience. In this way, the system supports the user's safe and comfortable driving.

[1636] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1637] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1638] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1639] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1640] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1641] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1642] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1643] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1644] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1645] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1646] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1647] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1648] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1649] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1650] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1651] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1652] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1653] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1654] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1655] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1656] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1657] The following is further disclosed regarding the above embodiment.

[1658] (Claim 1)

[1659] a means for collecting information about the driver and the vehicle's surroundings;

[1660] A means using a generative AI model to analyze the information in real time;

[1661] a means for notifying the driver based on the analysis result;

[1662] A method for judging the driver's level of fatigue by analyzing their facial expressions and blinking,

[1663] a means for issuing a notice to encourage a break according to the fatigue level;

[1664] A system including:

[1665] (Claim 2)

[1666] 2. The system according to claim 1, further comprising: means for setting the driver's destination and intermediate destinations by voice;

[1667] A means to obtain real-time traffic congestion information and calculate the optimal route,

[1668] The system further includes means for providing audio and visual guidance to the driver along the calculated route.

[1669] (Claim 3)

[1670] The system according to claim 1, wherein the vehicle surrounding information includes information on pedestrians, motorbikes, and bicycles;

[1671] The system further includes means for notifying the driver of the warning in real time via an in-vehicle display and voice assistant.

[1672] "Example 1"

[1673] (Claim 1)

[1674] a means for collecting information about the driver and the vehicle's surroundings;

[1675] A means using a generative AI model to analyze the information in real time;

[1676] a means for notifying the driver based on the analysis result;

[1677] A method for judging the driver's level of fatigue by analyzing their facial expressions and blinking,

[1678] a means for issuing a notice to encourage a break according to the fatigue level;

[1679] means for transmitting the collected data to a server in an appropriate format;

[1680] A means for generating a notification message based on the results of analysis by the generative AI model;

[1681] A means of conveying notification messages to the driver via the in-car display or voice assistant,

[1682] A system including:

[1683] (Claim 2)

[1684] A means for setting a destination and intermediate points by voice;

[1685] A means to obtain real-time traffic congestion information and calculate the optimal route,

[1686] and means for providing audio and visual guidance of the calculated route to the driver.

[1687] 10. The system of claim 1.

[1688] (Claim 3)

[1689] The surrounding information includes pedestrian, motorbike, and bicycle information, and

[1690] Further includes means for notifying the driver of warnings in real time via in-vehicle displays and voice assistants.

[1691] 10. The system of claim 1.

[1692] "Application Example 1"

[1693] (Claim 1)

[1694] a means for collecting information about the driver and the vehicle's surroundings;

[1695] A means using a generative AI model to analyze the information in real time;

[1696] a means for notifying the driver based on the analysis result;

[1697] A method for judging the driver's level of fatigue by analyzing their facial expressions and blinking,

[1698] a means for issuing a notice to encourage a break according to the fatigue level;

[1699] A means of monitoring the movement of pedestrians, motorbikes, and bicycles in the vicinity, as well as the status of traffic lights, and determining the level of danger;

[1700] Notifications can be sent via in-car displays or voice assistants,

[1701] A system including:

[1702] (Claim 2)

[1703] a means for setting the driver's destination and intermediate stops by voice;

[1704] A means to obtain real-time traffic congestion information and calculate the optimal route,

[1705] means for providing audio and visual guidance to the driver along the calculated route;

[1706] 10. The system of claim 1, wherein the system monitors surrounding activity and notifies the driver.

[1707] (Claim 3)

[1708] The vehicle's surrounding information includes information on pedestrians, motorbikes, and bicycles;

[1709] 10. The system of claim 1, further comprising means for notifying the driver of the warning in real time via an in-vehicle display and a voice assistant.

[1710] "Example 2: Combining Emotion Engines"

[1711] (Claim 1)

[1712] a means for collecting information about the driver and the vehicle's surroundings;

[1713] A means using a generative AI model to analyze the information in real time;

[1714] a means for notifying the driver based on the analysis result;

[1715] A means for analyzing the driver's facial expression and voice data to recognize the driver's emotional state;

[1716] means for providing advice or notification in response to said emotional state;

[1717] A system including:

[1718] (Claim 2)

[1719] a means for setting the driver's destination and intermediate stops by voice;

[1720] A means to obtain real-time traffic congestion information and calculate the optimal route,

[1721] and means for providing audio and visual guidance of the calculated route to the driver.

[1722] 10. The system of claim 1.

[1723] (Claim 3)

[1724] The vehicle's surrounding information includes information on pedestrians, motorbikes, and bicycles;

[1725] Further includes means for notifying the driver of warnings in real time via in-vehicle displays and voice assistants.

[1726] 10. The system of claim 1.

[1727] "Application example 2 when combining emotion engines"

[1728] (Claim 1)

[1729] a means for collecting information about the driver and the vehicle's surroundings;

[1730] A means using a generative AI model to analyze the information in real time;

[1731] a means for notifying the driver based on the analysis result;

[1732] A method for judging the driver's level of fatigue by analyzing their facial expressions and blinking,

[1733] a means for issuing a notice to encourage a break according to the fatigue level;

[1734] a means for recognizing the emotional state of the driver and providing appropriate advice or notification based on said emotional state;

[1735] a means of providing notifications to drivers using a mobile device such as a smartphone or smart glasses;

[1736] A system including:

[1737] (Claim 2)

[1738] 10. The system of claim 1, further comprising means for performing real-time scene analysis.

[1739] (Claim 3)

[1740] 10. The system of claim 1, further comprising means for automatically playing relaxation music when the driver's emotional state meets a specific condition. [Explanation of symbols]

[1741] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for collecting information about the driver and the vehicle's surroundings; A means using a generative AI model to analyze the information in real time; a means for notifying the driver based on the analysis result; A method for judging the driver's level of fatigue by analyzing their facial expressions and blinking, a means for issuing a notice to encourage a break according to the fatigue level; A system including:

2. 2. The system according to claim 1, further comprising: means for setting the driver's destination and intermediate points by voice; A means to obtain real-time traffic congestion information and calculate the optimal route, The system further includes means for providing audio and visual guidance to the driver along the calculated route.

3. 2. The system according to claim 1, wherein the vehicle surrounding information includes information on pedestrians, motorbikes, and bicycles; The system further includes means for notifying the driver of the warning in real time via an in-vehicle display and voice assistant.

Images