System

Abstract

To provide a system for supporting maintenance of attention of a driver and safety confirmation of the surroundings.SOLUTION: A system comprising: means for collecting facial information and voice information of a driver; means for analyzing the facial information and the voice information of the driver to determine a condition; and means for providing a topic of conversation according to the condition. The system includes a means for collecting video data from an on-vehicle camera, a means for analyzing the video data to analyze a surrounding traffic situation, and a means for calling driver's attention based on the traffic situation.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Driving alone for long periods of time can easily cause drivers to lose focus and become drowsy. It also increases the risk of encountering recklessly driven vehicles without being aware of the surrounding traffic conditions. These problems increase the probability of traffic accidents and often threaten driver safety. There is a need to develop a system that can improve this situation and help drivers maintain their focus and ensure safety around them. [Means for solving the problem]

[0005] In order to solve the above problems, the present invention provides the following means.

[0006] The system includes a means for collecting face information and voice information of the driver, a means for analyzing the face information and voice information of the driver to determine the condition, and a means for providing a topic of conversation according to the condition.

[0007] The system also includes a means for collecting video data from an on-board camera, a means for analyzing the video data to analyze surrounding traffic conditions, and a means for issuing a warning to the driver based on the traffic conditions.

[0008] The system further includes a means for uploading the collected video data to the cloud, a means for analyzing the video data on the cloud and detecting vehicles engaged in reckless driving, and a means for issuing a warning to the driver based on the results of the cloud analysis.

[0009] In this way, it is possible to support the driver in maintaining their attention and understanding the surrounding traffic conditions, thereby providing a safe driving environment.

[0010] "Driver's facial information" is data including facial expressions and features of a person captured by an in-vehicle camera.

[0011] "Voice information" is data including the tone and content of a person's voice picked up by an in-vehicle microphone.

[0012] The "means for determining the condition" is a means for analyzing the collected facial and audio information to determine the driver's level of fatigue and attention.

[0013] The "means for providing conversation topics" is a means for generating and presenting appropriate conversation content based on the determined driver's condition.

[0014] An "in-vehicle camera" is a device that is installed inside a vehicle and that captures video data of the surrounding area.

[0015] "Video data" refers to video and image data captured by an on-board camera.

[0016] The "means for analyzing surrounding traffic conditions" refers to a means for analyzing acquired video data to recognize traffic conditions such as emergency vehicles or rapidly approaching vehicles from behind.

[0017] "Means for issuing warnings" are means for alerting the driver by voice or display based on the recognized traffic conditions.

[0018] "Means for uploading to the cloud" refers to means for transmitting acquired video data to a cloud server via a network.

[0019] "Means of analyzing on the cloud" refers to a means of processing and analyzing uploaded video data on a cloud server to detect specific patterns.

[0020] The "means for detecting dangerous driving vehicles" refers to a means for identifying vehicles engaging in dangerous driving behavior from data analyzed on the cloud.

[0021] "Means for notifying warnings" refers to means for notifying drivers of dangerous information by voice or display based on the results of cloud analysis. [Brief explanation of the drawings]

[0022] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0023] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0024] First, the terms used in the following description will be explained.

[0025] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0026] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0027] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0028] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0029] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0030] [First embodiment]

[0031] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0032] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0033] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0034] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0035] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0036] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0037] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0038] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0039] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0040] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0041] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0042] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0043] This invention is a system that uses facial and audio information from the driver to maintain attention and support safe driving. The system works by collecting and analyzing data via an on-board camera and microphone.

[0044] System Configuration

[0045] 1. Collection of facial and voice information (terminal)

[0046] The device uses the onboard camera and microphone to collect facial and audio information from the driver. Facial information includes facial expressions and gaze data, while audio information includes what is being said and tone of voice.

[0047] 2. Condition Analysis (Server)

[0048] The server analyzes the collected facial and voice information to determine the driver's condition. Specifically, it measures the driver's level of fatigue using facial expression analysis and determines the driver's level of attention through voice tone analysis. As a result, the driver's level of fatigue and attention are output as numerical values.

[0049] 3. Providing conversation topics (server)

[0050] The server generates appropriate topics based on the determined condition. For example, if the driver is highly fatigued, it might suggest taking a break, or if the driver is alert, it might suggest talking about an interesting topic.

[0051] 4. Notifications and conversation initiation (device)

[0052] The device will then provide a voice notification to the driver about the topic received from the server, and the conversation will progress naturally based on the driver's responses.

[0053] Analysis of surrounding traffic conditions and alerts

[0054] 1. Video data collection (terminal)

[0055] The device collects video data from the in-vehicle camera, including information on surrounding vehicles and road conditions.

[0056] 2. Traffic situation analysis (server)

[0057] The server analyzes the video data to detect emergency vehicles and vehicles approaching from behind, using image recognition technology to identify specific vehicles and patterns.

[0058] 3. Warning (terminal)

[0059] Based on the analysis results, the device will give the driver appropriate warnings. For example, if an emergency vehicle is approaching, the device will notify the driver, "An emergency vehicle is approaching. Please give way."

[0060] Reckless driving vehicle detection and notification

[0061] 1. Upload data to the cloud (device)

[0062] The video data collected by the device is uploaded to the cloud, where advanced analytical functions enable efficient data processing.

[0063] 2. Cloud analysis (server)

[0064] A cloud server analyzes the uploaded data and detects dangerous driving vehicles using algorithms for pattern recognition and anomaly detection of driving behavior.

[0065] 3. Warning notification (terminal)

[0066] If cloud analysis detects a vehicle driving recklessly, the device will warn the driver, for example, by sending a message saying, "There is a vehicle driving recklessly nearby. Please be careful."

[0067] Specific examples

[0068] Example 1: When a driver becomes tired while driving

[0069] User: The user (driver) is driving for a long time.

[0070] Device: The onboard camera collects the driver's facial expression data, and the microphone collects audio data.

[0071] Server: The server analyzes the collected data and determines the driver's fatigue level as "high."

[0072] Server: The server generates the topic "Would you like to take a break?"

[0073] Device: The device will prompt the driver with a voice message asking, "Would you like to take a break?"

[0074] User: The driver responds, "Yeah, let's take a break," and takes a break.

[0075] Example 2: When an emergency vehicle is approaching

[0076] Terminal: The onboard camera collects video data of the surrounding area.

[0077] Server: The server analyzes the video data and detects emergency vehicles.

[0078] Server: The server generates a warning message saying "Emergency vehicle approaching. Please give way."

[0079] Device: The device will notify the driver via voice message, "An emergency vehicle is approaching. Please give way."

[0080] User: Drivers should check behind them and give way safely.

[0081] Example 3: Reckless driving vehicles nearby

[0082] Terminal: The onboard camera collects video data of the surrounding area.

[0083] Device: The device uploads the video data to the cloud.

[0084] Server (Cloud): The server analyzes the uploaded data and detects dangerous driving vehicles.

[0085] Server (Cloud): The server generates a warning message saying, "There is a dangerously driven vehicle nearby. Please be careful."

[0086] Device: The device will notify the driver by voice, "There is a vehicle driving recklessly nearby. Please be careful."

[0087] User: Drivers should be aware of their surroundings and drive safely.

[0088] This ensures that drivers always receive appropriate support tailored to the driving situation, enhancing the safe driving environment.

[0089] The processing flow will be explained below.

[0090] Chat function

[0091] Program processing

[0092] Step 1:

[0093] The device uses the vehicle's camera and microphone to collect facial and audio information from the driver, including their facial expressions, gaze, and tone of voice.

[0094] Step 2:

[0095] The terminal transmits the collected face information and voice information to the server.

[0096] Step 3:

[0097] The server analyzes the received facial and voice information to determine the driver's condition. Specifically, it measures the driver's fatigue level using facial expression analysis and determines the driver's level of attention through voice tone analysis.

[0098] Step 4:

[0099] Based on the results of the assessment, the server generates conversation topics that correspond to the driver's condition. For example, if the driver is highly fatigued, it will suggest taking a break, and if the driver is alert, it will suggest talking about a topic of interest.

[0100] Step 5:

[0101] The server sends the generated conversation topic to the terminal.

[0102] Step 6:

[0103] The device will notify the driver of the topic of conversation received by voice and begin the conversation.

[0104] Alert function

[0105] Program processing

[0106] Step 1:

[0107] The device uses an onboard camera to collect video data about the surrounding traffic, including other vehicles and road conditions.

[0108] Step 2:

[0109] The terminal transmits the collected video data to the server.

[0110] Step 3:

[0111] The server analyzes the video data and applies image recognition algorithms to detect emergency vehicles and rapidly approaching vehicles from behind.

[0112] Step 4:

[0113] If the server detects an abnormality based on the analysis results, it generates a warning message. For example, in the case of an emergency vehicle, it generates a message saying, "An emergency vehicle is approaching. Please give way."

[0114] Step 5:

[0115] The server generates a warning message and sends it to the terminal.

[0116] Step 6:

[0117] The device will notify the driver of any warning messages it receives via voice.

[0118] Sharing and notifying dangerous driving vehicle information

[0119] Program processing

[0120] Step 1:

[0121] The device uses an onboard camera to collect video data of the surrounding area.

[0122] Step 2:

[0123] The video data collected by the device is uploaded to a cloud server.

[0124] Step 3:

[0125] The cloud server analyzes the uploaded video data to detect dangerous driving vehicles, using algorithms that identify specific driving behaviors.

[0126] Step 4:

[0127] Based on the detection result, the cloud server generates a warning message if there is a reckless driving vehicle, for example, a message saying "There is a reckless driving vehicle nearby. Please be careful."

[0128] Step 5:

[0129] The cloud server generates a warning message and sends it to the terminal.

[0130] Step 6:

[0131] The device will notify the driver of any warning messages it receives via voice.

[0132] Specific examples

[0133] Specific examples of chat functions

[0134] Step 1:

[0135] The device uses the onboard camera and microphone to collect the driver's facial and voice information.

[0136] Step 2:

[0137] The device sends the collected information to the server.

[0138] Step 3:

[0139] The server analyzes the information and determines that the driver is tired.

[0140] Step 4:

[0141] The server generates a conversation topic suggesting a break, such as "Would you like to take a short break?"

[0142] Step 5:

[0143] The server sends the generated topics to the terminal.

[0144] Step 6:

[0145] The device will notify the driver of the topic it receives via voice and suggest, "Would you like to take a short break?"

[0146] Examples of alert functions

[0147] Step 1:

[0148] The device uses an onboard camera to collect information about surrounding traffic conditions.

[0149] Step 2:

[0150] The terminal transmits the collected video data to the server.

[0151] Step 3:

[0152] The server analyzes the video data and detects emergency vehicles.

[0153] Step 4:

[0154] The server generates a warning message saying "Emergency vehicle approaching. Please give way."

[0155] Step 5:

[0156] The server generates a message and sends it to the terminal.

[0157] Step 6:

[0158] The device will then notify the driver via voice of the message received, urging them to give way to the emergency vehicle.

[0159] Specific examples of dangerous driving vehicle information

[0160] Step 1:

[0161] The device uses an onboard camera to collect video data of the surrounding area.

[0162] Step 2:

[0163] The video data collected by the device is uploaded to a cloud server.

[0164] Step 3:

[0165] The cloud server analyzes the video data and detects vehicles driving recklessly.

[0166] Step 4:

[0167] The cloud server generates a warning message saying, "There is a dangerously driven vehicle nearby. Please be careful."

[0168] Step 5:

[0169] The cloud server sends the generated message to the terminal.

[0170] Step 6:

[0171] The device will notify the driver of the received message via voice, urging them to be aware of their surroundings.

[0172] The above is a description of the specific processing steps and operations in the system of the present invention.

[0173] Example 1

[0174] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0175] In today's modern traffic society, there are many accidents that are believed to be caused by driver inattention or fatigue. Furthermore, there is a need for appropriate responses to surrounding traffic conditions and the presence of dangerously driven vehicles. However, conventional systems have had the problem of being unable to accurately grasp the driver's condition and traffic situation in real time and provide appropriate support and warnings.

[0176] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0177] In this invention, the server includes means for collecting face information and voice information of the driver, means for analyzing the face information and voice information of the driver to determine the driver's condition, means for providing a topic of conversation according to the condition, means for notifying the driver of the topic of conversation, means for analyzing the driver's responses and advancing the conversation, means for collecting video data from an in-vehicle camera, means for analyzing the video data to analyze surrounding traffic conditions, means for issuing a warning to the driver based on the traffic conditions, means for uploading the collected video data to a cloud, means for analyzing the video data on the cloud and detecting reckless driving vehicles, and means for issuing a warning to the driver based on the cloud analysis results. This makes it possible to detect a driver's dwindling attention or fatigue in real time, and maintain attention by providing appropriate topics, while also issuing appropriate warnings regarding surrounding traffic conditions and detecting and notifying reckless driving vehicles.

[0178] "Driver's facial information" is data including the driver's facial features such as facial expression and gaze.

[0179] "Voice information" is data that includes voice characteristics such as the driver's tone of voice and what is being said.

[0180] The "means for collecting face information and voice information" refers to a device that acquires the driver's face information and voice information using an in-vehicle camera, microphone, etc.

[0181] The "means for determining the condition" is a device or algorithm that analyzes the collected facial and audio information to determine the driver's level of fatigue and attention.

[0182] A "means for providing conversation topics" is a device or algorithm that generates appropriate conversation topics based on the driver's condition.

[0183] "Means for notifying the driver of the topic of conversation" refers to a device that audibly or visually conveys the generated topic of conversation to the driver through a receiver.

[0184] The "means for analyzing driver responses and advancing the conversation" refers to a device or algorithm that collects responses from the driver, analyzes their content, generates new conversation topics, and advances the conversation.

[0185] The "means for collecting video data from an in-vehicle camera" is a device that acquires video data of the surroundings using a camera mounted on a vehicle.

[0186] The "means for analyzing the surrounding traffic conditions by analyzing the video data" refers to a device or algorithm that analyzes the acquired video data and identifies the traffic conditions of surrounding vehicles, emergency vehicles, etc.

[0187] The "means for issuing a warning to the driver based on the traffic conditions" is a device that issues an appropriate warning to the driver based on the analysis results.

[0188] The "means for uploading video data to the cloud" is a device that transfers collected video data to a cloud environment via the Internet.

[0189] The "means for analyzing video data on the cloud and detecting reckless driving vehicles" refers to a device or algorithm that analyzes video data in a cloud environment and detects signs of reckless driving.

[0190] The "means for issuing a warning to the driver based on the results of cloud analysis" is a device that issues a warning to the driver for safe driving based on the results of cloud analysis.

[0191] This invention is a system that uses facial and audio information of a driver to maintain attention and support safe driving. This system operates by collecting and analyzing data via an on-board camera and microphone. Specific embodiments of this system are described below.

[0192] System Configuration

[0193] Collecting facial and audio information

[0194] The device uses an onboard camera (e.g., a network camera) and microphone (e.g., a high-sensitivity microphone) to collect the driver's facial and audio information. Facial information includes the driver's facial expressions and gaze data. Audio information includes what the driver says and the tone of their voice. For example, the camera captures the driver's eye movements and facial expressions, and the microphone records the pitch and speed of their voice.

[0195] Condition Analysis

[0196] The server analyzes the collected facial and audio information to determine the driver's condition. Specifically, it measures the driver's level of fatigue using facial expression analysis (e.g., image processing algorithms) and determines the driver's level of attention through voice tone analysis (e.g., voice recognition software). As a result, the driver's level of fatigue and attention state are output as numerical values. For example, frequent eye closure and a restless voice are detected as signs of fatigue.

[0197] Providing topics for conversation

[0198] The server generates appropriate topics based on the determined condition. For example, if the driver is highly fatigued, it will "suggest a break," and if the driver is alert, it will "talk about a topic of interest." This process uses a generative AI model, which generates, for example, "driver's interests" based on the prompt sentence.

[0199] Notifications and conversation starters

[0200] The device then notifies the driver of the topic received from the server by voice. Specifically, it plays back a message generated using speech synthesis software (e.g., text-to-speech software). The conversation progresses naturally based on the driver's response. For example, the device may say, "Do you want to take a short break?" and the driver may respond, "Yes, let's take a short break."

[0201] Analysis of surrounding traffic conditions and alerts

[0202] The device collects video data from the onboard camera. This camera captures surrounding vehicles and road conditions in real time. The video data is sent to a server, which analyzes it to detect approaching emergency vehicles and rapidly approaching vehicles from behind. Image recognition technology (e.g., deep learning models) is used for the analysis. Based on the analysis results, the device issues appropriate warnings to the driver. For example, it may notify the driver, "An emergency vehicle is approaching. Please give way."

[0203] Reckless driving vehicle detection and notification

[0204] The video data collected by the device is uploaded to the cloud. The cloud environment is equipped with advanced analytical functions; for example, a server on the cloud analyzes the uploaded data to detect vehicles that are driving recklessly. The analysis uses driving behavior pattern recognition and anomaly detection algorithms. If a vehicle is detected as a result of cloud analysis, the device will notify the driver, "There is a vehicle that is driving recklessly nearby. Please be careful."

[0205] Specific examples

[0206] Example 1: When a driver becomes tired while driving

[0207] User: The driver is driving for a long time.

[0208] Device: The onboard camera collects the driver's facial expression data, and the microphone collects audio data.

[0209] Server: The server analyzes the collected data and determines the driver's fatigue level as "high."

[0210] Server: The server generates the message "Would you like to take a break?"

[0211] Device: Your device will say, "Would you like to take a break?"

[0212] User: The driver responds, "Yeah, let's take a break," and takes a break.

[0213] Example 2: When an emergency vehicle is approaching

[0214] Terminal: The onboard camera collects video data of the surrounding area.

[0215] Server: The server analyzes the video data and detects emergency vehicles.

[0216] Server: The server generates the message "Emergency vehicle approaching. Please give way."

[0217] Device: The device will announce "An emergency vehicle is approaching. Please give way."

[0218] User: Drivers should check behind them and give way safely.

[0219] Example 3: Reckless driving vehicles nearby

[0220] Device: The onboard camera collects images of the surrounding area.

[0221] Device: The device uploads the video data to the cloud.

[0222] Server (Cloud): The server analyzes the uploaded data and detects dangerous driving vehicles.

[0223] Server (Cloud): The server generates a message saying, "There is a dangerously driven vehicle nearby. Please be careful."

[0224] Device: The device will announce with a voice message, "There is a dangerously driven vehicle nearby. Please be careful."

[0225] User: The driver remains alert to their surroundings and continues driving safely.

[0226] The system is designed to provide drivers with appropriate support in real time while driving, thereby contributing to an enhanced safe driving environment.

[0227] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0228] Step 1:

[0229] (Data collection begins)

[0230] The device activates the on-board camera and microphone and starts collecting the driver's facial and audio information. The input is real-time data from the on-board camera and microphone, and the output is stored in the device as facial and audio information.

[0231] Step 2:

[0232] (Data import)

[0233] The device uses a camera to capture the driver's facial image and facial expressions, and a microphone to pick up audio. Specifically, the camera recognizes each part of the face (eyes, mouth, etc.), and the microphone records the pitch and speed of the audio. The input is video and audio data captured in real time, and the output is temporary storage of this data and format conversion for processing.

[0234] Step 3:

[0235] (Data transmission)

[0236] The device transmits the collected facial and audio information to the server in real time. The input is the temporarily stored facial and audio information, and the output is a data stream to the server.

[0237] Step 4:

[0238] (Facial expression analysis)

[0239] The server analyzes facial information, extracts facial expression data, and evaluates the driver's level of fatigue and attention. Specifically, the facial expression analysis algorithm detects the frequency of eye opening and closing and the upward and downward movement of the corners of the mouth. The input is the facial information sent to the server, and the output is a numerical evaluation of the driver's level of fatigue and attention.

[0240] Step 5:

[0241] (audio analysis)

[0242] The server analyzes the voice information and evaluates the driver's mental state based on the tone of the voice and the content of what is being said. Specifically, the voice recognition software analyzes changes in the tempo and volume of the voice. The input is the voice information sent to the server, and the output is the evaluation result of the driver's mental state.

[0243] Step 6:

[0244] (comprehensive evaluation)

[0245] The server combines the results of facial expression analysis and voice analysis to quantify the driver's condition. The input is the results of facial expression analysis and voice analysis, and the output is an integrated condition assessment.

[0246] Step 7:

[0247] (Topic generation)

[0248] The server generates appropriate conversation topics based on the condition evaluation results. Using a generative AI model, it generates, for example, "rest suggestions" or "topics of interest" based on the prompt. The input is the condition evaluation results, and the output is the generated conversation topic.

[0249] Step 8:

[0250] (Send message)

[0251] The server sends the generated message to the terminal. The input is the generated conversation topic, and the output is the message data to the terminal.

[0252] Step 9:

[0253] (Notification execution)

[0254] The device receives messages from the server and notifies the driver using a speech synthesis function. Specifically, the message is converted into speech using text-to-speech software. The input is message data from the server, and the output is a voice notification.

[0255] Step 10:

[0256] (Response analysis)

[0257] The device collects the driver's responses with a microphone and sends the audio to a server for analysis. The input is the driver's audio responses, and the output is a data stream to the server for analysis.

[0258] Step 11:

[0259] (Conversation progress)

[0260] The server generates a new topic based on the driver's response and continues the conversation through the device. It uses a generative AI model to generate the next appropriate topic. The input is the response analysis result, and the output is the next conversation topic.

[0261] Step 12:

[0262] (Video data collection)

[0263] The device collects video data from the in-vehicle camera in real time. The input is the surrounding video data, and the output is the video data stored in the device.

[0264] Step 13:

[0265] (Video data transmission)

[0266] The terminal sends the collected video data to the server. The input is the collected video data, and the output is a data stream to the server.

[0267] Step 14:

[0268] (traffic situation analysis)

[0269] The server analyzes the video data and identifies the approach of specific vehicles or emergency vehicles. Specifically, a deep learning model recognizes the lights and sirens of emergency vehicles. The input is the video data sent to the server, and the output is the analysis result of the traffic situation.

[0270] Step 15:

[0271] (Warning message generation)

[0272] The server generates the necessary warning messages based on the analysis results. The input is the traffic situation analysis results, and the output is the generated warning messages.

[0273] Step 16:

[0274] (Alert notice)

[0275] The terminal notifies the driver of the attention message received from the server by voice. The input is the message data from the server, and the output is the voice notification.

[0276] Step 17:

[0277] (Cloud Upload)

[0278] The device uploads the collected video data to the cloud. The input is the collected video data, and the output is a data stream to the cloud.

[0279] Step 18:

[0280] (Cloud analysis)

[0281] The server (cloud) analyzes the uploaded video data and detects dangerous driving vehicles. It uses driving behavior pattern recognition and anomaly detection algorithms. The input is the video data uploaded to the cloud, and the output is the detection results of dangerous driving vehicles.

[0282] Step 19:

[0283] (Send cloud analysis results)

[0284] The server (cloud) generates a warning message based on the analysis results and sends it to the terminal. The input is the cloud analysis result, and the output is the message data to the terminal.

[0285] Step 20:

[0286] (Warning notice)

[0287] The terminal notifies the driver of warning messages received from the server (cloud) by voice. The input is message data from the cloud, and the output is a voice notification.

[0288] (Application example 1)

[0289] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0290] Conventional driver assistance systems have difficulty providing appropriate rest and alerts based on the driver's level of fatigue or reduced attention span. They also lack real-time warnings and notifications to adapt to surrounding traffic conditions. Self-driving vehicles, in particular, are required to appropriately monitor the driver's condition and respond immediately when necessary. Therefore, there is a need for a new system that comprehensively alerts the driver and notifies them of surrounding traffic conditions, supporting safe driving.

[0291] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0292] In this invention, the server includes means for collecting face information and voice information of the driver, means for analyzing the face information and voice information of the driver to determine the condition, means for providing appropriate actions and warnings by synthetic voice based on the determined condition, and means for notifying the driver of traffic conditions by synthetic voice. This makes it possible to analyze the driver's condition in real time, prompt the driver to take a break or be careful at the appropriate time, and support safe driving by providing real-time notifications according to the surrounding traffic conditions.

[0293] "Driver's facial information" is a general term for visual data related to the driver's face, such as facial expression, gaze position, and whether the eyes are open or closed.

[0294] "Voice information" is a general term for data related to the driver's voice, such as what they say, their tone of voice, volume, and voice patterns.

[0295] "Condition" refers to the psychological and physiological state that affects driving, such as the driver's level of fatigue, attention, and physical condition.

[0296] "Synthetic speech" refers to artificial speech generated by converting text data into speech data.

[0297] "Traffic conditions" refers to the state of the external environment while driving, such as surrounding vehicles, road congestion, and the location of emergency vehicles.

[0298] "Warning" refers to providing information to encourage or warn drivers to take specific actions.

[0299] "Video data" refers to visual digital data collected via in-vehicle cameras, etc., and includes image information about surrounding conditions and objects.

[0300] "Cloud" refers to a group of data centers and servers accessible via the Internet, providing a platform for storing and analyzing data.

[0301] "Dangerous driving vehicles" refer to vehicles that exhibit unusual driving behavior and may pose a threat to the safety of those around them.

[0302] "Providing appropriate actions or warnings using synthetic voice" refers to the act of using synthetic voice to communicate messages generated based on the driver's state.

[0303] "HMD" is an abbreviation for head-mounted display, a display device worn on the user's head that displays visual information.

[0304] The present invention is a system that uses facial information and voice information of the driver to maintain the driver's attention and support safe driving. Specific embodiments for carrying out the present invention are described below.

[0305] Hardware Configuration

[0306] Collecting driver's facial and voice information

[0307] In-vehicle camera: An HD camera is used to collect the driver's facial information, which allows for the acquisition of facial expression and gaze data.

[0308] Microphone: The driver's voice information is collected using an in-vehicle microphone or a microphone built into the head-mounted display (HMD).

[0309] Notifications and Interactions

[0310] Head-mounted display (HMD): Functions as a display and provides visual notifications to the driver. It also provides audio notifications using a built-in speaker.

[0311] Software and Data Processing

[0312] OpenCV: A library for image processing and facial recognition. It analyzes video data from an in-vehicle camera and extracts facial landmarks.

[0313] dlib: A library for detecting facial landmarks. Used to analyze facial expressions and gaze.

[0314] speech_recognition: A library that performs speech recognition and analyzes the driver's voice information.

[0315] pyttsx3: A library that performs speech synthesis and notifies the driver via voice of appropriate warnings and actions.

[0316] System Operation

[0317] 1. Collecting facial and audio information

[0318] The system uses an onboard camera and microphone to collect the driver's facial and voice information in real time. Facial information includes facial expressions and eye gaze position, while voice information includes what the driver says and the tone of their voice.

[0319] 2. Driver status analysis

[0320] The server analyzes the collected facial and voice information to determine the driver's condition, specifically quantifying the driver's level of fatigue and attention from facial expressions, eye movements, and voice tone.

[0321] 3. Traffic situation notifications and alerts

[0322] The system analyzes surrounding video data collected by the in-vehicle camera to detect emergency vehicles and vehicles engaged in reckless driving. This analysis uses image recognition technology. Based on the analysis results, a visual notification is displayed on the HMD and a synthesized voice is used to warn the driver.

[0323] Specific examples

[0324] Example 1: If a driver has been driving for a long time, the server determines that the driver's fatigue level is "high" based on facial information. As a result, a message saying "You need to take a break" is displayed on the HMD and a voice notification is also given.

[0325] Example 2: If the server determines that the driver's attention is declining, it generates a message saying "Please concentrate," displays it on the HMD, and also notifies the driver by voice.

[0326] Example 3: Surrounding traffic conditions are collected from an in-car camera. If an emergency vehicle is approaching, the server analyzes the video data and displays a warning message on the HMD saying, "An emergency vehicle is approaching. Please give way.", and also provides an audio notification.

[0327] Example prompts for generative AI models

[0328] Design a system that analyzes the driver's facial and audio information in real time to quantify their level of fatigue and attention. This system should collect data using an on-board camera and microphone, and based on the analysis results, display a message on the HMD and provide an audio notification to encourage the driver to take appropriate action.

[0329] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0330] Step 1:

[0331] The device uses an onboard camera and microphone to collect the driver's facial and audio information in real time.

[0332] Input: Video data from the dashcam and audio data from the microphone.

[0333] Data processing: Face detection and landmark extraction are performed on video data, and voice recognition is performed on audio data.

[0334] Output: Driver's facial expression data, gaze data, and speech-to-text information.

[0335] Step 2:

[0336] The server analyzes the collected facial and audio information to determine the driver's condition.

[0337] Input: Facial expression data, gaze data, speech-to-text information.

[0338] Data calculation: Facial expression analysis, gaze analysis, and voice tone analysis are performed to quantify the driver's fatigue level and attention level.

[0339] Output: Driver condition (e.g. fatigue level or alertness).

[0340] Step 3:

[0341] The server generates a message in synthesized voice based on the determined condition, providing appropriate action or attention.

[0342] Input: Driver condition (fatigue level and attention level).

[0343] Data calculation: Generative AI models are used to generate warning messages that are appropriate for the driver's current situation.

[0344] Output: Synthesized voice message (e.g., "You need a break" or "Please stay focused").

[0345] Step 4:

[0346] The device displays a synthesized voice message on the HMD and notifies the driver by voice.

[0347] Input: Synthesized voice message.

[0348] Data calculation: A message is displayed on the HMD and a voice notification is given through the speaker.

[0349] Output: Visual and audio feedback to the driver.

[0350] Step 5:

[0351] The device collects video data from the onboard camera and uploads it to the cloud.

[0352] Input: Surrounding video data.

[0353] Data processing: cleaning and optimizing video data.

[0354] Output: Optimized video data uploaded to the cloud.

[0355] Step 6:

[0356] The server analyzes the video data uploaded to the cloud and detects vehicles driving recklessly.

[0357] Input: Video data uploaded to the cloud.

[0358] Data Computing: Uses image recognition algorithms to detect abnormal driving patterns.

[0359] Output: Detection results of dangerous driving vehicles.

[0360] Step 7:

[0361] The device will display a warning on the HMD based on the cloud analysis results.

[0362] Input: Detection results of dangerous driving vehicles.

[0363] Data processing: generating warning messages and creating synthetic voices.

[0364] Output: Visual and audio warning notification to the driver.

[0365] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0366] This system uses facial and voice information from the driver to maintain attention and support safe driving. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it provides more appropriate support.

[0367] System Configuration

[0368] 1. Collection of facial and voice information (terminal)

[0369] The device uses the onboard camera and microphone to collect facial and audio information from the driver. Facial information includes facial expressions and gaze data, while audio information includes what is being said and tone of voice.

[0370] 2. Condition Analysis (Server)

[0371] The server analyzes the collected facial and voice information to determine the driver's condition. Specifically, it measures the driver's level of fatigue using facial expression analysis and determines the driver's level of attention through voice tone analysis. As a result, the driver's level of fatigue and attention are output as numerical values.

[0372] 3. Emotion Recognition (Emotion Engine)

[0373] The emotion engine further analyzes the collected facial and audio information to determine the driver's emotional state, which can include anger, sadness, joy, surprise, fear, etc.

[0374] 4. Providing conversation topics (server)

[0375] The server generates appropriate topics based on the driver's condition and emotional state. For example, if the driver is highly fatigued and stressed, it will provide "relaxation topics." If the driver is alert and emotionally stable, it will provide "talk about topics of interest."

[0376] 5. Notifications and conversation initiation (device)

[0377] The device will then provide a voice notification to the driver about the topic received from the server, and the conversation will progress naturally based on the driver's responses.

[0378] Analysis of surrounding traffic conditions and alerts

[0379] 1. Video data collection (terminal)

[0380] The device uses an onboard camera to collect video data about the surrounding traffic, including other vehicles and road conditions.

[0381] 2. Traffic situation analysis (server)

[0382] The server analyzes the video data and applies image recognition algorithms to detect emergency vehicles and approaching vehicles from behind, using technology to identify specific vehicles and patterns.

[0383] 3. Warning (terminal)

[0384] Based on the analysis results, the device will give the driver appropriate warnings. For example, if an emergency vehicle is approaching, the device will notify the driver, "An emergency vehicle is approaching. Please give way."

[0385] Reckless driving vehicle detection and notification

[0386] 1. Upload data to the cloud (device)

[0387] The device uploads the video data collected from the in-vehicle camera to a cloud server, where the cloud environment is equipped with advanced analytical functions, enabling efficient data processing.

[0388] 2. Cloud analysis (server)

[0389] A cloud server analyzes the uploaded data and detects dangerous driving vehicles using algorithms for pattern recognition and anomaly detection of driving behavior.

[0390] 3. Warning notification (terminal)

[0391] If cloud analysis detects a vehicle driving recklessly, the device will warn the driver, for example, by sending a message saying, "There is a vehicle driving recklessly nearby. Please be careful."

[0392] Specific examples

[0393] Specific examples of chat functions

[0394] 1. User: The driver is driving for a long time.

[0395] 2. Device: Collects the driver's facial and voice information using the vehicle's camera and microphone.

[0396] 3. Terminal: Sends the collected information to the server.

[0397] 4. Server: Analyzes the information and determines if the driver is tired.

[0398] 5. Emotion Engine: Detects fatigue and high stress levels at the same time.

[0399] 6. Server: Generate "relaxation topics."

[0400] 7. Terminal: Notifies the driver of the received topic via voice and suggests, "Why don't you think about relaxing for a moment?"

[0401] 8. User: The driver responds, "Well, let's talk about how to relax."

[0402] Examples of alert functions

[0403] 1. Device: Collects surrounding traffic conditions using an in-vehicle camera.

[0404] 2. Terminal: Sends collected video data to the server.

[0405] 3. Server: Analyzes video data and detects emergency vehicles.

[0406] 4. Server: Generate a warning message saying "An emergency vehicle is approaching. Please give way."

[0407] 5. Device: Notifies the driver of the received message via voice prompt and prompts them to give way to the emergency vehicle.

[0408] 6. User: The driver checks behind them and gives way safely.

[0409] Specific examples of dangerous driving vehicle information

[0410] 1. Terminal: Collects surrounding video data using an onboard camera.

[0411] 2. Terminal: Uploads the collected video data to the cloud server.

[0412] 3. Server (Cloud): Analyzes the uploaded data and detects dangerous driving vehicles.

[0413] 4. Server (Cloud): Generates a warning message saying, "There is a dangerously driven vehicle nearby. Please be careful."

[0414] 5. Device: The device will notify the driver of the received message via voice and urge them to be aware of their surroundings.

[0415] 6. User: Drivers should be aware of their surroundings and drive safely.

[0416] The above is a description of the specific processing steps and operations of the system of the present invention, which allows the driver to always receive appropriate support according to the driving situation, thereby enhancing the safe driving environment.

[0417] The processing flow will be explained below.

[0418] Chat function

[0419] Program processing

[0420] Step 1:

[0421] The device uses the vehicle's camera and microphone to collect the driver's facial and voice information. The facial information includes the driver's facial expressions and gaze data, and the voice information includes what is being said and the tone of voice.

[0422] Step 2:

[0423] The terminal transmits the collected face information and voice information to the server.

[0424] Step 3:

[0425] The server analyzes the received facial and voice information to determine the driver's condition. Specifically, it measures the driver's fatigue level using facial expression analysis and determines the driver's level of attention through voice tone analysis.

[0426] Step 4:

[0427] The server sends the results of the assessment to the emotion engine, which further analyzes the driver's emotional state, determining emotions such as anger, sadness, joy, surprise, and fear.

[0428] Step 5:

[0429] The server generates conversation topics appropriate for the driver based on the condition assessment results and the emotion engine's assessment results. For example, if fatigue and stress levels are high, it will provide topics related to relaxation.

[0430] Step 6:

[0431] The server sends the generated conversation topic to the terminal.

[0432] Step 7:

[0433] The device will notify the driver of the topic of conversation received by voice and begin the conversation.

[0434] Alert function

[0435] Program processing

[0436] Step 1:

[0437] The device uses an onboard camera to collect surrounding traffic information, including other vehicles and road conditions.

[0438] Step 2:

[0439] The terminal transmits the collected video data to the server.

[0440] Step 3:

[0441] The server analyzes the video data and applies image recognition algorithms to detect emergency vehicles and rapidly approaching vehicles from behind.

[0442] Step 4:

[0443] If the server detects an abnormality based on the analysis results, it generates a warning message. In the case of an emergency vehicle, it generates a message saying, "An emergency vehicle is approaching. Please give way."

[0444] Step 5:

[0445] The server generates a warning message and sends it to the terminal.

[0446] Step 6:

[0447] The device will notify the driver of any warning messages it receives via voice.

[0448] Sharing and notifying dangerous driving vehicle information

[0449] Program processing

[0450] Step 1:

[0451] The device uses an onboard camera to collect video data of the surrounding area.

[0452] Step 2:

[0453] The video data collected by the device is uploaded to a cloud server.

[0454] Step 3:

[0455] The cloud server analyzes the uploaded video data and detects dangerous driving vehicles using driving behavior pattern recognition and anomaly detection algorithms.

[0456] Step 4:

[0457] Based on the detection result, the cloud server generates a warning message if there is a reckless driving vehicle, for example, a message saying "There is a reckless driving vehicle nearby. Please be careful."

[0458] Step 5:

[0459] The cloud server generates a warning message and sends it to the terminal.

[0460] Step 6:

[0461] The device will notify the driver of any warning messages it receives via voice.

[0462] Specific examples

[0463] Specific examples of chat functions

[0464] Step 1:

[0465] The device uses the onboard camera to collect the driver's facial information (facial expressions) and voice information.

[0466] Step 2:

[0467] The terminal transmits the collected face information and voice information to the server.

[0468] Step 3:

[0469] The server analyzes facial and audio information and determines the driver's level of fatigue as "high."

[0470] Step 4:

[0471] The server sends the fatigue level assessment result to the emotion engine, which then determines that the stress level is also "high."

[0472] Step 5:

[0473] The server generates "relaxation topics" based on "high" fatigue and "high" stress levels.

[0474] Step 6:

[0475] The server sends the generated topics to the terminal.

[0476] Step 7:

[0477] The device will notify the driver of the topic it receives via voice and suggest, "Why don't you think about relaxing for a bit?"

[0478] Examples of alert functions

[0479] Step 1:

[0480] The device uses an onboard camera to collect information about surrounding traffic conditions.

[0481] Step 2:

[0482] The terminal transmits the collected video data to the server.

[0483] Step 3:

[0484] The server analyzes the video data and detects emergency vehicles.

[0485] Step 4:

[0486] The server generates a warning message saying "Emergency vehicle approaching. Please give way."

[0487] Step 5:

[0488] The server generates a message and sends it to the terminal.

[0489] Step 6:

[0490] The device will then notify the driver via voice of the message received, urging them to give way to the emergency vehicle.

[0491] Specific examples of dangerous driving vehicle information

[0492] Step 1:

[0493] The device uses an onboard camera to collect video data of the surrounding area.

[0494] Step 2:

[0495] The video data collected by the device is uploaded to a cloud server.

[0496] Step 3:

[0497] The cloud server analyzes the uploaded data and detects dangerously driving vehicles.

[0498] Step 4:

[0499] The cloud server generates a warning message saying, "There is a dangerously driven vehicle nearby. Please be careful."

[0500] Step 5:

[0501] The cloud server sends the generated message to the terminal.

[0502] Step 6:

[0503] The device will notify the driver of the received message via voice, urging them to be aware of their surroundings.

[0504] Example 2

[0505] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0506] In recent years, road traffic has become increasingly complex, creating a need for systems that can monitor drivers' attention and emotional state in real time to support safe driving. However, current systems only analyze the driver's condition obtained from facial and voice information, and lack the functionality to generate topics to provide appropriate support. Furthermore, they lack the functionality to accurately analyze surrounding traffic conditions, detect reckless driving vehicles early, and issue warnings. Therefore, there is a need for systems that can effectively support safe driving.

[0507] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for analyzing the driver's facial information and voice information to determine the driver's condition, a means for generating a conversation topic based on the driver's condition and emotional state, and a means for notifying the driver of the topic by voice and advancing the conversation. This makes it possible to monitor the driver's attention and emotional state in real time and provide appropriate assistance. Furthermore, by including a means for analyzing video data collected from an on-board camera and accurately analyzing the surrounding traffic conditions, and a means for uploading the collected video data to the cloud and analyzing it on the cloud to detect reckless driving vehicles, it becomes possible to promptly alert and warn the driver.

[0508] "Driver" means a person who drives a vehicle.

[0509] "Facial information" refers to data related to the driver's facial expressions, gaze, etc.

[0510] "Voice information" refers to data related to the driver's voice, such as what they say and their tone of voice.

[0511] "Terminal" refers to input devices such as cameras and microphones installed in the vehicle.

[0512] "Server" refers to the central system that analyzes the collected data and generates appropriate responses based on the results.

[0513] "Emotion engine" refers to a system that determines the driver's emotional state based on facial and voice information.

[0514] "Video data" refers to video information about surrounding traffic conditions collected by an in-vehicle camera.

[0515] "Condition" refers to the driver's level of fatigue and attention.

[0516] "Conversational topics" refer to topics that are generated based on the driver's condition and emotional state.

[0517] "Traffic conditions" refers to the traffic environment, including other vehicles and surrounding road conditions.

[0518] "Caution" refers to a warning message to encourage drivers to drive safely.

[0519] "Dangerous driving vehicle" refers to a vehicle that is driving abnormally in the surrounding traffic conditions and may pose a safety hazard.

[0520] "Cloud" refers to a general term for computer resources provided through multiple servers and storage on the Internet.

[0521] "Cloud analytics" refers to data analysis processing performed on the cloud.

[0522] This invention is a system that uses facial and audio information of the driver to maintain attention and support safe driving. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, more appropriate support is provided. Specific embodiments are described in detail below.

[0523] System configuration and program processing

[0524] 1. Collection of facial and voice information (terminal)

[0525] The device uses an onboard camera and microphone to collect the driver's facial and audio information. Facial information includes facial expressions and gaze data, while audio information includes what is being said and tone of voice. This collection process uses a high-resolution camera and a highly sensitive microphone.

[0526] 2. Condition Analysis (Server)

[0527] The server analyzes the collected facial and audio information to determine the driver's condition. Specifically, it measures the driver's level of fatigue using facial expression analysis and determines the driver's level of attention through voice tone analysis. This analysis uses facial recognition algorithms (e.g., OpenCV and DeepFace) and voice analysis algorithms (e.g., Google Cloud Speech-to-Text API). The resulting numerical output represents the driver's level of fatigue and attention.

[0528] 3. Emotion Recognition (Emotion Engine)

[0529] The emotion engine further analyzes the collected facial and audio information to determine the driver's emotional state, which can include anger, sadness, joy, surprise, fear, etc. This analysis is performed using a deep learning model (e.g., an emotion recognition model using PyTorch or TensorFlow).

[0530] 4. Providing conversation topics (server)

[0531] The server generates appropriate topics based on the determined condition and emotional state. For example, if the driver is highly fatigued and stress levels are rising, it will provide "topics related to relaxation." If the driver is alert and emotionally stable, it will generate "talk about topics of interest." This topic generation uses a natural language generation model (e.g., a generative AI model).

[0532] 5. Notifications and conversation initiation (device)

[0533] The device then notifies the driver of the topic received from the server by voice. The notification is in the form of a message such as, "Would you like to think about relaxing for a moment?" When the driver responds, the system will proceed with the conversation based on that response. This also uses a high-definition speaker to notify the driver by voice.

[0534] Analysis of surrounding traffic conditions and alerts

[0535] 1. Video data collection (terminal)

[0536] The device uses an onboard camera to collect surrounding traffic conditions, including other vehicles and road conditions. The camera has a wide-angle lens, allowing it to collect high-resolution images even at night.

[0537] 2. Traffic situation analysis (server)

[0538] The server analyzes the video data and applies image recognition algorithms to detect emergency vehicles and approaching vehicles from behind, using techniques such as YOLOv5 and Mask R-CNN to identify specific vehicles and patterns.

[0539] 3. Warning (terminal)

[0540] Based on the analysis results, the device will give the driver appropriate warnings. For example, if an emergency vehicle is approaching, the device will notify the driver, "An emergency vehicle is approaching. Please give way."

[0541] Reckless driving vehicle detection and notification

[0542] 1. Upload data to the cloud (device)

[0543] The device uploads the video data collected from the in-vehicle camera to a cloud server, which is equipped with advanced analytical functions, enabling efficient data processing.

[0544] 2. Cloud analysis (server)

[0545] The cloud server analyzes the uploaded data and detects dangerous driving vehicles using driving behavior pattern recognition and anomaly detection algorithms (such as AWS Rekognition and Google Cloud Vision API).

[0546] 3. Warning notification (terminal)

[0547] If cloud analysis detects a vehicle driving recklessly, the device will warn the driver. For example, it will notify the driver by saying, "There is a vehicle driving recklessly nearby. Please be careful." The driver can then remain vigilant and drive safely based on the warning.

[0548] Specific examples

[0549] Specific examples of chat functions

[0550] User: The driver is driving for a long time.

[0551] Device: Collects facial and audio information from the driver using the vehicle's camera and microphone.

[0552] Terminal: Sends collected information to the server.

[0553] Server: Analyzes the information and determines whether the driver is tired.

[0554] Emotion engine: Determines fatigue and high stress levels at the same time.

[0555] Server: Generate "relaxation topics."

[0556] Device: Notifies the driver of the received topic via voice and suggests, "Why don't you think about relaxing for a moment?"

[0557] User: The driver responds, "Well, let's talk about how to relax."

[0558] Example prompt: "When drivers are on the road for long periods of time, come up with ways to talk about relaxation to help them stay alert."

[0559] Examples of alert functions

[0560] Device: Uses an in-car camera to collect information about surrounding traffic conditions.

[0561] Terminal: Sends collected video data to the server.

[0562] Server: Analyzes video data and detects emergency vehicles.

[0563] Server: Generate a warning message saying "Emergency vehicle approaching. Please give way."

[0564] Device: The device will notify the driver of the received message via voice and prompt them to give way to the emergency vehicle.

[0565] User: Drivers should check behind them and give way safely.

[0566] Example prompt: "If an emergency vehicle is approaching, think of a way to quickly notify the driver."

[0567] Specific examples of dangerous driving vehicle information

[0568] Terminal: Collects video data of the surrounding area using an onboard camera.

[0569] Terminal: Uploads collected video data to a cloud server.

[0570] Server (Cloud): Analyzes uploaded data and detects dangerous driving vehicles.

[0571] Server (Cloud): Generates a warning message saying, "There is a dangerously driven vehicle nearby. Please be careful."

[0572] Device: The device will notify the driver of received messages via voice and urge them to be aware of their surroundings.

[0573] User: Drivers should be aware of their surroundings and drive safely.

[0574] Example prompt: "If there is a dangerously driving vehicle nearby, suggest ways to warn the driver and encourage safe driving."

[0575] The above is a specific embodiment of the present invention. By using this system, the driver can always receive appropriate support according to the driving situation, and the safe driving environment is strengthened.

[0576] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0577] Program processing flow and specific operations

[0578] Driver health analysis and support

[0579] Step 1:

[0580] The device uses the onboard camera and microphone to collect the driver's facial and audio information, including facial expression and gaze data, as well as voice tone and content.

[0581] Input: Video and audio data from the onboard camera and microphone

[0582] Output: Collected facial and audio information

[0583] Step 2:

[0584] The terminal transmits the collected face information and voice information to the server.

[0585] Input: Face and voice information

[0586] Output: Data sent to the server

[0587] Step 3:

[0588] The server analyzes the collected facial and voice information to determine the driver's condition. Specifically, it measures the driver's fatigue level using facial expression analysis and analyzes the tone of the voice.

[0589] Input: Face and voice information

[0590] Output: Numerical data on driver fatigue and alertness

[0591] Step 4:

[0592] The emotion engine performs additional analysis to determine the driver's emotional state, specifically by analyzing facial and audio information using deep learning models (e.g., PyTorch) to identify emotions.

[0593] Input: Face and voice information

[0594] Output: Driver's emotional state (e.g., anger, sadness, joy, etc.)

[0595] Step 5:

[0596] The server generates appropriate conversation topics based on your condition and emotional state, using generative AI models to generate topics of relaxation, interest, etc.

[0597] Input: Driver condition and emotional state

[0598] Output: Generated conversation topics

[0599] Step 6:

[0600] The device will then notify the driver of the generated topic by voice and start a conversation. The notification content may be something like, "Why don't you think about relaxing for a bit?"

[0601] Input: Generated conversation topics

[0602] Output: Audio notification to the driver

[0603] Analysis of surrounding traffic conditions and alerts

[0604] Step 7:

[0605] The device uses an onboard camera to collect information about the surrounding traffic conditions, including other vehicles and road conditions.

[0606] Input: Surrounding image data from a camera

[0607] Output: Collected traffic video data

[0608] Step 8:

[0609] The terminal transmits the collected video data to the server.

[0610] Input: Surrounding video data

[0611] Output: Video data sent to the server

[0612] Step 9:

[0613] The server analyzes the video data and determines the traffic situation, using an image recognition algorithm (e.g., YOLOv5) to detect emergency vehicles and vehicles approaching from behind.

[0614] Input: Surrounding traffic video data

[0615] Output: Parsed traffic data

[0616] Step 10:

[0617] Based on the analysis results, the server generates a warning message for the driver.

[0618] Input: Parsed traffic data

[0619] Output: Attention message

[0620] Step 11:

[0621] The device will then notify the driver of the warning message it receives by voice, urging them to take appropriate action. For example, it may say, "An emergency vehicle is approaching. Please give way."

[0622] Input: Attention message

[0623] Output: Audio notification to the driver

[0624] Reckless driving vehicle detection and notification

[0625] Step 12:

[0626] The device uploads the video data collected through the onboard camera to a cloud server.

[0627] Input: Video data

[0628] Output: Uploaded video data

[0629] Step 13:

[0630] The cloud server analyzes the uploaded data and detects reckless driving vehicles using advanced pattern recognition and anomaly detection algorithms (e.g., AWS Rekognition).

[0631] Input: Uploaded video data

[0632] Output: Detected dangerous driving vehicles

[0633] Step 14:

[0634] The cloud server generates a warning message to the driver based on the analysis results.

[0635] Input: Detected dangerous driving vehicles

[0636] Output: Warning message

[0637] Step 15:

[0638] The device will then generate a warning message and notify the driver by voice, urging them to be vigilant about their surroundings. For example, it will say, "There is a vehicle driving recklessly nearby. Please be careful."

[0639] Input: warning message

[0640] Output: Audio notification to the driver

[0641] (Application example 2)

[0642] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0643] In autonomous vehicles, the system certainly takes over most of the driving, but it is impossible to completely eliminate the dangers of driver distraction or fatigue. Furthermore, the driver's emotional state while driving also has a significant impact on driving safety, so appropriate reminders and support are essential. However, conventional systems have had difficulty accurately recognizing the driver's emotional state and providing appropriate real-time support. This has led to problems in effectively ensuring driver safety.

[0644] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting face information and voice information of the driver, means for analyzing the face information and voice information of the driver to determine the condition, means for providing conversation topics according to the condition, means for recognizing the emotional state of the driver, and means for providing voice notification in real time based on the emotional state. This makes it possible to monitor the driver's attention lapse, fatigue, and emotional state in real time and provide appropriate warnings and support.

[0645] "Driver" means the passenger who operates and monitors an automated driving vehicle.

[0646] "Facial information" refers to image data of the driver's face acquired from a camera, and includes facial expression and gaze data.

[0647] "Voice information" refers to data about the driver's speech and tone of voice obtained from a microphone.

[0648] "Condition" refers to the driver's level of attention and fatigue, which is analyzed based on facial and voice information.

[0649] "Emotional state" refers to the driver's emotions (e.g., anger, sadness, joy, etc.) analyzed from facial and voice information.

[0650] "Real-time" means that data collection, analysis, and notification occur immediately.

[0651] The "means for providing conversation topics" is a system that generates appropriate conversation themes and topics based on the driver's condition and emotional state.

[0652] "Means for providing voice notification" refers to a system that uses voice output to alert the driver and provide conversation based on the analysis results.

[0653] "Surrounding traffic conditions" refers to data about other vehicles and road conditions collected through in-vehicle cameras.

[0654] "Warning" refers to warning drivers about road conditions or emergency situations based on analyzed information.

[0655] The "cloud" is a remote server environment that analyzes and stores data via the Internet.

[0656] "Dangerous driving vehicles" refer to nearby vehicles whose abnormal driving behavior has been detected based on the analysis results.

[0657] The "means of issuing a warning" is a system that notifies drivers of the presence of a vehicle engaged in dangerous driving based on the results of cloud analysis.

[0658] This invention is a system that monitors the driver's attention and emotional state in real time in an autonomous vehicle and provides appropriate support and warnings. The system uses an on-board camera and microphone to collect the driver's facial and voice information, and analyzes this data on a server.

[0659] System Configuration

[0660] 1. Data Collection

[0661] Hardware: In-car camera (any USB camera), microphone

[0662] Software: OpenCV(cv2), PyAudio

[0663] Processing: Collecting facial and audio information from the driver. For example, an onboard camera records the driver's facial expressions and gaze, and a microphone records speech and tone of voice.

[0664] 2. Data Analysis

[0665] Hardware: High-performance server

[0666] Software: EmotionEngine (emotion recognition engine), custom voice analysis algorithm

[0667] Processing: The server analyzes the collected facial and audio information to determine the driver's condition and emotional state. Specifically, the server measures the driver's level of fatigue from facial expressions and analyzes their level of attention from the tone of their voice.

[0668] 3. Determining Conditions and Emotional States

[0669] Example: If a driver has been driving for a long time, the server will determine that the driver is tired based on facial information and that the driver's attention level is low based on the tone of voice. In this case, the server will provide the driver with appropriate conversation topics.

[0670] 4. Real-time support and notifications

[0671] Hardware: In-car audio system

[0672] Software: pyttsx3 (speech synthesis library)

[0673] Processing: Based on the results of the assessment, the server generates appropriate conversation topics and warning messages. For example, if the server determines that the driver is tired, it will notify the driver by voice, "Would you like to rest your eyes for a while?"

[0674] 5. Traffic situation analysis

[0675] Hardware: In-car camera

[0676] Software: Image recognition algorithms

[0677] Processing: The onboard camera collects information on surrounding traffic conditions in real time and analyzes emergency vehicles and vehicles approaching rapidly from behind, etc. Based on this, necessary warnings are given to the driver via voice.

[0678] 6. Cloud Analysis

[0679] Hardware: Cloud Server

[0680] Software: Advanced analysis algorithms

[0681] Processing: Collected video data is uploaded to the cloud and analyzed there. The analysis results are fed back to the vehicle in real time, and the presence of reckless driving vehicles is notified.

[0682] Examples and prompts

[0683] Specific examples

[0684] 1. Example of high fatigue: The driver is determined to be tired and a voice notification is given saying, "Would you like to talk about relaxing a bit?"

[0685] 2. Example of inattention: The driver is determined to be inattentive and an appropriate voice warning is given.

[0686] 3. Example of a traffic situation-based warning: If an emergency vehicle is approaching from behind, a voice notification will say, "An emergency vehicle is approaching. Please give way."

[0687] Prompt Sentence Examples

[0688] "The facial image and voice data indicate that the driver is tired. The driver's fatigue level is high and their attention level is low. In this case, please generate conversation topic suggestions that are effective in helping the driver relax."

[0689] conclusion

[0690] This system can improve driving safety by monitoring the driver's attention and emotional state in real time and providing appropriate support and warnings.

[0691] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0692] Step 1:

[0693] The device uses an onboard camera and microphone to collect facial and audio information from the driver.

[0694] Input: Driver's face image, voice data

[0695] How it works: The onboard camera captures the driver's facial expressions and gaze information, and the microphone records the driver's speech and tone of voice. These data are acquired in real time.

[0696] Output: Collected facial image data and voice data

[0697] Step 2:

[0698] The terminal transmits the collected face information and voice information to the server.

[0699] Input: collected facial image data, voice data

[0700] Specific operation: The terminal uploads the data stored in the local buffer to the server via the network.

[0701] Output: Facial image data and audio data sent to the server

[0702] Step 3:

[0703] The server analyzes facial and audio information to determine the driver's condition.

[0704] Input: Facial image data, audio data

[0705] How it works: The EmotionEngine on the server analyzes facial image data and infers the driver's fatigue level and emotional state from their facial expressions. A custom voice analysis algorithm is used on the voice data to determine the driver's level of attention through tone and language analysis.

[0706] Output: Driver condition data (fatigue level, attention state, emotional state)

[0707] Step 4:

[0708] The server generates appropriate conversation topics and reminder messages based on the determined condition and emotional state.

[0709] Input: Driver condition data

[0710] Specific operation: Based on the results of the assessment, the server uses an AI model to generate optimal conversation topics and attention-grabbing content for the driver. For example, if the driver's level of fatigue is high, it will suggest topics related to relaxation, and if the driver's attention level is low, it will generate a warning message.

[0711] Output: Conversation topics, warning messages

[0712] Step 5:

[0713] The device notifies the driver by voice of the conversation topic and warning messages received from the server.

[0714] Input: conversation topics, warning messages

[0715] Specific operation: The device uses a speech synthesis library such as pyttsx3 to convert text data into speech data and notify the driver through the car speaker.

[0716] Output: Voice message notified to the driver

[0717] Step 6:

[0718] The terminal collects video data from the vehicle-mounted camera and transmits it to a server.

[0719] Input: Surrounding traffic video data

[0720] Specific operation: The onboard camera captures the surrounding traffic conditions, collects the video data in real time, and sends it to a server via the network.

[0721] Output: Video data sent to the server

[0722] Step 7:

[0723] The server analyzes the video data to detect emergency vehicles and vehicles driving recklessly.

[0724] Input: Video data

[0725] How it works: Image recognition algorithms on the server process video data to detect specific traffic patterns and unusual driving behavior. The algorithms identify emergency vehicles and rapidly approaching vehicles.

[0726] Output: Analysis of surrounding traffic conditions (location information for emergency vehicles and dangerously driving vehicles)

[0727] Step 8:

[0728] The server generates a warning message for the driver based on the analysis results and sends it to the terminal.

[0729] Input: Traffic situation analysis results

[0730] Specific operation: The server generates a warning message for the driver based on the detected emergency vehicle or dangerous driving vehicle information and sends it to the terminal.

[0731] Output: Warning message

[0732] Step 9:

[0733] The terminal notifies the driver of the warning message received from the server by voice.

[0734] Input: warning message

[0735] Specific operation: The device uses a speech synthesis library to convert the warning message into voice data and notify the driver through the car speaker.

[0736] Output: Audio warning message notified to the driver

[0737] This allows drivers to always receive appropriate support in real time that is tailored to the driving situation, enhancing the safe driving environment.

[0738] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0739] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0740] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0741] [Second embodiment]

[0742] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0743] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0744] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0745] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0746] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0747] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0748] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0749] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0750] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0751] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0752] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0753] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0754] This invention is a system that uses facial and audio information from the driver to maintain attention and support safe driving. The system works by collecting and analyzing data via an on-board camera and microphone.

[0755] System Configuration

[0756] 1. Collection of facial and voice information (terminal)

[0757] The device uses the onboard camera and microphone to collect facial and audio information from the driver. Facial information includes facial expressions and gaze data, while audio information includes what is being said and tone of voice.

[0758] 2. Condition Analysis (Server)

[0759] The server analyzes the collected facial and voice information to determine the driver's condition. Specifically, it measures the driver's level of fatigue using facial expression analysis and determines the driver's level of attention through voice tone analysis. As a result, the driver's level of fatigue and attention are output as numerical values.

[0760] 3. Providing conversation topics (server)

[0761] The server generates appropriate topics based on the determined condition. For example, if the driver is highly fatigued, it might suggest taking a break, or if the driver is alert, it might suggest talking about an interesting topic.

[0762] 4. Notifications and conversation initiation (device)

[0763] The device will then provide a voice notification to the driver about the topic received from the server, and the conversation will progress naturally based on the driver's responses.

[0764] Analysis of surrounding traffic conditions and alerts

[0765] 1. Video data collection (terminal)

[0766] The device collects video data from the in-vehicle camera, including information on surrounding vehicles and road conditions.

[0767] 2. Traffic situation analysis (server)

[0768] The server analyzes the video data to detect emergency vehicles and vehicles approaching from behind, using image recognition technology to identify specific vehicles and patterns.

[0769] 3. Warning (terminal)

[0770] Based on the analysis results, the device will give the driver appropriate warnings. For example, if an emergency vehicle is approaching, the device will notify the driver, "An emergency vehicle is approaching. Please give way."

[0771] Reckless driving vehicle detection and notification

[0772] 1. Upload data to the cloud (device)

[0773] The video data collected by the device is uploaded to the cloud, where advanced analytical functions enable efficient data processing.

[0774] 2. Cloud analysis (server)

[0775] A cloud server analyzes the uploaded data and detects dangerous driving vehicles using algorithms for pattern recognition and anomaly detection of driving behavior.

[0776] 3. Warning notification (terminal)

[0777] If cloud analysis detects a vehicle driving recklessly, the device will warn the driver, for example, by sending a message saying, "There is a vehicle driving recklessly nearby. Please be careful."

[0778] Specific examples

[0779] Example 1: When a driver becomes tired while driving

[0780] User: The user (driver) is driving for a long time.

[0781] Device: The onboard camera collects the driver's facial expression data, and the microphone collects audio data.

[0782] Server: The server analyzes the collected data and determines the driver's fatigue level as "high."

[0783] Server: The server generates the topic "Would you like to take a break?"

[0784] Device: The device will prompt the driver with a voice message asking, "Would you like to take a break?"

[0785] User: The driver responds, "Yeah, let's take a break," and takes a break.

[0786] Example 2: When an emergency vehicle is approaching

[0787] Terminal: The onboard camera collects video data of the surrounding area.

[0788] Server: The server analyzes the video data and detects emergency vehicles.

[0789] Server: The server generates a warning message saying "Emergency vehicle approaching. Please give way."

[0790] Device: The device will notify the driver via voice message, "An emergency vehicle is approaching. Please give way."

[0791] User: Drivers should check behind them and give way safely.

[0792] Example 3: Reckless driving vehicles nearby

[0793] Terminal: The onboard camera collects video data of the surrounding area.

[0794] Device: The device uploads the video data to the cloud.

[0795] Server (Cloud): The server analyzes the uploaded data and detects dangerous driving vehicles.

[0796] Server (Cloud): The server generates a warning message saying, "There is a dangerously driven vehicle nearby. Please be careful."

[0797] Device: The device will notify the driver by voice, "There is a vehicle driving recklessly nearby. Please be careful."

[0798] User: Drivers should be aware of their surroundings and drive safely.

[0799] This ensures that drivers always receive appropriate support tailored to the driving situation, enhancing the safe driving environment.

[0800] The processing flow will be explained below.

[0801] Chat function

[0802] Program processing

[0803] Step 1:

[0804] The device uses the vehicle's camera and microphone to collect facial and audio information from the driver, including their facial expressions, gaze, and tone of voice.

[0805] Step 2:

[0806] The terminal transmits the collected face information and voice information to the server.

[0807] Step 3:

[0808] The server analyzes the received facial and voice information to determine the driver's condition. Specifically, it measures the driver's fatigue level using facial expression analysis and determines the driver's level of attention through voice tone analysis.

[0809] Step 4:

[0810] Based on the results of the assessment, the server generates conversation topics that correspond to the driver's condition. For example, if the driver is highly fatigued, it will suggest taking a break, and if the driver is alert, it will suggest talking about a topic of interest.

[0811] Step 5:

[0812] The server sends the generated conversation topic to the terminal.

[0813] Step 6:

[0814] The device will notify the driver of the topic of conversation received by voice and begin the conversation.

[0815] Alert function

[0816] Program processing

[0817] Step 1:

[0818] The device uses an onboard camera to collect video data about the surrounding traffic, including other vehicles and road conditions.

[0819] Step 2:

[0820] The terminal transmits the collected video data to the server.

[0821] Step 3:

[0822] The server analyzes the video data and applies image recognition algorithms to detect emergency vehicles and rapidly approaching vehicles from behind.

[0823] Step 4:

[0824] If the server detects an abnormality based on the analysis results, it generates a warning message. For example, in the case of an emergency vehicle, it generates a message saying, "An emergency vehicle is approaching. Please give way."

[0825] Step 5:

[0826] The server generates a warning message and sends it to the terminal.

[0827] Step 6:

[0828] The device will notify the driver of any warning messages it receives via voice.

[0829] Sharing and notifying dangerous driving vehicle information

[0830] Program processing

[0831] Step 1:

[0832] The device uses an onboard camera to collect video data of the surrounding area.

[0833] Step 2:

[0834] The video data collected by the device is uploaded to a cloud server.

[0835] Step 3:

[0836] The cloud server analyzes the uploaded video data to detect dangerous driving vehicles, using algorithms that identify specific driving behaviors.

[0837] Step 4:

[0838] Based on the detection result, the cloud server generates a warning message if there is a reckless driving vehicle, for example, a message saying "There is a reckless driving vehicle nearby. Please be careful."

[0839] Step 5:

[0840] The cloud server generates a warning message and sends it to the terminal.

[0841] Step 6:

[0842] The device will notify the driver of any warning messages it receives via voice.

[0843] Specific examples

[0844] Specific examples of chat functions

[0845] Step 1:

[0846] The device uses the onboard camera and microphone to collect the driver's facial and voice information.

[0847] Step 2:

[0848] The device sends the collected information to the server.

[0849] Step 3:

[0850] The server analyzes the information and determines that the driver is tired.

[0851] Step 4:

[0852] The server generates a conversation topic suggesting a break, such as "Would you like to take a short break?"

[0853] Step 5:

[0854] The server sends the generated topics to the terminal.

[0855] Step 6:

[0856] The device will notify the driver of the topic it receives via voice and suggest, "Would you like to take a short break?"

[0857] Examples of alert functions

[0858] Step 1:

[0859] The device uses an onboard camera to collect information about surrounding traffic conditions.

[0860] Step 2:

[0861] The terminal transmits the collected video data to the server.

[0862] Step 3:

[0863] The server analyzes the video data and detects emergency vehicles.

[0864] Step 4:

[0865] The server generates a warning message saying "Emergency vehicle approaching. Please give way."

[0866] Step 5:

[0867] The server generates a message and sends it to the terminal.

[0868] Step 6:

[0869] The device will then notify the driver via voice of the message received, urging them to give way to the emergency vehicle.

[0870] Specific examples of dangerous driving vehicle information

[0871] Step 1:

[0872] The device uses an onboard camera to collect video data of the surrounding area.

[0873] Step 2:

[0874] The video data collected by the device is uploaded to a cloud server.

[0875] Step 3:

[0876] The cloud server analyzes the video data and detects vehicles driving recklessly.

[0877] Step 4:

[0878] The cloud server generates a warning message saying, "There is a dangerously driven vehicle nearby. Please be careful."

[0879] Step 5:

[0880] The cloud server sends the generated message to the terminal.

[0881] Step 6:

[0882] The device will notify the driver of the received message via voice, urging them to be aware of their surroundings.

[0883] The above is a description of the specific processing steps and operations in the system of the present invention.

[0884] Example 1

[0885] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0886] In today's modern traffic society, there are many accidents that are believed to be caused by driver inattention or fatigue. Furthermore, there is a need for appropriate responses to surrounding traffic conditions and the presence of dangerously driven vehicles. However, conventional systems have had the problem of being unable to accurately grasp the driver's condition and traffic situation in real time and provide appropriate support and warnings.

[0887] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0888] In this invention, the server includes means for collecting face information and voice information of the driver, means for analyzing the face information and voice information of the driver to determine the driver's condition, means for providing a topic of conversation according to the condition, means for notifying the driver of the topic of conversation, means for analyzing the driver's responses and advancing the conversation, means for collecting video data from an in-vehicle camera, means for analyzing the video data to analyze surrounding traffic conditions, means for issuing a warning to the driver based on the traffic conditions, means for uploading the collected video data to a cloud, means for analyzing the video data on the cloud and detecting reckless driving vehicles, and means for issuing a warning to the driver based on the cloud analysis results. This makes it possible to detect a driver's dwindling attention or fatigue in real time, and maintain attention by providing appropriate topics, while also issuing appropriate warnings regarding surrounding traffic conditions and detecting and notifying reckless driving vehicles.

[0889] "Driver's facial information" is data including the driver's facial features such as facial expression and gaze.

[0890] "Voice information" is data that includes voice characteristics such as the driver's tone of voice and what is being said.

[0891] The "means for collecting face information and voice information" refers to a device that acquires the driver's face information and voice information using an in-vehicle camera, microphone, etc.

[0892] The "means for determining the condition" is a device or algorithm that analyzes the collected facial and audio information to determine the driver's level of fatigue and attention.

[0893] A "means for providing conversation topics" is a device or algorithm that generates appropriate conversation topics based on the driver's condition.

[0894] "Means for notifying the driver of the topic of conversation" refers to a device that audibly or visually conveys the generated topic of conversation to the driver through a receiver.

[0895] The "means for analyzing driver responses and advancing the conversation" refers to a device or algorithm that collects responses from the driver, analyzes their content, generates new conversation topics, and advances the conversation.

[0896] The "means for collecting video data from an in-vehicle camera" is a device that acquires video data of the surroundings using a camera mounted on a vehicle.

[0897] The "means for analyzing the surrounding traffic conditions by analyzing the video data" refers to a device or algorithm that analyzes the acquired video data and identifies the traffic conditions of surrounding vehicles, emergency vehicles, etc.

[0898] The "means for issuing a warning to the driver based on the traffic conditions" is a device that issues an appropriate warning to the driver based on the analysis results.

[0899] The "means for uploading video data to the cloud" is a device that transfers collected video data to a cloud environment via the Internet.

[0900] The "means for analyzing video data on the cloud and detecting reckless driving vehicles" refers to a device or algorithm that analyzes video data in a cloud environment and detects signs of reckless driving.

[0901] The "means for issuing a warning to the driver based on the results of cloud analysis" is a device that issues a warning to the driver for safe driving based on the results of cloud analysis.

[0902] This invention is a system that uses facial and audio information of a driver to maintain attention and support safe driving. This system operates by collecting and analyzing data via an on-board camera and microphone. Specific embodiments of this system are described below.

[0903] System Configuration

[0904] Collecting facial and audio information

[0905] The device uses an onboard camera (e.g., a network camera) and microphone (e.g., a high-sensitivity microphone) to collect the driver's facial and audio information. Facial information includes the driver's facial expressions and gaze data. Audio information includes what the driver says and the tone of their voice. For example, the camera captures the driver's eye movements and facial expressions, and the microphone records the pitch and speed of their voice.

[0906] Condition Analysis

[0907] The server analyzes the collected facial and audio information to determine the driver's condition. Specifically, it measures the driver's level of fatigue using facial expression analysis (e.g., image processing algorithms) and determines the driver's level of attention through voice tone analysis (e.g., voice recognition software). As a result, the driver's level of fatigue and attention state are output as numerical values. For example, frequent eye closure and a restless voice are detected as signs of fatigue.

[0908] Providing topics for conversation

[0909] The server generates appropriate topics based on the determined condition. For example, if the driver is highly fatigued, it will "suggest a break," and if the driver is alert, it will "talk about a topic of interest." This process uses a generative AI model, which generates, for example, "driver's interests" based on the prompt sentence.

[0910] Notifications and conversation starters

[0911] The device then notifies the driver of the topic received from the server by voice. Specifically, it plays back a message generated using speech synthesis software (e.g., text-to-speech software). The conversation progresses naturally based on the driver's response. For example, the device may say, "Do you want to take a short break?" and the driver may respond, "Yes, let's take a short break."

[0912] Analysis of surrounding traffic conditions and alerts

[0913] The device collects video data from the onboard camera. This camera captures surrounding vehicles and road conditions in real time. The video data is sent to a server, which analyzes it to detect approaching emergency vehicles and rapidly approaching vehicles from behind. Image recognition technology (e.g., deep learning models) is used for the analysis. Based on the analysis results, the device issues appropriate warnings to the driver. For example, it may notify the driver, "An emergency vehicle is approaching. Please give way."

[0914] Reckless driving vehicle detection and notification

[0915] The video data collected by the device is uploaded to the cloud. The cloud environment is equipped with advanced analytical functions; for example, a server on the cloud analyzes the uploaded data to detect vehicles that are driving recklessly. The analysis uses driving behavior pattern recognition and anomaly detection algorithms. If a vehicle is detected as a result of cloud analysis, the device will notify the driver, "There is a vehicle that is driving recklessly nearby. Please be careful."

[0916] Specific examples

[0917] Example 1: When a driver becomes tired while driving

[0918] User: The driver is driving for a long time.

[0919] Device: The onboard camera collects the driver's facial expression data, and the microphone collects audio data.

[0920] Server: The server analyzes the collected data and determines the driver's fatigue level as "high."

[0921] Server: The server generates the message "Would you like to take a break?"

[0922] Device: Your device will say, "Would you like to take a break?"

[0923] User: The driver responds, "Yeah, let's take a break," and takes a break.

[0924] Example 2: When an emergency vehicle is approaching

[0925] Terminal: The onboard camera collects video data of the surrounding area.

[0926] Server: The server analyzes the video data and detects emergency vehicles.

[0927] Server: The server generates the message "Emergency vehicle approaching. Please give way."

[0928] Device: The device will announce "An emergency vehicle is approaching. Please give way."

[0929] User: Drivers should check behind them and give way safely.

[0930] Example 3: Reckless driving vehicles nearby

[0931] Device: The onboard camera collects images of the surrounding area.

[0932] Device: The device uploads the video data to the cloud.

[0933] Server (Cloud): The server analyzes the uploaded data and detects dangerous driving vehicles.

[0934] Server (Cloud): The server generates a message saying, "There is a dangerously driven vehicle nearby. Please be careful."

[0935] Device: The device will announce with a voice message, "There is a dangerously driven vehicle nearby. Please be careful."

[0936] User: The driver remains alert to their surroundings and continues driving safely.

[0937] The system is designed to provide drivers with appropriate support in real time while driving, thereby contributing to an enhanced safe driving environment.

[0938] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0939] Step 1:

[0940] (Data collection begins)

[0941] The device activates the on-board camera and microphone and starts collecting the driver's facial and audio information. The input is real-time data from the on-board camera and microphone, and the output is stored in the device as facial and audio information.

[0942] Step 2:

[0943] (Data import)

[0944] The device uses a camera to capture the driver's facial image and facial expressions, and a microphone to pick up audio. Specifically, the camera recognizes each part of the face (eyes, mouth, etc.), and the microphone records the pitch and speed of the audio. The input is video and audio data captured in real time, and the output is temporary storage of this data and format conversion for processing.

[0945] Step 3:

[0946] (Data transmission)

[0947] The device transmits the collected facial and audio information to the server in real time. The input is the temporarily stored facial and audio information, and the output is a data stream to the server.

[0948] Step 4:

[0949] (Facial expression analysis)

[0950] The server analyzes facial information, extracts facial expression data, and evaluates the driver's level of fatigue and attention. Specifically, the facial expression analysis algorithm detects the frequency of eye opening and closing and the upward and downward movement of the corners of the mouth. The input is the facial information sent to the server, and the output is a numerical evaluation of the driver's level of fatigue and attention.

[0951] Step 5:

[0952] (audio analysis)

[0953] The server analyzes the voice information and evaluates the driver's mental state based on the tone of the voice and the content of what is being said. Specifically, the voice recognition software analyzes changes in the tempo and volume of the voice. The input is the voice information sent to the server, and the output is the evaluation result of the driver's mental state.

[0954] Step 6:

[0955] (comprehensive evaluation)

[0956] The server combines the results of facial expression analysis and voice analysis to quantify the driver's condition. The input is the results of facial expression analysis and voice analysis, and the output is an integrated condition assessment.

[0957] Step 7:

[0958] (Topic generation)

[0959] The server generates appropriate conversation topics based on the condition evaluation results. Using a generative AI model, it generates, for example, "rest suggestions" or "topics of interest" based on the prompt. The input is the condition evaluation results, and the output is the generated conversation topic.

[0960] Step 8:

[0961] (Send message)

[0962] The server sends the generated message to the terminal. The input is the generated conversation topic, and the output is the message data to the terminal.

[0963] Step 9:

[0964] (Notification execution)

[0965] The device receives messages from the server and notifies the driver using a speech synthesis function. Specifically, the message is converted into speech using text-to-speech software. The input is message data from the server, and the output is a voice notification.

[0966] Step 10:

[0967] (Response analysis)

[0968] The device collects the driver's responses with a microphone and sends the audio to a server for analysis. The input is the driver's audio responses, and the output is a data stream to the server for analysis.

[0969] Step 11:

[0970] (Conversation progress)

[0971] The server generates a new topic based on the driver's response and continues the conversation through the device. It uses a generative AI model to generate the next appropriate topic. The input is the response analysis result, and the output is the next conversation topic.

[0972] Step 12:

[0973] (Video data collection)

[0974] The device collects video data from the in-vehicle camera in real time. The input is the surrounding video data, and the output is the video data stored in the device.

[0975] Step 13:

[0976] (Video data transmission)

[0977] The terminal sends the collected video data to the server. The input is the collected video data, and the output is a data stream to the server.

[0978] Step 14:

[0979] (traffic situation analysis)

[0980] The server analyzes the video data and identifies the approach of specific vehicles or emergency vehicles. Specifically, a deep learning model recognizes the lights and sirens of emergency vehicles. The input is the video data sent to the server, and the output is the analysis result of the traffic situation.

[0981] Step 15:

[0982] (Warning message generation)

[0983] The server generates the necessary warning messages based on the analysis results. The input is the traffic situation analysis results, and the output is the generated warning messages.

[0984] Step 16:

[0985] (Alert notice)

[0986] The terminal notifies the driver of the attention message received from the server by voice. The input is the message data from the server, and the output is the voice notification.

[0987] Step 17:

[0988] (Cloud Upload)

[0989] The device uploads the collected video data to the cloud. The input is the collected video data, and the output is a data stream to the cloud.

[0990] Step 18:

[0991] (Cloud analysis)

[0992] The server (cloud) analyzes the uploaded video data and detects dangerous driving vehicles. It uses driving behavior pattern recognition and anomaly detection algorithms. The input is the video data uploaded to the cloud, and the output is the detection results of dangerous driving vehicles.

[0993] Step 19:

[0994] (Send cloud analysis results)

[0995] The server (cloud) generates a warning message based on the analysis results and sends it to the terminal. The input is the cloud analysis result, and the output is the message data to the terminal.

[0996] Step 20:

[0997] (Warning notice)

[0998] The terminal notifies the driver of warning messages received from the server (cloud) by voice. The input is message data from the cloud, and the output is a voice notification.

[0999] (Application example 1)

[1000] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1001] Conventional driver assistance systems have difficulty providing appropriate rest and alerts based on the driver's level of fatigue or reduced attention span. They also lack real-time warnings and notifications to adapt to surrounding traffic conditions. Self-driving vehicles, in particular, are required to appropriately monitor the driver's condition and respond immediately when necessary. Therefore, there is a need for a new system that comprehensively alerts the driver and notifies them of surrounding traffic conditions, supporting safe driving.

[1002] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1003] In this invention, the server includes means for collecting face information and voice information of the driver, means for analyzing the face information and voice information of the driver to determine the condition, means for providing appropriate actions and warnings by synthetic voice based on the determined condition, and means for notifying the driver of traffic conditions by synthetic voice. This makes it possible to analyze the driver's condition in real time, prompt the driver to take a break or be careful at the appropriate time, and support safe driving by providing real-time notifications according to the surrounding traffic conditions.

[1004] "Driver's facial information" is a general term for visual data related to the driver's face, such as facial expression, gaze position, and whether the eyes are open or closed.

[1005] "Voice information" is a general term for data related to the driver's voice, such as what they say, their tone of voice, volume, and voice patterns.

[1006] "Condition" refers to the psychological and physiological state that affects driving, such as the driver's level of fatigue, attention, and physical condition.

[1007] "Synthetic speech" refers to artificial speech generated by converting text data into speech data.

[1008] "Traffic conditions" refers to the state of the external environment while driving, such as surrounding vehicles, road congestion, and the location of emergency vehicles.

[1009] "Warning" refers to providing information to encourage or warn drivers to take specific actions.

[1010] "Video data" refers to visual digital data collected via in-vehicle cameras, etc., and includes image information about surrounding conditions and objects.

[1011] "Cloud" refers to a group of data centers and servers accessible via the Internet, providing a platform for storing and analyzing data.

[1012] "Dangerous driving vehicles" refer to vehicles that exhibit unusual driving behavior and may pose a threat to the safety of those around them.

[1013] "Providing appropriate actions or warnings using synthetic voice" refers to the act of using synthetic voice to communicate messages generated based on the driver's state.

[1014] "HMD" is an abbreviation for head-mounted display, a display device worn on the user's head that displays visual information.

[1015] The present invention is a system that uses facial information and voice information of the driver to maintain the driver's attention and support safe driving. Specific embodiments for carrying out the present invention are described below.

[1016] Hardware Configuration

[1017] Collecting driver's facial and voice information

[1018] In-vehicle camera: An HD camera is used to collect the driver's facial information, which allows for the acquisition of facial expression and gaze data.

[1019] Microphone: The driver's voice information is collected using an in-vehicle microphone or a microphone built into the head-mounted display (HMD).

[1020] Notifications and Interactions

[1021] Head-mounted display (HMD): Functions as a display and provides visual notifications to the driver. It also provides audio notifications using a built-in speaker.

[1022] Software and Data Processing

[1023] OpenCV: A library for image processing and facial recognition. It analyzes video data from an in-vehicle camera and extracts facial landmarks.

[1024] dlib: A library for detecting facial landmarks. Used to analyze facial expressions and gaze.

[1025] speech_recognition: A library that performs speech recognition and analyzes the driver's voice information.

[1026] pyttsx3: A library that performs speech synthesis and notifies the driver via voice of appropriate warnings and actions.

[1027] System Operation

[1028] 1. Collecting facial and audio information

[1029] The system uses an onboard camera and microphone to collect the driver's facial and voice information in real time. Facial information includes facial expressions and eye gaze position, while voice information includes what the driver says and the tone of their voice.

[1030] 2. Driver status analysis

[1031] The server analyzes the collected facial and voice information to determine the driver's condition, specifically quantifying the driver's level of fatigue and attention from facial expressions, eye movements, and voice tone.

[1032] 3. Traffic situation notifications and alerts

[1033] The system analyzes surrounding video data collected by the in-vehicle camera to detect emergency vehicles and vehicles engaged in reckless driving. This analysis uses image recognition technology. Based on the analysis results, a visual notification is displayed on the HMD and a synthesized voice is used to warn the driver.

[1034] Specific examples

[1035] Example 1: If a driver has been driving for a long time, the server determines that the driver's fatigue level is "high" based on facial information. As a result, a message saying "You need to take a break" is displayed on the HMD and a voice notification is also given.

[1036] Example 2: If the server determines that the driver's attention is declining, it generates a message saying "Please concentrate," displays it on the HMD, and also notifies the driver by voice.

[1037] Example 3: Surrounding traffic conditions are collected from an in-car camera. If an emergency vehicle is approaching, the server analyzes the video data and displays a warning message on the HMD saying, "An emergency vehicle is approaching. Please give way.", and also provides an audio notification.

[1038] Example prompts for generative AI models

[1039] Design a system that analyzes the driver's facial and audio information in real time to quantify their level of fatigue and attention. This system should collect data using an on-board camera and microphone, and based on the analysis results, display a message on the HMD and provide an audio notification to encourage the driver to take appropriate action.

[1040] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1041] Step 1:

[1042] The device uses an onboard camera and microphone to collect the driver's facial and audio information in real time.

[1043] Input: Video data from the dashcam and audio data from the microphone.

[1044] Data processing: Face detection and landmark extraction are performed on video data, and voice recognition is performed on audio data.

[1045] Output: Driver's facial expression data, gaze data, and speech-to-text information.

[1046] Step 2:

[1047] The server analyzes the collected facial and audio information to determine the driver's condition.

[1048] Input: Facial expression data, gaze data, speech-to-text information.

[1049] Data calculation: Facial expression analysis, gaze analysis, and voice tone analysis are performed to quantify the driver's fatigue level and attention level.

[1050] Output: Driver condition (e.g. fatigue level or alertness).

[1051] Step 3:

[1052] The server generates a message in synthesized voice based on the determined condition, providing appropriate action or attention.

[1053] Input: Driver condition (fatigue level and attention level).

[1054] Data calculation: Generative AI models are used to generate warning messages that are appropriate for the driver's current situation.

[1055] Output: Synthesized voice message (e.g., "You need a break" or "Please stay focused").

[1056] Step 4:

[1057] The device displays a synthesized voice message on the HMD and notifies the driver by voice.

[1058] Input: Synthesized voice message.

[1059] Data calculation: A message is displayed on the HMD and a voice notification is given through the speaker.

[1060] Output: Visual and audio feedback to the driver.

[1061] Step 5:

[1062] The device collects video data from the onboard camera and uploads it to the cloud.

[1063] Input: Surrounding video data.

[1064] Data processing: cleaning and optimizing video data.

[1065] Output: Optimized video data uploaded to the cloud.

[1066] Step 6:

[1067] The server analyzes the video data uploaded to the cloud and detects vehicles driving recklessly.

[1068] Input: Video data uploaded to the cloud.

[1069] Data Computing: Uses image recognition algorithms to detect abnormal driving patterns.

[1070] Output: Detection results of dangerous driving vehicles.

[1071] Step 7:

[1072] The device will display a warning on the HMD based on the cloud analysis results.

[1073] Input: Detection results of dangerous driving vehicles.

[1074] Data processing: generating warning messages and creating synthetic voices.

[1075] Output: Visual and audio warning notification to the driver.

[1076] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1077] This system uses facial and voice information from the driver to maintain attention and support safe driving. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it provides more appropriate support.

[1078] System Configuration

[1079] 1. Collection of facial and voice information (terminal)

[1080] The device uses the onboard camera and microphone to collect facial and audio information from the driver. Facial information includes facial expressions and gaze data, while audio information includes what is being said and tone of voice.

[1081] 2. Condition Analysis (Server)

[1082] The server analyzes the collected facial and voice information to determine the driver's condition. Specifically, it measures the driver's level of fatigue using facial expression analysis and determines the driver's level of attention through voice tone analysis. As a result, the driver's level of fatigue and attention are output as numerical values.

[1083] 3. Emotion Recognition (Emotion Engine)

[1084] The emotion engine further analyzes the collected facial and audio information to determine the driver's emotional state, which can include anger, sadness, joy, surprise, fear, etc.

[1085] 4. Providing conversation topics (server)

[1086] The server generates appropriate topics based on the driver's condition and emotional state. For example, if the driver is highly fatigued and stressed, it will provide "relaxation topics." If the driver is alert and emotionally stable, it will provide "talk about topics of interest."

[1087] 5. Notifications and conversation initiation (device)

[1088] The device will then provide a voice notification to the driver about the topic received from the server, and the conversation will progress naturally based on the driver's responses.

[1089] Analysis of surrounding traffic conditions and alerts

[1090] 1. Video data collection (terminal)

[1091] The device uses an onboard camera to collect video data about the surrounding traffic, including other vehicles and road conditions.

[1092] 2. Traffic situation analysis (server)

[1093] The server analyzes the video data and applies image recognition algorithms to detect emergency vehicles and approaching vehicles from behind, using technology to identify specific vehicles and patterns.

[1094] 3. Warning (terminal)

[1095] Based on the analysis results, the device will give the driver appropriate warnings. For example, if an emergency vehicle is approaching, the device will notify the driver, "An emergency vehicle is approaching. Please give way."

[1096] Reckless driving vehicle detection and notification

[1097] 1. Upload data to the cloud (device)

[1098] The device uploads the video data collected from the in-vehicle camera to a cloud server, where the cloud environment is equipped with advanced analytical functions, enabling efficient data processing.

[1099] 2. Cloud analysis (server)

[1100] A cloud server analyzes the uploaded data and detects dangerous driving vehicles using algorithms for pattern recognition and anomaly detection of driving behavior.

[1101] 3. Warning notification (terminal)

[1102] If cloud analysis detects a vehicle driving recklessly, the device will warn the driver, for example, by sending a message saying, "There is a vehicle driving recklessly nearby. Please be careful."

[1103] Specific examples

[1104] Specific examples of chat functions

[1105] 1. User: The driver is driving for a long time.

[1106] 2. Device: Collects the driver's facial and voice information using the vehicle's camera and microphone.

[1107] 3. Terminal: Sends the collected information to the server.

[1108] 4. Server: Analyzes the information and determines if the driver is tired.

[1109] 5. Emotion Engine: Detects fatigue and high stress levels at the same time.

[1110] 6. Server: Generate "relaxation topics."

[1111] 7. Terminal: Notifies the driver of the received topic via voice and suggests, "Why don't you think about relaxing for a moment?"

[1112] 8. User: The driver responds, "Well, let's talk about how to relax."

[1113] Examples of alert functions

[1114] 1. Device: Collects surrounding traffic conditions using an in-vehicle camera.

[1115] 2. Terminal: Sends collected video data to the server.

[1116] 3. Server: Analyzes video data and detects emergency vehicles.

[1117] 4. Server: Generate a warning message saying "An emergency vehicle is approaching. Please give way."

[1118] 5. Device: Notifies the driver of the received message via voice prompt and prompts them to give way to the emergency vehicle.

[1119] 6. User: The driver checks behind them and gives way safely.

[1120] Specific examples of dangerous driving vehicle information

[1121] 1. Terminal: Collects surrounding video data using an onboard camera.

[1122] 2. Terminal: Uploads the collected video data to the cloud server.

[1123] 3. Server (Cloud): Analyzes the uploaded data and detects dangerous driving vehicles.

[1124] 4. Server (Cloud): Generates a warning message saying, "There is a dangerously driven vehicle nearby. Please be careful."

[1125] 5. Device: The device will notify the driver of the received message via voice and urge them to be aware of their surroundings.

[1126] 6. User: Drivers should be aware of their surroundings and drive safely.

[1127] The above is a description of the specific processing steps and operations of the system of the present invention, which allows the driver to always receive appropriate support according to the driving situation, thereby enhancing the safe driving environment.

[1128] The processing flow will be explained below.

[1129] Chat function

[1130] Program processing

[1131] Step 1:

[1132] The device uses the vehicle's camera and microphone to collect the driver's facial and voice information. The facial information includes the driver's facial expressions and gaze data, and the voice information includes what is being said and the tone of voice.

[1133] Step 2:

[1134] The terminal transmits the collected face information and voice information to the server.

[1135] Step 3:

[1136] The server analyzes the received facial and voice information to determine the driver's condition. Specifically, it measures the driver's fatigue level using facial expression analysis and determines the driver's level of attention through voice tone analysis.

[1137] Step 4:

[1138] The server sends the results of the assessment to the emotion engine, which further analyzes the driver's emotional state, determining emotions such as anger, sadness, joy, surprise, and fear.

[1139] Step 5:

[1140] The server generates conversation topics appropriate for the driver based on the condition assessment results and the emotion engine's assessment results. For example, if fatigue and stress levels are high, it will provide topics related to relaxation.

[1141] Step 6:

[1142] The server sends the generated conversation topic to the terminal.

[1143] Step 7:

[1144] The device will notify the driver of the topic of conversation received by voice and begin the conversation.

[1145] Alert function

[1146] Program processing

[1147] Step 1:

[1148] The device uses an onboard camera to collect surrounding traffic information, including other vehicles and road conditions.

[1149] Step 2:

[1150] The terminal transmits the collected video data to the server.

[1151] Step 3:

[1152] The server analyzes the video data and applies image recognition algorithms to detect emergency vehicles and rapidly approaching vehicles from behind.

[1153] Step 4:

[1154] If the server detects an abnormality based on the analysis results, it generates a warning message. In the case of an emergency vehicle, it generates a message saying, "An emergency vehicle is approaching. Please give way."

[1155] Step 5:

[1156] The server generates a warning message and sends it to the terminal.

[1157] Step 6:

[1158] The device will notify the driver of any warning messages it receives via voice.

[1159] Sharing and notifying dangerous driving vehicle information

[1160] Program processing

[1161] Step 1:

[1162] The device uses an onboard camera to collect video data of the surrounding area.

[1163] Step 2:

[1164] The video data collected by the device is uploaded to a cloud server.

[1165] Step 3:

[1166] The cloud server analyzes the uploaded video data and detects dangerous driving vehicles using driving behavior pattern recognition and anomaly detection algorithms.

[1167] Step 4:

[1168] Based on the detection result, the cloud server generates a warning message if there is a reckless driving vehicle, for example, a message saying "There is a reckless driving vehicle nearby. Please be careful."

[1169] Step 5:

[1170] The cloud server generates a warning message and sends it to the terminal.

[1171] Step 6:

[1172] The device will notify the driver of any warning messages it receives via voice.

[1173] Specific examples

[1174] Specific examples of chat functions

[1175] Step 1:

[1176] The device uses the onboard camera to collect the driver's facial information (facial expressions) and voice information.

[1177] Step 2:

[1178] The terminal transmits the collected face information and voice information to the server.

[1179] Step 3:

[1180] The server analyzes facial and audio information and determines the driver's level of fatigue as "high."

[1181] Step 4:

[1182] The server sends the fatigue level assessment result to the emotion engine, which then determines that the stress level is also "high."

[1183] Step 5:

[1184] The server generates "relaxation topics" based on "high" fatigue and "high" stress levels.

[1185] Step 6:

[1186] The server sends the generated topics to the terminal.

[1187] Step 7:

[1188] The device will notify the driver of the topic it receives via voice and suggest, "Why don't you think about relaxing for a bit?"

[1189] Examples of alert functions

[1190] Step 1:

[1191] The device uses an onboard camera to collect information about surrounding traffic conditions.

[1192] Step 2:

[1193] The terminal transmits the collected video data to the server.

[1194] Step 3:

[1195] The server analyzes the video data and detects emergency vehicles.

[1196] Step 4:

[1197] The server generates a warning message saying "Emergency vehicle approaching. Please give way."

[1198] Step 5:

[1199] The server generates a message and sends it to the terminal.

[1200] Step 6:

[1201] The device will then notify the driver via voice of the message received, urging them to give way to the emergency vehicle.

[1202] Specific examples of dangerous driving vehicle information

[1203] Step 1:

[1204] The device uses an onboard camera to collect video data of the surrounding area.

[1205] Step 2:

[1206] The video data collected by the device is uploaded to a cloud server.

[1207] Step 3:

[1208] The cloud server analyzes the uploaded data and detects dangerously driving vehicles.

[1209] Step 4:

[1210] The cloud server generates a warning message saying, "There is a dangerously driven vehicle nearby. Please be careful."

[1211] Step 5:

[1212] The cloud server sends the generated message to the terminal.

[1213] Step 6:

[1214] The device will notify the driver of the received message via voice, urging them to be aware of their surroundings.

[1215] Example 2

[1216] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1217] In recent years, road traffic has become increasingly complex, creating a need for systems that can monitor drivers' attention and emotional state in real time to support safe driving. However, current systems only analyze the driver's condition obtained from facial and voice information, and lack the functionality to generate topics to provide appropriate support. Furthermore, they lack the functionality to accurately analyze surrounding traffic conditions, detect reckless driving vehicles early, and issue warnings. Therefore, there is a need for systems that can effectively support safe driving.

[1218] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for analyzing the driver's facial information and voice information to determine the driver's condition, a means for generating a conversation topic based on the driver's condition and emotional state, and a means for notifying the driver of the topic by voice and advancing the conversation. This makes it possible to monitor the driver's attention and emotional state in real time and provide appropriate assistance. Furthermore, by including a means for analyzing video data collected from an on-board camera and accurately analyzing the surrounding traffic conditions, and a means for uploading the collected video data to the cloud and analyzing it on the cloud to detect reckless driving vehicles, it becomes possible to promptly alert and warn the driver.

[1219] "Driver" means a person who drives a vehicle.

[1220] "Facial information" refers to data related to the driver's facial expressions, gaze, etc.

[1221] "Voice information" refers to data related to the driver's voice, such as what they say and their tone of voice.

[1222] "Terminal" refers to input devices such as cameras and microphones installed in the vehicle.

[1223] "Server" refers to the central system that analyzes the collected data and generates appropriate responses based on the results.

[1224] "Emotion engine" refers to a system that determines the driver's emotional state based on facial and voice information.

[1225] "Video data" refers to video information about surrounding traffic conditions collected by an in-vehicle camera.

[1226] "Condition" refers to the driver's level of fatigue and attention.

[1227] "Conversational topics" refer to topics that are generated based on the driver's condition and emotional state.

[1228] "Traffic conditions" refers to the traffic environment, including other vehicles and surrounding road conditions.

[1229] "Caution" refers to a warning message to encourage drivers to drive safely.

[1230] "Dangerous driving vehicle" refers to a vehicle that is driving abnormally in the surrounding traffic conditions and may pose a safety hazard.

[1231] "Cloud" refers to a general term for computer resources provided through multiple servers and storage on the Internet.

[1232] "Cloud analytics" refers to data analysis processing performed on the cloud.

[1233] This invention is a system that uses facial and audio information of the driver to maintain attention and support safe driving. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, more appropriate support is provided. Specific embodiments are described in detail below.

[1234] System configuration and program processing

[1235] 1. Collection of facial and voice information (terminal)

[1236] The device uses an onboard camera and microphone to collect the driver's facial and audio information. Facial information includes facial expressions and gaze data, while audio information includes what is being said and tone of voice. This collection process uses a high-resolution camera and a highly sensitive microphone.

[1237] 2. Condition Analysis (Server)

[1238] The server analyzes the collected facial and audio information to determine the driver's condition. Specifically, it measures the driver's level of fatigue using facial expression analysis and determines the driver's level of attention through voice tone analysis. This analysis uses facial recognition algorithms (e.g., OpenCV and DeepFace) and voice analysis algorithms (e.g., Google Cloud Speech-to-Text API). The resulting numerical output represents the driver's level of fatigue and attention.

[1239] 3. Emotion Recognition (Emotion Engine)

[1240] The emotion engine further analyzes the collected facial and audio information to determine the driver's emotional state, which can include anger, sadness, joy, surprise, fear, etc. This analysis is performed using a deep learning model (e.g., an emotion recognition model using PyTorch or TensorFlow).

[1241] 4. Providing conversation topics (server)

[1242] The server generates appropriate topics based on the determined condition and emotional state. For example, if the driver is highly fatigued and stress levels are rising, it will provide "topics related to relaxation." If the driver is alert and emotionally stable, it will generate "talk about topics of interest." This topic generation uses a natural language generation model (e.g., a generative AI model).

[1243] 5. Notifications and conversation initiation (device)

[1244] The device then notifies the driver of the topic received from the server by voice. The notification is in the form of a message such as, "Would you like to think about relaxing for a moment?" When the driver responds, the system will proceed with the conversation based on that response. This also uses a high-definition speaker to notify the driver by voice.

[1245] Analysis of surrounding traffic conditions and alerts

[1246] 1. Video data collection (terminal)

[1247] The device uses an onboard camera to collect surrounding traffic conditions, including other vehicles and road conditions. The camera has a wide-angle lens, allowing it to collect high-resolution images even at night.

[1248] 2. Traffic situation analysis (server)

[1249] The server analyzes the video data and applies image recognition algorithms to detect emergency vehicles and approaching vehicles from behind, using techniques such as YOLOv5 and Mask R-CNN to identify specific vehicles and patterns.

[1250] 3. Warning (terminal)

[1251] Based on the analysis results, the device will give the driver appropriate warnings. For example, if an emergency vehicle is approaching, the device will notify the driver, "An emergency vehicle is approaching. Please give way."

[1252] Reckless driving vehicle detection and notification

[1253] 1. Upload data to the cloud (device)

[1254] The device uploads the video data collected from the in-vehicle camera to a cloud server, which is equipped with advanced analytical functions, enabling efficient data processing.

[1255] 2. Cloud analysis (server)

[1256] The cloud server analyzes the uploaded data and detects dangerous driving vehicles using driving behavior pattern recognition and anomaly detection algorithms (such as AWS Rekognition and Google Cloud Vision API).

[1257] 3. Warning notification (terminal)

[1258] If cloud analysis detects a vehicle driving recklessly, the device will warn the driver. For example, it will notify the driver by saying, "There is a vehicle driving recklessly nearby. Please be careful." The driver can then remain vigilant and drive safely based on the warning.

[1259] Specific examples

[1260] Specific examples of chat functions

[1261] User: The driver is driving for a long time.

[1262] Device: Collects facial and audio information from the driver using the vehicle's camera and microphone.

[1263] Terminal: Sends collected information to the server.

[1264] Server: Analyzes the information and determines whether the driver is tired.

[1265] Emotion engine: Determines fatigue and high stress levels at the same time.

[1266] Server: Generate "relaxation topics."

[1267] Device: Notifies the driver of the received topic via voice and suggests, "Why don't you think about relaxing for a moment?"

[1268] User: The driver responds, "Well, let's talk about how to relax."

[1269] Example prompt: "When drivers are on the road for long periods of time, come up with ways to talk about relaxation to help them stay alert."

[1270] Examples of alert functions

[1271] Device: Uses an in-car camera to collect information about surrounding traffic conditions.

[1272] Terminal: Sends collected video data to the server.

[1273] Server: Analyzes video data and detects emergency vehicles.

[1274] Server: Generate a warning message saying "Emergency vehicle approaching. Please give way."

[1275] Device: The device will notify the driver of the received message via voice and prompt them to give way to the emergency vehicle.

[1276] User: Drivers should check behind them and give way safely.

[1277] Example prompt: "If an emergency vehicle is approaching, think of a way to quickly notify the driver."

[1278] Specific examples of dangerous driving vehicle information

[1279] Terminal: Collects video data of the surrounding area using an onboard camera.

[1280] Terminal: Uploads collected video data to a cloud server.

[1281] Server (Cloud): Analyzes uploaded data and detects dangerous driving vehicles.

[1282] Server (Cloud): Generates a warning message saying, "There is a dangerously driven vehicle nearby. Please be careful."

[1283] Device: The device will notify the driver of received messages via voice and urge them to be aware of their surroundings.

[1284] User: Drivers should be aware of their surroundings and drive safely.

[1285] Example prompt: "If there is a dangerously driving vehicle nearby, suggest ways to warn the driver and encourage safe driving."

[1286] The above is a specific embodiment of the present invention. By using this system, the driver can always receive appropriate support according to the driving situation, and the safe driving environment is strengthened.

[1287] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1288] Program processing flow and specific operations

[1289] Driver health analysis and support

[1290] Step 1:

[1291] The device uses the onboard camera and microphone to collect the driver's facial and audio information, including facial expression and gaze data, as well as voice tone and content.

[1292] Input: Video and audio data from the onboard camera and microphone

[1293] Output: Collected facial and audio information

[1294] Step 2:

[1295] The terminal transmits the collected face information and voice information to the server.

[1296] Input: Face and voice information

[1297] Output: Data sent to the server

[1298] Step 3:

[1299] The server analyzes the collected facial and voice information to determine the driver's condition. Specifically, it measures the driver's fatigue level using facial expression analysis and analyzes the tone of the voice.

[1300] Input: Face and voice information

[1301] Output: Numerical data on driver fatigue and alertness

[1302] Step 4:

[1303] The emotion engine performs additional analysis to determine the driver's emotional state, specifically by analyzing facial and audio information using deep learning models (e.g., PyTorch) to identify emotions.

[1304] Input: Face and voice information

[1305] Output: Driver's emotional state (e.g., anger, sadness, joy, etc.)

[1306] Step 5:

[1307] The server generates appropriate conversation topics based on your condition and emotional state, using generative AI models to generate topics of relaxation, interest, etc.

[1308] Input: Driver condition and emotional state

[1309] Output: Generated conversation topics

[1310] Step 6:

[1311] The device will then notify the driver of the generated topic by voice and start a conversation. The notification content may be something like, "Why don't you think about relaxing for a bit?"

[1312] Input: Generated conversation topics

[1313] Output: Audio notification to the driver

[1314] Analysis of surrounding traffic conditions and alerts

[1315] Step 7:

[1316] The device uses an onboard camera to collect information about the surrounding traffic conditions, including other vehicles and road conditions.

[1317] Input: Surrounding image data from a camera

[1318] Output: Collected traffic video data

[1319] Step 8:

[1320] The terminal transmits the collected video data to the server.

[1321] Input: Surrounding video data

[1322] Output: Video data sent to the server

[1323] Step 9:

[1324] The server analyzes the video data and determines the traffic situation, using an image recognition algorithm (e.g., YOLOv5) to detect emergency vehicles and vehicles approaching from behind.

[1325] Input: Surrounding traffic video data

[1326] Output: Parsed traffic data

[1327] Step 10:

[1328] Based on the analysis results, the server generates a warning message for the driver.

[1329] Input: Parsed traffic data

[1330] Output: Attention message

[1331] Step 11:

[1332] The device will then notify the driver of the warning message it receives by voice, urging them to take appropriate action. For example, it may say, "An emergency vehicle is approaching. Please give way."

[1333] Input: Attention message

[1334] Output: Audio notification to the driver

[1335] Reckless driving vehicle detection and notification

[1336] Step 12:

[1337] The device uploads the video data collected through the onboard camera to a cloud server.

[1338] Input: Video data

[1339] Output: Uploaded video data

[1340] Step 13:

[1341] The cloud server analyzes the uploaded data and detects reckless driving vehicles using advanced pattern recognition and anomaly detection algorithms (e.g., AWS Rekognition).

[1342] Input: Uploaded video data

[1343] Output: Detected dangerous driving vehicles

[1344] Step 14:

[1345] The cloud server generates a warning message to the driver based on the analysis results.

[1346] Input: Detected dangerous driving vehicles

[1347] Output: Warning message

[1348] Step 15:

[1349] The device will then generate a warning message and notify the driver by voice, urging them to be vigilant about their surroundings. For example, it will say, "There is a vehicle driving recklessly nearby. Please be careful."

[1350] Input: warning message

[1351] Output: Audio notification to the driver

[1352] (Application example 2)

[1353] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1354] In autonomous vehicles, the system certainly takes over most of the driving, but it is impossible to completely eliminate the dangers of driver distraction or fatigue. Furthermore, the driver's emotional state while driving also has a significant impact on driving safety, so appropriate reminders and support are essential. However, conventional systems have had difficulty accurately recognizing the driver's emotional state and providing appropriate real-time support. This has led to problems in effectively ensuring driver safety.

[1355] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting face information and voice information of the driver, means for analyzing the face information and voice information of the driver to determine the condition, means for providing conversation topics according to the condition, means for recognizing the emotional state of the driver, and means for providing voice notification in real time based on the emotional state. This makes it possible to monitor the driver's attention lapse, fatigue, and emotional state in real time and provide appropriate warnings and support.

[1356] "Driver" means the passenger who operates and monitors an automated driving vehicle.

[1357] "Facial information" refers to image data of the driver's face acquired from a camera, and includes facial expression and gaze data.

[1358] "Voice information" refers to data about the driver's speech and tone of voice obtained from a microphone.

[1359] "Condition" refers to the driver's level of attention and fatigue, which is analyzed based on facial and voice information.

[1360] "Emotional state" refers to the driver's emotions (e.g., anger, sadness, joy, etc.) analyzed from facial and voice information.

[1361] "Real-time" means that data collection, analysis, and notification occur immediately.

[1362] The "means for providing conversation topics" is a system that generates appropriate conversation themes and topics based on the driver's condition and emotional state.

[1363] "Means for providing voice notification" refers to a system that uses voice output to alert the driver and provide conversation based on the analysis results.

[1364] "Surrounding traffic conditions" refers to data about other vehicles and road conditions collected through in-vehicle cameras.

[1365] "Warning" refers to warning drivers about road conditions or emergency situations based on analyzed information.

[1366] The "cloud" is a remote server environment that analyzes and stores data via the Internet.

[1367] "Dangerous driving vehicles" refer to nearby vehicles whose abnormal driving behavior has been detected based on the analysis results.

[1368] The "means of issuing a warning" is a system that notifies drivers of the presence of a vehicle engaged in dangerous driving based on the results of cloud analysis.

[1369] This invention is a system that monitors the driver's attention and emotional state in real time in an autonomous vehicle and provides appropriate support and warnings. The system uses an on-board camera and microphone to collect the driver's facial and voice information, and analyzes this data on a server.

[1370] System Configuration

[1371] 1. Data Collection

[1372] Hardware: In-car camera (any USB camera), microphone

[1373] Software: OpenCV(cv2), PyAudio

[1374] Processing: Collecting facial and audio information from the driver. For example, an onboard camera records the driver's facial expressions and gaze, and a microphone records speech and tone of voice.

[1375] 2. Data Analysis

[1376] Hardware: High-performance server

[1377] Software: EmotionEngine (emotion recognition engine), custom voice analysis algorithm

[1378] Processing: The server analyzes the collected facial and audio information to determine the driver's condition and emotional state. Specifically, the server measures the driver's level of fatigue from facial expressions and analyzes their level of attention from the tone of their voice.

[1379] 3. Determining Conditions and Emotional States

[1380] Example: If a driver has been driving for a long time, the server will determine that the driver is tired based on facial information and that the driver's attention level is low based on the tone of voice. In this case, the server will provide the driver with appropriate conversation topics.

[1381] 4. Real-time support and notifications

[1382] Hardware: In-car audio system

[1383] Software: pyttsx3 (speech synthesis library)

[1384] Processing: Based on the results of the assessment, the server generates appropriate conversation topics and warning messages. For example, if the server determines that the driver is tired, it will notify the driver by voice, "Would you like to rest your eyes for a while?"

[1385] 5. Traffic situation analysis

[1386] Hardware: In-car camera

[1387] Software: Image recognition algorithms

[1388] Processing: The onboard camera collects information on surrounding traffic conditions in real time and analyzes emergency vehicles and vehicles approaching rapidly from behind, etc. Based on this, necessary warnings are given to the driver via voice.

[1389] 6. Cloud Analysis

[1390] Hardware: Cloud Server

[1391] Software: Advanced analysis algorithms

[1392] Processing: Collected video data is uploaded to the cloud and analyzed there. The analysis results are fed back to the vehicle in real time, and the presence of reckless driving vehicles is notified.

[1393] Examples and prompts

[1394] Specific examples

[1395] 1. Example of high fatigue: The driver is determined to be tired and a voice notification is given saying, "Would you like to talk about relaxing a bit?"

[1396] 2. Example of inattention: The driver is determined to be inattentive and an appropriate voice warning is given.

[1397] 3. Example of a traffic situation-based warning: If an emergency vehicle is approaching from behind, a voice notification will say, "An emergency vehicle is approaching. Please give way."

[1398] Prompt Sentence Examples

[1399] "The facial image and voice data indicate that the driver is tired. The driver's fatigue level is high and their attention level is low. In this case, please generate conversation topic suggestions that are effective in helping the driver relax."

[1400] conclusion

[1401] This system can improve driving safety by monitoring the driver's attention and emotional state in real time and providing appropriate support and warnings.

[1402] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1403] Step 1:

[1404] The device uses an onboard camera and microphone to collect facial and audio information from the driver.

[1405] Input: Driver's face image, voice data

[1406] How it works: The onboard camera captures the driver's facial expressions and gaze information, and the microphone records the driver's speech and tone of voice. These data are acquired in real time.

[1407] Output: Collected facial image data and voice data

[1408] Step 2:

[1409] The terminal transmits the collected face information and voice information to the server.

[1410] Input: collected facial image data, voice data

[1411] Specific operation: The terminal uploads the data stored in the local buffer to the server via the network.

[1412] Output: Facial image data and audio data sent to the server

[1413] Step 3:

[1414] The server analyzes facial and audio information to determine the driver's condition.

[1415] Input: Facial image data, audio data

[1416] How it works: The EmotionEngine on the server analyzes facial image data and infers the driver's fatigue level and emotional state from their facial expressions. A custom voice analysis algorithm is used on the voice data to determine the driver's level of attention through tone and language analysis.

[1417] Output: Driver condition data (fatigue level, attention state, emotional state)

[1418] Step 4:

[1419] The server generates appropriate conversation topics and reminder messages based on the determined condition and emotional state.

[1420] Input: Driver condition data

[1421] Specific operation: Based on the results of the assessment, the server uses an AI model to generate optimal conversation topics and attention-grabbing content for the driver. For example, if the driver's level of fatigue is high, it will suggest topics related to relaxation, and if the driver's attention level is low, it will generate a warning message.

[1422] Output: Conversation topics, warning messages

[1423] Step 5:

[1424] The device notifies the driver by voice of the conversation topic and warning messages received from the server.

[1425] Input: conversation topics, warning messages

[1426] Specific operation: The device uses a speech synthesis library such as pyttsx3 to convert text data into speech data and notify the driver through the car speaker.

[1427] Output: Voice message notified to the driver

[1428] Step 6:

[1429] The terminal collects video data from the vehicle-mounted camera and transmits it to a server.

[1430] Input: Surrounding traffic video data

[1431] Specific operation: The onboard camera captures the surrounding traffic conditions, collects the video data in real time, and sends it to a server via the network.

[1432] Output: Video data sent to the server

[1433] Step 7:

[1434] The server analyzes the video data to detect emergency vehicles and vehicles driving recklessly.

[1435] Input: Video data

[1436] How it works: Image recognition algorithms on the server process video data to detect specific traffic patterns and unusual driving behavior. The algorithms identify emergency vehicles and rapidly approaching vehicles.

[1437] Output: Analysis of surrounding traffic conditions (location information for emergency vehicles and dangerously driving vehicles)

[1438] Step 8:

[1439] The server generates a warning message for the driver based on the analysis results and sends it to the terminal.

[1440] Input: Traffic situation analysis results

[1441] Specific operation: The server generates a warning message for the driver based on the detected emergency vehicle or dangerous driving vehicle information and sends it to the terminal.

[1442] Output: Warning message

[1443] Step 9:

[1444] The terminal notifies the driver of the warning message received from the server by voice.

[1445] Input: warning message

[1446] Specific operation: The device uses a speech synthesis library to convert the warning message into voice data and notify the driver through the car speaker.

[1447] Output: Audio warning message notified to the driver

[1448] This allows drivers to always receive appropriate support in real time that is tailored to the driving situation, enhancing the safe driving environment.

[1449] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1450] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1451] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1452] [Third embodiment]

[1453] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1454] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1455] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1456] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1457] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1458] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1459] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1460] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1461] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1462] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1463] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1464] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1465] This invention is a system that uses facial and audio information from the driver to maintain attention and support safe driving. The system works by collecting and analyzing data via an on-board camera and microphone.

[1466] System Configuration

[1467] 1. Collection of facial and voice information (terminal)

[1468] The device uses the onboard camera and microphone to collect facial and audio information from the driver. Facial information includes facial expressions and gaze data, while audio information includes what is being said and tone of voice.

[1469] 2. Condition Analysis (Server)

[1470] The server analyzes the collected facial and voice information to determine the driver's condition. Specifically, it measures the driver's level of fatigue using facial expression analysis and determines the driver's level of attention through voice tone analysis. As a result, the driver's level of fatigue and attention are output as numerical values.

[1471] 3. Providing conversation topics (server)

[1472] The server generates appropriate topics based on the determined condition. For example, if the driver is highly fatigued, it might suggest taking a break, or if the driver is alert, it might suggest talking about an interesting topic.

[1473] 4. Notifications and conversation initiation (device)

[1474] The device will then provide a voice notification to the driver about the topic received from the server, and the conversation will progress naturally based on the driver's responses.

[1475] Analysis of surrounding traffic conditions and alerts

[1476] 1. Video data collection (terminal)

[1477] The device collects video data from the in-vehicle camera, including information on surrounding vehicles and road conditions.

[1478] 2. Traffic situation analysis (server)

[1479] The server analyzes the video data to detect emergency vehicles and vehicles approaching from behind, using image recognition technology to identify specific vehicles and patterns.

[1480] 3. Warning (terminal)

[1481] Based on the analysis results, the device will give the driver appropriate warnings. For example, if an emergency vehicle is approaching, the device will notify the driver, "An emergency vehicle is approaching. Please give way."

[1482] Reckless driving vehicle detection and notification

[1483] 1. Upload data to the cloud (device)

[1484] The video data collected by the device is uploaded to the cloud, where advanced analytical functions enable efficient data processing.

[1485] 2. Cloud analysis (server)

[1486] A cloud server analyzes the uploaded data and detects dangerous driving vehicles using algorithms for pattern recognition and anomaly detection of driving behavior.

[1487] 3. Warning notification (terminal)

[1488] If cloud analysis detects a vehicle driving recklessly, the device will warn the driver, for example, by sending a message saying, "There is a vehicle driving recklessly nearby. Please be careful."

[1489] Specific examples

[1490] Example 1: When a driver becomes tired while driving

[1491] User: The user (driver) is driving for a long time.

[1492] Device: The onboard camera collects the driver's facial expression data, and the microphone collects audio data.

[1493] Server: The server analyzes the collected data and determines the driver's fatigue level as "high."

[1494] Server: The server generates the topic "Would you like to take a break?"

[1495] Device: The device will prompt the driver with a voice message asking, "Would you like to take a break?"

[1496] User: The driver responds, "Yeah, let's take a break," and takes a break.

[1497] Example 2: When an emergency vehicle is approaching

[1498] Terminal: The onboard camera collects video data of the surrounding area.

[1499] Server: The server analyzes the video data and detects emergency vehicles.

[1500] Server: The server generates a warning message saying "Emergency vehicle approaching. Please give way."

[1501] Device: The device will notify the driver via voice message, "An emergency vehicle is approaching. Please give way."

[1502] User: Drivers should check behind them and give way safely.

[1503] Example 3: Reckless driving vehicles nearby

[1504] Terminal: The onboard camera collects video data of the surrounding area.

[1505] Device: The device uploads the video data to the cloud.

[1506] Server (Cloud): The server analyzes the uploaded data and detects dangerous driving vehicles.

[1507] Server (Cloud): The server generates a warning message saying, "There is a dangerously driven vehicle nearby. Please be careful."

[1508] Device: The device will notify the driver by voice, "There is a vehicle driving recklessly nearby. Please be careful."

[1509] User: Drivers should be aware of their surroundings and drive safely.

[1510] This ensures that drivers always receive appropriate support tailored to the driving situation, enhancing the safe driving environment.

[1511] The processing flow will be explained below.

[1512] Chat function

[1513] Program processing

[1514] Step 1:

[1515] The device uses the vehicle's camera and microphone to collect facial and audio information from the driver, including their facial expressions, gaze, and tone of voice.

[1516] Step 2:

[1517] The terminal transmits the collected face information and voice information to the server.

[1518] Step 3:

[1519] The server analyzes the received facial and voice information to determine the driver's condition. Specifically, it measures the driver's fatigue level using facial expression analysis and determines the driver's level of attention through voice tone analysis.

[1520] Step 4:

[1521] Based on the results of the assessment, the server generates conversation topics that correspond to the driver's condition. For example, if the driver is highly fatigued, it will suggest taking a break, and if the driver is alert, it will suggest talking about a topic of interest.

[1522] Step 5:

[1523] The server sends the generated conversation topic to the terminal.

[1524] Step 6:

[1525] The device will notify the driver of the topic of conversation received by voice and begin the conversation.

[1526] Alert function

[1527] Program processing

[1528] Step 1:

[1529] The device uses an onboard camera to collect video data about the surrounding traffic, including other vehicles and road conditions.

[1530] Step 2:

[1531] The terminal transmits the collected video data to the server.

[1532] Step 3:

[1533] The server analyzes the video data and applies image recognition algorithms to detect emergency vehicles and rapidly approaching vehicles from behind.

[1534] Step 4:

[1535] If the server detects an abnormality based on the analysis results, it generates a warning message. For example, in the case of an emergency vehicle, it generates a message saying, "An emergency vehicle is approaching. Please give way."

[1536] Step 5:

[1537] The server generates a warning message and sends it to the terminal.

[1538] Step 6:

[1539] The device will notify the driver of any warning messages it receives via voice.

[1540] Sharing and notifying dangerous driving vehicle information

[1541] Program processing

[1542] Step 1:

[1543] The device uses an onboard camera to collect video data of the surrounding area.

[1544] Step 2:

[1545] The video data collected by the device is uploaded to a cloud server.

[1546] Step 3:

[1547] The cloud server analyzes the uploaded video data to detect dangerous driving vehicles, using algorithms that identify specific driving behaviors.

[1548] Step 4:

[1549] Based on the detection result, the cloud server generates a warning message if there is a reckless driving vehicle, for example, a message saying "There is a reckless driving vehicle nearby. Please be careful."

[1550] Step 5:

[1551] The cloud server generates a warning message and sends it to the terminal.

[1552] Step 6:

[1553] The device will notify the driver of any warning messages it receives via voice.

[1554] Specific examples

[1555] Specific examples of chat functions

[1556] Step 1:

[1557] The device uses the onboard camera and microphone to collect the driver's facial and voice information.

[1558] Step 2:

[1559] The device sends the collected information to the server.

[1560] Step 3:

[1561] The server analyzes the information and determines that the driver is tired.

[1562] Step 4:

[1563] The server generates a conversation topic suggesting a break, such as "Would you like to take a short break?"

[1564] Step 5:

[1565] The server sends the generated topics to the terminal.

[1566] Step 6:

[1567] The device will notify the driver of the topic it receives via voice and suggest, "Would you like to take a short break?"

[1568] Examples of alert functions

[1569] Step 1:

[1570] The device uses an onboard camera to collect information about surrounding traffic conditions.

[1571] Step 2:

[1572] The terminal transmits the collected video data to the server.

[1573] Step 3:

[1574] The server analyzes the video data and detects emergency vehicles.

[1575] Step 4:

[1576] The server generates a warning message saying "Emergency vehicle approaching. Please give way."

[1577] Step 5:

[1578] The server generates a message and sends it to the terminal.

[1579] Step 6:

[1580] The device will then notify the driver via voice of the message received, urging them to give way to the emergency vehicle.

[1581] Specific examples of dangerous driving vehicle information

[1582] Step 1:

[1583] The device uses an onboard camera to collect video data of the surrounding area.

[1584] Step 2:

[1585] The video data collected by the device is uploaded to a cloud server.

[1586] Step 3:

[1587] The cloud server analyzes the video data and detects vehicles driving recklessly.

[1588] Step 4:

[1589] The cloud server generates a warning message saying, "There is a dangerously driven vehicle nearby. Please be careful."

[1590] Step 5:

[1591] The cloud server sends the generated message to the terminal.

[1592] Step 6:

[1593] The device will notify the driver of the received message via voice, urging them to be aware of their surroundings.

[1594] The above is a description of the specific processing steps and operations in the system of the present invention.

[1595] Example 1

[1596] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1597] In today's modern traffic society, there are many accidents that are believed to be caused by driver inattention or fatigue. Furthermore, there is a need for appropriate responses to surrounding traffic conditions and the presence of dangerously driven vehicles. However, conventional systems have had the problem of being unable to accurately grasp the driver's condition and traffic situation in real time and provide appropriate support and warnings.

[1598] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1599] In this invention, the server includes means for collecting face information and voice information of the driver, means for analyzing the face information and voice information of the driver to determine the driver's condition, means for providing a topic of conversation according to the condition, means for notifying the driver of the topic of conversation, means for analyzing the driver's responses and advancing the conversation, means for collecting video data from an in-vehicle camera, means for analyzing the video data to analyze surrounding traffic conditions, means for issuing a warning to the driver based on the traffic conditions, means for uploading the collected video data to a cloud, means for analyzing the video data on the cloud and detecting reckless driving vehicles, and means for issuing a warning to the driver based on the cloud analysis results. This makes it possible to detect a driver's dwindling attention or fatigue in real time, and maintain attention by providing appropriate topics, while also issuing appropriate warnings regarding surrounding traffic conditions and detecting and notifying reckless driving vehicles.

[1600] "Driver's facial information" is data including the driver's facial features such as facial expression and gaze.

[1601] "Voice information" is data that includes voice characteristics such as the driver's tone of voice and what is being said.

[1602] The "means for collecting face information and voice information" refers to a device that acquires the driver's face information and voice information using an in-vehicle camera, microphone, etc.

[1603] The "means for determining the condition" is a device or algorithm that analyzes the collected facial and audio information to determine the driver's level of fatigue and attention.

[1604] A "means for providing conversation topics" is a device or algorithm that generates appropriate conversation topics based on the driver's condition.

[1605] "Means for notifying the driver of the topic of conversation" refers to a device that audibly or visually conveys the generated topic of conversation to the driver through a receiver.

[1606] The "means for analyzing driver responses and advancing the conversation" refers to a device or algorithm that collects responses from the driver, analyzes their content, generates new conversation topics, and advances the conversation.

[1607] The "means for collecting video data from an in-vehicle camera" is a device that acquires video data of the surroundings using a camera mounted on a vehicle.

[1608] The "means for analyzing the surrounding traffic conditions by analyzing the video data" refers to a device or algorithm that analyzes the acquired video data and identifies the traffic conditions of surrounding vehicles, emergency vehicles, etc.

[1609] The "means for issuing a warning to the driver based on the traffic conditions" is a device that issues an appropriate warning to the driver based on the analysis results.

[1610] The "means for uploading video data to the cloud" is a device that transfers collected video data to a cloud environment via the Internet.

[1611] The "means for analyzing video data on the cloud and detecting reckless driving vehicles" refers to a device or algorithm that analyzes video data in a cloud environment and detects signs of reckless driving.

[1612] The "means for issuing a warning to the driver based on the results of cloud analysis" is a device that issues a warning to the driver for safe driving based on the results of cloud analysis.

[1613] This invention is a system that uses facial and audio information of a driver to maintain attention and support safe driving. This system operates by collecting and analyzing data via an on-board camera and microphone. Specific embodiments of this system are described below.

[1614] System Configuration

[1615] Collecting facial and audio information

[1616] The device uses an onboard camera (e.g., a network camera) and microphone (e.g., a high-sensitivity microphone) to collect the driver's facial and audio information. Facial information includes the driver's facial expressions and gaze data. Audio information includes what the driver says and the tone of their voice. For example, the camera captures the driver's eye movements and facial expressions, and the microphone records the pitch and speed of their voice.

[1617] Condition Analysis

[1618] The server analyzes the collected facial and audio information to determine the driver's condition. Specifically, it measures the driver's level of fatigue using facial expression analysis (e.g., image processing algorithms) and determines the driver's level of attention through voice tone analysis (e.g., voice recognition software). As a result, the driver's level of fatigue and attention state are output as numerical values. For example, frequent eye closure and a restless voice are detected as signs of fatigue.

[1619] Providing topics for conversation

[1620] The server generates appropriate topics based on the determined condition. For example, if the driver is highly fatigued, it will "suggest a break," and if the driver is alert, it will "talk about a topic of interest." This process uses a generative AI model, which generates, for example, "driver's interests" based on the prompt sentence.

[1621] Notifications and conversation starters

[1622] The device then notifies the driver of the topic received from the server by voice. Specifically, it plays back a message generated using speech synthesis software (e.g., text-to-speech software). The conversation progresses naturally based on the driver's response. For example, the device may say, "Do you want to take a short break?" and the driver may respond, "Yes, let's take a short break."

[1623] Analysis of surrounding traffic conditions and alerts

[1624] The device collects video data from the onboard camera. This camera captures surrounding vehicles and road conditions in real time. The video data is sent to a server, which analyzes it to detect approaching emergency vehicles and rapidly approaching vehicles from behind. Image recognition technology (e.g., deep learning models) is used for the analysis. Based on the analysis results, the device issues appropriate warnings to the driver. For example, it may notify the driver, "An emergency vehicle is approaching. Please give way."

[1625] Reckless driving vehicle detection and notification

[1626] The video data collected by the device is uploaded to the cloud. The cloud environment is equipped with advanced analytical functions; for example, a server on the cloud analyzes the uploaded data to detect vehicles that are driving recklessly. The analysis uses driving behavior pattern recognition and anomaly detection algorithms. If a vehicle is detected as a result of cloud analysis, the device will notify the driver, "There is a vehicle that is driving recklessly nearby. Please be careful."

[1627] Specific examples

[1628] Example 1: When a driver becomes tired while driving

[1629] User: The driver is driving for a long time.

[1630] Device: The onboard camera collects the driver's facial expression data, and the microphone collects audio data.

[1631] Server: The server analyzes the collected data and determines the driver's fatigue level as "high."

[1632] Server: The server generates the message "Would you like to take a break?"

[1633] Device: Your device will say, "Would you like to take a break?"

[1634] User: The driver responds, "Yeah, let's take a break," and takes a break.

[1635] Example 2: When an emergency vehicle is approaching

[1636] Terminal: The onboard camera collects video data of the surrounding area.

[1637] Server: The server analyzes the video data and detects emergency vehicles.

[1638] Server: The server generates the message "Emergency vehicle approaching. Please give way."

[1639] Device: The device will announce "An emergency vehicle is approaching. Please give way."

[1640] User: Drivers should check behind them and give way safely.

[1641] Example 3: Reckless driving vehicles nearby

[1642] Device: The onboard camera collects images of the surrounding area.

[1643] Device: The device uploads the video data to the cloud.

[1644] Server (Cloud): The server analyzes the uploaded data and detects dangerous driving vehicles.

[1645] Server (Cloud): The server generates a message saying, "There is a dangerously driven vehicle nearby. Please be careful."

[1646] Device: The device will announce with a voice message, "There is a dangerously driven vehicle nearby. Please be careful."

[1647] User: The driver remains alert to their surroundings and continues driving safely.

[1648] The system is designed to provide drivers with appropriate support in real time while driving, thereby contributing to an enhanced safe driving environment.

[1649] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1650] Step 1:

[1651] (Data collection begins)

[1652] The device activates the on-board camera and microphone and starts collecting the driver's facial and audio information. The input is real-time data from the on-board camera and microphone, and the output is stored in the device as facial and audio information.

[1653] Step 2:

[1654] (Data import)

[1655] The device uses a camera to capture the driver's facial image and facial expressions, and a microphone to pick up audio. Specifically, the camera recognizes each part of the face (eyes, mouth, etc.), and the microphone records the pitch and speed of the audio. The input is video and audio data captured in real time, and the output is temporary storage of this data and format conversion for processing.

[1656] Step 3:

[1657] (Data transmission)

[1658] The device transmits the collected facial and audio information to the server in real time. The input is the temporarily stored facial and audio information, and the output is a data stream to the server.

[1659] Step 4:

[1660] (Facial expression analysis)

[1661] The server analyzes facial information, extracts facial expression data, and evaluates the driver's level of fatigue and attention. Specifically, the facial expression analysis algorithm detects the frequency of eye opening and closing and the upward and downward movement of the corners of the mouth. The input is the facial information sent to the server, and the output is a numerical evaluation of the driver's level of fatigue and attention.

[1662] Step 5:

[1663] (audio analysis)

[1664] The server analyzes the voice information and evaluates the driver's mental state based on the tone of the voice and the content of what is being said. Specifically, the voice recognition software analyzes changes in the tempo and volume of the voice. The input is the voice information sent to the server, and the output is the evaluation result of the driver's mental state.

[1665] Step 6:

[1666] (comprehensive evaluation)

[1667] The server combines the results of facial expression analysis and voice analysis to quantify the driver's condition. The input is the results of facial expression analysis and voice analysis, and the output is an integrated condition assessment.

[1668] Step 7:

[1669] (Topic generation)

[1670] The server generates appropriate conversation topics based on the condition evaluation results. Using a generative AI model, it generates, for example, "rest suggestions" or "topics of interest" based on the prompt. The input is the condition evaluation results, and the output is the generated conversation topic.

[1671] Step 8:

[1672] (Send message)

[1673] The server sends the generated message to the terminal. The input is the generated conversation topic, and the output is the message data to the terminal.

[1674] Step 9:

[1675] (Notification execution)

[1676] The device receives messages from the server and notifies the driver using a speech synthesis function. Specifically, the message is converted into speech using text-to-speech software. The input is message data from the server, and the output is a voice notification.

[1677] Step 10:

[1678] (Response analysis)

[1679] The device collects the driver's responses with a microphone and sends the audio to a server for analysis. The input is the driver's audio responses, and the output is a data stream to the server for analysis.

[1680] Step 11:

[1681] (Conversation progress)

[1682] The server generates a new topic based on the driver's response and continues the conversation through the device. It uses a generative AI model to generate the next appropriate topic. The input is the response analysis result, and the output is the next conversation topic.

[1683] Step 12:

[1684] (Video data collection)

[1685] The device collects video data from the in-vehicle camera in real time. The input is the surrounding video data, and the output is the video data stored in the device.

[1686] Step 13:

[1687] (Video data transmission)

[1688] The terminal sends the collected video data to the server. The input is the collected video data, and the output is a data stream to the server.

[1689] Step 14:

[1690] (traffic situation analysis)

[1691] The server analyzes the video data and identifies the approach of specific vehicles or emergency vehicles. Specifically, a deep learning model recognizes the lights and sirens of emergency vehicles. The input is the video data sent to the server, and the output is the analysis result of the traffic situation.

[1692] Step 15:

[1693] (Warning message generation)

[1694] The server generates the necessary warning messages based on the analysis results. The input is the traffic situation analysis results, and the output is the generated warning messages.

[1695] Step 16:

[1696] (Alert notice)

[1697] The terminal notifies the driver of the attention message received from the server by voice. The input is the message data from the server, and the output is the voice notification.

[1698] Step 17:

[1699] (Cloud Upload)

[1700] The device uploads the collected video data to the cloud. The input is the collected video data, and the output is a data stream to the cloud.

[1701] Step 18:

[1702] (Cloud analysis)

[1703] The server (cloud) analyzes the uploaded video data and detects dangerous driving vehicles. It uses driving behavior pattern recognition and anomaly detection algorithms. The input is the video data uploaded to the cloud, and the output is the detection results of dangerous driving vehicles.

[1704] Step 19:

[1705] (Send cloud analysis results)

[1706] The server (cloud) generates a warning message based on the analysis results and sends it to the terminal. The input is the cloud analysis result, and the output is the message data to the terminal.

[1707] Step 20:

[1708] (Warning notice)

[1709] The terminal notifies the driver of warning messages received from the server (cloud) by voice. The input is message data from the cloud, and the output is a voice notification.

[1710] (Application example 1)

[1711] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1712] Conventional driver assistance systems have difficulty providing appropriate rest and alerts based on the driver's level of fatigue or reduced attention span. They also lack real-time warnings and notifications to adapt to surrounding traffic conditions. Self-driving vehicles, in particular, are required to appropriately monitor the driver's condition and respond immediately when necessary. Therefore, there is a need for a new system that comprehensively alerts the driver and notifies them of surrounding traffic conditions, supporting safe driving.

[1713] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1714] In this invention, the server includes means for collecting face information and voice information of the driver, means for analyzing the face information and voice information of the driver to determine the condition, means for providing appropriate actions and warnings by synthetic voice based on the determined condition, and means for notifying the driver of traffic conditions by synthetic voice. This makes it possible to analyze the driver's condition in real time, prompt the driver to take a break or be careful at the appropriate time, and support safe driving by providing real-time notifications according to the surrounding traffic conditions.

[1715] "Driver's facial information" is a general term for visual data related to the driver's face, such as facial expression, gaze position, and whether the eyes are open or closed.

[1716] "Voice information" is a general term for data related to the driver's voice, such as what they say, their tone of voice, volume, and voice patterns.

[1717] "Condition" refers to the psychological and physiological state that affects driving, such as the driver's level of fatigue, attention, and physical condition.

[1718] "Synthetic speech" refers to artificial speech generated by converting text data into speech data.

[1719] "Traffic conditions" refers to the state of the external environment while driving, such as surrounding vehicles, road congestion, and the location of emergency vehicles.

[1720] "Warning" refers to providing information to encourage or warn drivers to take specific actions.

[1721] "Video data" refers to visual digital data collected via in-vehicle cameras, etc., and includes image information about surrounding conditions and objects.

[1722] "Cloud" refers to a group of data centers and servers accessible via the Internet, providing a platform for storing and analyzing data.

[1723] "Dangerous driving vehicles" refer to vehicles that exhibit unusual driving behavior and may pose a threat to the safety of those around them.

[1724] "Providing appropriate actions or warnings using synthetic voice" refers to the act of using synthetic voice to communicate messages generated based on the driver's state.

[1725] "HMD" is an abbreviation for head-mounted display, a display device worn on the user's head that displays visual information.

[1726] The present invention is a system that uses facial information and voice information of the driver to maintain the driver's attention and support safe driving. Specific embodiments for carrying out the present invention are described below.

[1727] Hardware Configuration

[1728] Collecting driver's facial and voice information

[1729] In-vehicle camera: An HD camera is used to collect the driver's facial information, which allows for the acquisition of facial expression and gaze data.

[1730] Microphone: The driver's voice information is collected using an in-vehicle microphone or a microphone built into the head-mounted display (HMD).

[1731] Notifications and Interactions

[1732] Head-mounted display (HMD): Functions as a display and provides visual notifications to the driver. It also provides audio notifications using a built-in speaker.

[1733] Software and Data Processing

[1734] OpenCV: A library for image processing and facial recognition. It analyzes video data from an in-vehicle camera and extracts facial landmarks.

[1735] dlib: A library for detecting facial landmarks. Used to analyze facial expressions and gaze.

[1736] speech_recognition: A library that performs speech recognition and analyzes the driver's voice information.

[1737] pyttsx3: A library that performs speech synthesis and notifies the driver via voice of appropriate warnings and actions.

[1738] System Operation

[1739] 1. Collecting facial and audio information

[1740] The system uses an onboard camera and microphone to collect the driver's facial and voice information in real time. Facial information includes facial expressions and eye gaze position, while voice information includes what the driver says and the tone of their voice.

[1741] 2. Driver status analysis

[1742] The server analyzes the collected facial and voice information to determine the driver's condition, specifically quantifying the driver's level of fatigue and attention from facial expressions, eye movements, and voice tone.

[1743] 3. Traffic situation notifications and alerts

[1744] The system analyzes surrounding video data collected by the in-vehicle camera to detect emergency vehicles and vehicles engaged in reckless driving. This analysis uses image recognition technology. Based on the analysis results, a visual notification is displayed on the HMD and a synthesized voice is used to warn the driver.

[1745] Specific examples

[1746] Example 1: If a driver has been driving for a long time, the server determines that the driver's fatigue level is "high" based on facial information. As a result, a message saying "You need to take a break" is displayed on the HMD and a voice notification is also given.

[1747] Example 2: If the server determines that the driver's attention is declining, it generates a message saying "Please concentrate," displays it on the HMD, and also notifies the driver by voice.

[1748] Example 3: Surrounding traffic conditions are collected from an in-car camera. If an emergency vehicle is approaching, the server analyzes the video data and displays a warning message on the HMD saying, "An emergency vehicle is approaching. Please give way.", and also provides an audio notification.

[1749] Example prompts for generative AI models

[1750] Design a system that analyzes the driver's facial and audio information in real time to quantify their level of fatigue and attention. This system should collect data using an on-board camera and microphone, and based on the analysis results, display a message on the HMD and provide an audio notification to encourage the driver to take appropriate action.

[1751] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1752] Step 1:

[1753] The device uses an onboard camera and microphone to collect the driver's facial and audio information in real time.

[1754] Input: Video data from the dashcam and audio data from the microphone.

[1755] Data processing: Face detection and landmark extraction are performed on video data, and voice recognition is performed on audio data.

[1756] Output: Driver's facial expression data, gaze data, and speech-to-text information.

[1757] Step 2:

[1758] The server analyzes the collected facial and audio information to determine the driver's condition.

[1759] Input: Facial expression data, gaze data, speech-to-text information.

[1760] Data calculation: Facial expression analysis, gaze analysis, and voice tone analysis are performed to quantify the driver's fatigue level and attention level.

[1761] Output: Driver condition (e.g. fatigue level or alertness).

[1762] Step 3:

[1763] The server generates a message in synthesized voice based on the determined condition, providing appropriate action or attention.

[1764] Input: Driver condition (fatigue level and attention level).

[1765] Data calculation: Generative AI models are used to generate warning messages that are appropriate for the driver's current situation.

[1766] Output: Synthesized voice message (e.g., "You need a break" or "Please stay focused").

[1767] Step 4:

[1768] The device displays a synthesized voice message on the HMD and notifies the driver by voice.

[1769] Input: Synthesized voice message.

[1770] Data calculation: A message is displayed on the HMD and a voice notification is given through the speaker.

[1771] Output: Visual and audio feedback to the driver.

[1772] Step 5:

[1773] The device collects video data from the onboard camera and uploads it to the cloud.

[1774] Input: Surrounding video data.

[1775] Data processing: cleaning and optimizing video data.

[1776] Output: Optimized video data uploaded to the cloud.

[1777] Step 6:

[1778] The server analyzes the video data uploaded to the cloud and detects vehicles driving recklessly.

[1779] Input: Video data uploaded to the cloud.

[1780] Data Computing: Uses image recognition algorithms to detect abnormal driving patterns.

[1781] Output: Detection results of dangerous driving vehicles.

[1782] Step 7:

[1783] The device will display a warning on the HMD based on the cloud analysis results.

[1784] Input: Detection results of dangerous driving vehicles.

[1785] Data processing: generating warning messages and creating synthetic voices.

[1786] Output: Visual and audio warning notification to the driver.

[1787] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1788] This system uses facial and voice information from the driver to maintain attention and support safe driving. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, it provides more appropriate support.

[1789] System Configuration

[1790] 1. Collection of facial and voice information (terminal)

[1791] The device uses the onboard camera and microphone to collect facial and audio information from the driver. Facial information includes facial expressions and gaze data, while audio information includes what is being said and tone of voice.

[1792] 2. Condition Analysis (Server)

[1793] The server analyzes the collected facial and voice information to determine the driver's condition. Specifically, it measures the driver's level of fatigue using facial expression analysis and determines the driver's level of attention through voice tone analysis. As a result, the driver's level of fatigue and attention are output as numerical values.

[1794] 3. Emotion Recognition (Emotion Engine)

[1795] The emotion engine further analyzes the collected facial and audio information to determine the driver's emotional state, which can include anger, sadness, joy, surprise, fear, etc.

[1796] 4. Providing conversation topics (server)

[1797] The server generates appropriate topics based on the driver's condition and emotional state. For example, if the driver is highly fatigued and stressed, it will provide "relaxation topics." If the driver is alert and emotionally stable, it will provide "talk about topics of interest."

[1798] 5. Notifications and conversation initiation (device)

[1799] The device will then provide a voice notification to the driver about the topic received from the server, and the conversation will progress naturally based on the driver's responses.

[1800] Analysis of surrounding traffic conditions and alerts

[1801] 1. Video data collection (terminal)

[1802] The device uses an onboard camera to collect video data about the surrounding traffic, including other vehicles and road conditions.

[1803] 2. Traffic situation analysis (server)

[1804] The server analyzes the video data and applies image recognition algorithms to detect emergency vehicles and approaching vehicles from behind, using technology to identify specific vehicles and patterns.

[1805] 3. Warning (terminal)

[1806] Based on the analysis results, the device will give the driver appropriate warnings. For example, if an emergency vehicle is approaching, the device will notify the driver, "An emergency vehicle is approaching. Please give way."

[1807] Reckless driving vehicle detection and notification

[1808] 1. Upload data to the cloud (device)

[1809] The device uploads the video data collected from the in-vehicle camera to a cloud server, where the cloud environment is equipped with advanced analytical functions, enabling efficient data processing.

[1810] 2. Cloud analysis (server)

[1811] A cloud server analyzes the uploaded data and detects dangerous driving vehicles using algorithms for pattern recognition and anomaly detection of driving behavior.

[1812] 3. Warning notification (terminal)

[1813] If cloud analysis detects a vehicle driving recklessly, the device will warn the driver, for example, by sending a message saying, "There is a vehicle driving recklessly nearby. Please be careful."

[1814] Specific examples

[1815] Specific examples of chat functions

[1816] 1. User: The driver is driving for a long time.

[1817] 2. Device: Collects the driver's facial and voice information using the vehicle's camera and microphone.

[1818] 3. Terminal: Sends the collected information to the server.

[1819] 4. Server: Analyzes the information and determines if the driver is tired.

[1820] 5. Emotion Engine: Detects fatigue and high stress levels at the same time.

[1821] 6. Server: Generate "relaxation topics."

[1822] 7. Terminal: Notifies the driver of the received topic via voice and suggests, "Why don't you think about relaxing for a moment?"

[1823] 8. User: The driver responds, "Well, let's talk about how to relax."

[1824] Examples of alert functions

[1825] 1. Device: Collects surrounding traffic conditions using an in-vehicle camera.

[1826] 2. Terminal: Sends collected video data to the server.

[1827] 3. Server: Analyzes video data and detects emergency vehicles.

[1828] 4. Server: Generate a warning message saying "An emergency vehicle is approaching. Please give way."

[1829] 5. Device: Notifies the driver of the received message via voice prompt and prompts them to give way to the emergency vehicle.

[1830] 6. User: The driver checks behind them and gives way safely.

[1831] Specific examples of dangerous driving vehicle information

[1832] 1. Terminal: Collects surrounding video data using an onboard camera.

[1833] 2. Terminal: Uploads the collected video data to the cloud server.

[1834] 3. Server (Cloud): Analyzes the uploaded data and detects dangerous driving vehicles.

[1835] 4. Server (Cloud): Generates a warning message saying, "There is a dangerously driven vehicle nearby. Please be careful."

[1836] 5. Device: The device will notify the driver of the received message via voice and urge them to be aware of their surroundings.

[1837] 6. User: Drivers should be aware of their surroundings and drive safely.

[1838] The above is a description of the specific processing steps and operations of the system of the present invention, which allows the driver to always receive appropriate support according to the driving situation, thereby enhancing the safe driving environment.

[1839] The processing flow will be explained below.

[1840] Chat function

[1841] Program processing

[1842] Step 1:

[1843] The device uses the vehicle's camera and microphone to collect the driver's facial and voice information. The facial information includes the driver's facial expressions and gaze data, and the voice information includes what is being said and the tone of voice.

[1844] Step 2:

[1845] The terminal transmits the collected face information and voice information to the server.

[1846] Step 3:

[1847] The server analyzes the received facial and voice information to determine the driver's condition. Specifically, it measures the driver's fatigue level using facial expression analysis and determines the driver's level of attention through voice tone analysis.

[1848] Step 4:

[1849] The server sends the results of the assessment to the emotion engine, which further analyzes the driver's emotional state, determining emotions such as anger, sadness, joy, surprise, and fear.

[1850] Step 5:

[1851] The server generates conversation topics appropriate for the driver based on the condition assessment results and the emotion engine's assessment results. For example, if fatigue and stress levels are high, it will provide topics related to relaxation.

[1852] Step 6:

[1853] The server sends the generated conversation topic to the terminal.

[1854] Step 7:

[1855] The device will notify the driver of the topic of conversation received by voice and begin the conversation.

[1856] Alert function

[1857] Program processing

[1858] Step 1:

[1859] The device uses an onboard camera to collect surrounding traffic information, including other vehicles and road conditions.

[1860] Step 2:

[1861] The terminal transmits the collected video data to the server.

[1862] Step 3:

[1863] The server analyzes the video data and applies image recognition algorithms to detect emergency vehicles and rapidly approaching vehicles from behind.

[1864] Step 4:

[1865] If the server detects an abnormality based on the analysis results, it generates a warning message. In the case of an emergency vehicle, it generates a message saying, "An emergency vehicle is approaching. Please give way."

[1866] Step 5:

[1867] The server generates a warning message and sends it to the terminal.

[1868] Step 6:

[1869] The device will notify the driver of any warning messages it receives via voice.

[1870] Sharing and notifying dangerous driving vehicle information

[1871] Program processing

[1872] Step 1:

[1873] The device uses an onboard camera to collect video data of the surrounding area.

[1874] Step 2:

[1875] The video data collected by the device is uploaded to a cloud server.

[1876] Step 3:

[1877] The cloud server analyzes the uploaded video data and detects dangerous driving vehicles using driving behavior pattern recognition and anomaly detection algorithms.

[1878] Step 4:

[1879] Based on the detection result, the cloud server generates a warning message if there is a reckless driving vehicle, for example, a message saying "There is a reckless driving vehicle nearby. Please be careful."

[1880] Step 5:

[1881] The cloud server generates a warning message and sends it to the terminal.

[1882] Step 6:

[1883] The device will notify the driver of any warning messages it receives via voice.

[1884] Specific examples

[1885] Specific examples of chat functions

[1886] Step 1:

[1887] The device uses the onboard camera to collect the driver's facial information (facial expressions) and voice information.

[1888] Step 2:

[1889] The terminal transmits the collected face information and voice information to the server.

[1890] Step 3:

[1891] The server analyzes facial and audio information and determines the driver's level of fatigue as "high."

[1892] Step 4:

[1893] The server sends the fatigue level assessment result to the emotion engine, which then determines that the stress level is also "high."

[1894] Step 5:

[1895] The server generates "relaxation topics" based on "high" fatigue and "high" stress levels.

[1896] Step 6:

[1897] The server sends the generated topics to the terminal.

[1898] Step 7:

[1899] The device will notify the driver of the topic it receives via voice and suggest, "Why don't you think about relaxing for a bit?"

[1900] Examples of alert functions

[1901] Step 1:

[1902] The device uses an onboard camera to collect information about surrounding traffic conditions.

[1903] Step 2:

[1904] The terminal transmits the collected video data to the server.

[1905] Step 3:

[1906] The server analyzes the video data and detects emergency vehicles.

[1907] Step 4:

[1908] The server generates a warning message saying "Emergency vehicle approaching. Please give way."

[1909] Step 5:

[1910] The server generates a message and sends it to the terminal.

[1911] Step 6:

[1912] The device will then notify the driver via voice of the message received, urging them to give way to the emergency vehicle.

[1913] Specific examples of dangerous driving vehicle information

[1914] Step 1:

[1915] The device uses an onboard camera to collect video data of the surrounding area.

[1916] Step 2:

[1917] The video data collected by the device is uploaded to a cloud server.

[1918] Step 3:

[1919] The cloud server analyzes the uploaded data and detects dangerously driving vehicles.

[1920] Step 4:

[1921] The cloud server generates a warning message saying, "There is a dangerously driven vehicle nearby. Please be careful."

[1922] Step 5:

[1923] The cloud server sends the generated message to the terminal.

[1924] Step 6:

[1925] The device will notify the driver of the received message via voice, urging them to be aware of their surroundings.

[1926] Example 2

[1927] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1928] In recent years, road traffic has become increasingly complex, creating a need for systems that can monitor drivers' attention and emotional state in real time to support safe driving. However, current systems only analyze the driver's condition obtained from facial and voice information, and lack the functionality to generate topics to provide appropriate support. Furthermore, they lack the functionality to accurately analyze surrounding traffic conditions, detect reckless driving vehicles early, and issue warnings. Therefore, there is a need for systems that can effectively support safe driving.

[1929] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for analyzing the driver's facial information and voice information to determine the driver's condition, a means for generating a conversation topic based on the driver's condition and emotional state, and a means for notifying the driver of the topic by voice and advancing the conversation. This makes it possible to monitor the driver's attention and emotional state in real time and provide appropriate assistance. Furthermore, by including a means for analyzing video data collected from an on-board camera and accurately analyzing the surrounding traffic conditions, and a means for uploading the collected video data to the cloud and analyzing it on the cloud to detect reckless driving vehicles, it becomes possible to promptly alert and warn the driver.

[1930] "Driver" means a person who drives a vehicle.

[1931] "Facial information" refers to data related to the driver's facial expressions, gaze, etc.

[1932] "Voice information" refers to data related to the driver's voice, such as what they say and their tone of voice.

[1933] "Terminal" refers to input devices such as cameras and microphones installed in the vehicle.

[1934] "Server" refers to the central system that analyzes the collected data and generates appropriate responses based on the results.

[1935] "Emotion engine" refers to a system that determines the driver's emotional state based on facial and voice information.

[1936] "Video data" refers to video information about surrounding traffic conditions collected by an in-vehicle camera.

[1937] "Condition" refers to the driver's level of fatigue and attention.

[1938] "Conversational topics" refer to topics that are generated based on the driver's condition and emotional state.

[1939] "Traffic conditions" refers to the traffic environment, including other vehicles and surrounding road conditions.

[1940] "Caution" refers to a warning message to encourage drivers to drive safely.

[1941] "Dangerous driving vehicle" refers to a vehicle that is driving abnormally in the surrounding traffic conditions and may pose a safety hazard.

[1942] "Cloud" refers to a general term for computer resources provided through multiple servers and storage on the Internet.

[1943] "Cloud analytics" refers to data analysis processing performed on the cloud.

[1944] This invention is a system that uses facial and audio information of the driver to maintain attention and support safe driving. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, more appropriate support is provided. Specific embodiments are described in detail below.

[1945] System configuration and program processing

[1946] 1. Collection of facial and voice information (terminal)

[1947] The device uses an onboard camera and microphone to collect the driver's facial and audio information. Facial information includes facial expressions and gaze data, while audio information includes what is being said and tone of voice. This collection process uses a high-resolution camera and a highly sensitive microphone.

[1948] 2. Condition Analysis (Server)

[1949] The server analyzes the collected facial and audio information to determine the driver's condition. Specifically, it measures the driver's level of fatigue using facial expression analysis and determines the driver's level of attention through voice tone analysis. This analysis uses facial recognition algorithms (e.g., OpenCV and DeepFace) and voice analysis algorithms (e.g., Google Cloud Speech-to-Text API). The resulting numerical output represents the driver's level of fatigue and attention.

[1950] 3. Emotion Recognition (Emotion Engine)

[1951] The emotion engine further analyzes the collected facial and audio information to determine the driver's emotional state, which can include anger, sadness, joy, surprise, fear, etc. This analysis is performed using a deep learning model (e.g., an emotion recognition model using PyTorch or TensorFlow).

[1952] 4. Providing conversation topics (server)

[1953] The server generates appropriate topics based on the determined condition and emotional state. For example, if the driver is highly fatigued and stress levels are rising, it will provide "topics related to relaxation." If the driver is alert and emotionally stable, it will generate "talk about topics of interest." This topic generation uses a natural language generation model (e.g., a generative AI model).

[1954] 5. Notifications and conversation initiation (device)

[1955] The device then notifies the driver of the topic received from the server by voice. The notification is in the form of a message such as, "Would you like to think about relaxing for a moment?" When the driver responds, the system will proceed with the conversation based on that response. This also uses a high-definition speaker to notify the driver by voice.

[1956] Analysis of surrounding traffic conditions and alerts

[1957] 1. Video data collection (terminal)

[1958] The device uses an onboard camera to collect surrounding traffic conditions, including other vehicles and road conditions. The camera has a wide-angle lens, allowing it to collect high-resolution images even at night.

[1959] 2. Traffic situation analysis (server)

[1960] The server analyzes the video data and applies image recognition algorithms to detect emergency vehicles and approaching vehicles from behind, using techniques such as YOLOv5 and Mask R-CNN to identify specific vehicles and patterns.

[1961] 3. Warning (terminal)

[1962] Based on the analysis results, the device will give the driver appropriate warnings. For example, if an emergency vehicle is approaching, the device will notify the driver, "An emergency vehicle is approaching. Please give way."

[1963] Reckless driving vehicle detection and notification

[1964] 1. Upload data to the cloud (device)

[1965] The device uploads the video data collected from the in-vehicle camera to a cloud server, which is equipped with advanced analytical functions, enabling efficient data processing.

[1966] 2. Cloud analysis (server)

[1967] The cloud server analyzes the uploaded data and detects dangerous driving vehicles using driving behavior pattern recognition and anomaly detection algorithms (such as AWS Rekognition and Google Cloud Vision API).

[1968] 3. Warning notification (terminal)

[1969] If cloud analysis detects a vehicle driving recklessly, the device will warn the driver. For example, it will notify the driver by saying, "There is a vehicle driving recklessly nearby. Please be careful." The driver can then remain vigilant and drive safely based on the warning.

[1970] Specific examples

[1971] Specific examples of chat functions

[1972] User: The driver is driving for a long time.

[1973] Device: Collects facial and audio information from the driver using the vehicle's camera and microphone.

[1974] Terminal: Sends collected information to the server.

[1975] Server: Analyzes the information and determines whether the driver is tired.

[1976] Emotion engine: Determines fatigue and high stress levels at the same time.

[1977] Server: Generate "relaxation topics."

[1978] Device: Notifies the driver of the received topic via voice and suggests, "Why don't you think about relaxing for a moment?"

[1979] User: The driver responds, "Well, let's talk about how to relax."

[1980] Example prompt: "When drivers are on the road for long periods of time, come up with ways to talk about relaxation to help them stay alert."

[1981] Examples of alert functions

[1982] Device: Uses an in-car camera to collect information about surrounding traffic conditions.

[1983] Terminal: Sends collected video data to the server.

[1984] Server: Analyzes video data and detects emergency vehicles.

[1985] Server: Generate a warning message saying "Emergency vehicle approaching. Please give way."

[1986] Device: The device will notify the driver of the received message via voice and prompt them to give way to the emergency vehicle.

[1987] User: Drivers should check behind them and give way safely.

[1988] Example prompt: "If an emergency vehicle is approaching, think of a way to quickly notify the driver."

[1989] Specific examples of dangerous driving vehicle information

[1990] Terminal: Collects video data of the surrounding area using an onboard camera.

[1991] Terminal: Uploads collected video data to a cloud server.

[1992] Server (Cloud): Analyzes uploaded data and detects dangerous driving vehicles.

[1993] Server (Cloud): Generates a warning message saying, "There is a dangerously driven vehicle nearby. Please be careful."

[1994] Device: The device will notify the driver of received messages via voice and urge them to be aware of their surroundings.

[1995] User: Drivers should be aware of their surroundings and drive safely.

[1996] Example prompt: "If there is a dangerously driving vehicle nearby, suggest ways to warn the driver and encourage safe driving."

[1997] The above is a specific embodiment of the present invention. By using this system, the driver can always receive appropriate support according to the driving situation, and the safe driving environment is strengthened.

[1998] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1999] Program processing flow and specific operations

[2000] Driver health analysis and support

[2001] Step 1:

[2002] The device uses the onboard camera and microphone to collect the driver's facial and audio information, including facial expression and gaze data, as well as voice tone and content.

[2003] Input: Video and audio data from the onboard camera and microphone

[2004] Output: Collected facial and audio information

[2005] Step 2:

[2006] The terminal transmits the collected face information and voice information to the server.

[2007] Input: Face and voice information

[2008] Output: Data sent to the server

[2009] Step 3:

[2010] The server analyzes the collected facial and voice information to determine the driver's condition. Specifically, it measures the driver's fatigue level using facial expression analysis and analyzes the tone of the voice.

[2011] Input: Face and voice information

[2012] Output: Numerical data on driver fatigue and alertness

[2013] Step 4:

[2014] The emotion engine performs additional analysis to determine the driver's emotional state, specifically by analyzing facial and audio information using deep learning models (e.g., PyTorch) to identify emotions.

[2015] Input: Face and voice information

[2016] Output: Driver's emotional state (e.g., anger, sadness, joy, etc.)

[2017] Step 5:

[2018] The server generates appropriate conversation topics based on your condition and emotional state, using generative AI models to generate topics of relaxation, interest, etc.

[2019] Input: Driver condition and emotional state

[2020] Output: Generated conversation topics

[2021] Step 6:

[2022] The device will then notify the driver of the generated topic by voice and start a conversation. The notification content may be something like, "Why don't you think about relaxing for a bit?"

[2023] Input: Generated conversation topics

[2024] Output: Audio notification to the driver

[2025] Analysis of surrounding traffic conditions and alerts

[2026] Step 7:

[2027] The device uses an onboard camera to collect information about the surrounding traffic conditions, including other vehicles and road conditions.

[2028] Input: Surrounding image data from a camera

[2029] Output: Collected traffic video data

[2030] Step 8:

[2031] The terminal transmits the collected video data to the server.

[2032] Input: Surrounding video data

[2033] Output: Video data sent to the server

[2034] Step 9:

[2035] The server analyzes the video data and determines the traffic situation, using an image recognition algorithm (e.g., YOLOv5) to detect emergency vehicles and vehicles approaching from behind.

[2036] Input: Surrounding traffic video data

[2037] Output: Parsed traffic data

[2038] Step 10:

[2039] Based on the analysis results, the server generates a warning message for the driver.

[2040] Input: Parsed traffic data

[2041] Output: Attention message

[2042] Step 11:

[2043] The device will then notify the driver of the warning message it receives by voice, urging them to take appropriate action. For example, it may say, "An emergency vehicle is approaching. Please give way."

[2044] Input: Attention message

[2045] Output: Audio notification to the driver

[2046] Reckless driving vehicle detection and notification

[2047] Step 12:

[2048] The device uploads the video data collected through the onboard camera to a cloud server.

[2049] Input: Video data

[2050] Output: Uploaded video data

[2051] Step 13:

[2052] The cloud server analyzes the uploaded data and detects reckless driving vehicles using advanced pattern recognition and anomaly detection algorithms (e.g., AWS Rekognition).

[2053] Input: Uploaded video data

[2054] Output: Detected dangerous driving vehicles

[2055] Step 14:

[2056] The cloud server generates a warning message to the driver based on the analysis results.

[2057] Input: Detected dangerous driving vehicles

[2058] Output: Warning message

[2059] Step 15:

[2060] The device will then generate a warning message and notify the driver by voice, urging them to be vigilant about their surroundings. For example, it will say, "There is a vehicle driving recklessly nearby. Please be careful."

[2061] Input: warning message

[2062] Output: Audio notification to the driver

[2063] (Application example 2)

[2064] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[2065] In autonomous vehicles, the system certainly takes over most of the driving, but it is impossible to completely eliminate the dangers of driver distraction or fatigue. Furthermore, the driver's emotional state while driving also has a significant impact on driving safety, so appropriate reminders and support are essential. However, conventional systems have had difficulty accurately recognizing the driver's emotional state and providing appropriate real-time support. This has led to problems in effectively ensuring driver safety.

[2066] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting face information and voice information of the driver, means for analyzing the face information and voice information of the driver to determine the condition, means for providing conversation topics according to the condition, means for recognizing the emotional state of the driver, and means for providing voice notification in real time based on the emotional state. This makes it possible to monitor the driver's attention lapse, fatigue, and emotional state in real time and provide appropriate warnings and support.

[2067] "Driver" means the passenger who operates and monitors an automated driving vehicle.

[2068] "Facial information" refers to image data of the driver's face acquired from a camera, and includes facial expression and gaze data.

[2069] "Voice information" refers to data about the driver's speech and tone of voice obtained from a microphone.

[2070] "Condition" refers to the driver's level of attention and fatigue, which is analyzed based on facial and voice information.

[2071] "Emotional state" refers to the driver's emotions (e.g., anger, sadness, joy, etc.) analyzed from facial and voice information.

[2072] "Real-time" means that data collection, analysis, and notification occur immediately.

[2073] The "means for providing conversation topics" is a system that generates appropriate conversation themes and topics based on the driver's condition and emotional state.

[2074] "Means for providing voice notification" refers to a system that uses voice output to alert the driver and provide conversation based on the analysis results.

[2075] "Surrounding traffic conditions" refers to data about other vehicles and road conditions collected through in-vehicle cameras.

[2076] "Warning" refers to warning drivers about road conditions or emergency situations based on analyzed information.

[2077] The "cloud" is a remote server environment that analyzes and stores data via the Internet.

[2078] "Dangerous driving vehicles" refer to nearby vehicles whose abnormal driving behavior has been detected based on the analysis results.

[2079] The "means of issuing a warning" is a system that notifies drivers of the presence of a vehicle engaged in dangerous driving based on the results of cloud analysis.

[2080] This invention is a system that monitors the driver's attention and emotional state in real time in an autonomous vehicle and provides appropriate support and warnings. The system uses an on-board camera and microphone to collect the driver's facial and voice information, and analyzes this data on a server.

[2081] System Configuration

[2082] 1. Data Collection

[2083] Hardware: In-car camera (any USB camera), microphone

[2084] Software: OpenCV(cv2), PyAudio

[2085] Processing: Collecting facial and audio information from the driver. For example, an onboard camera records the driver's facial expressions and gaze, and a microphone records speech and tone of voice.

[2086] 2. Data Analysis

[2087] Hardware: High-performance server

[2088] Software: EmotionEngine (emotion recognition engine), custom voice analysis algorithm

[2089] Processing: The server analyzes the collected facial and audio information to determine the driver's condition and emotional state. Specifically, the server measures the driver's level of fatigue from facial expressions and analyzes their level of attention from the tone of their voice.

[2090] 3. Determining Conditions and Emotional States

[2091] Example: If a driver has been driving for a long time, the server will determine that the driver is tired based on facial information and that the driver's attention level is low based on the tone of voice. In this case, the server will provide the driver with appropriate conversation topics.

[2092] 4. Real-time support and notifications

[2093] Hardware: In-car audio system

[2094] Software: pyttsx3 (speech synthesis library)

[2095] Processing: Based on the results of the assessment, the server generates appropriate conversation topics and warning messages. For example, if the server determines that the driver is tired, it will notify the driver by voice, "Would you like to rest your eyes for a while?"

[2096] 5. Traffic situation analysis

[2097] Hardware: In-car camera

[2098] Software: Image recognition algorithms

[2099] Processing: The onboard camera collects information on surrounding traffic conditions in real time and analyzes emergency vehicles and vehicles approaching rapidly from behind, etc. Based on this, necessary warnings are given to the driver via voice.

[2100] 6. Cloud Analysis

[2101] Hardware: Cloud Server

[2102] Software: Advanced analysis algorithms

[2103] Processing: Collected video data is uploaded to the cloud and analyzed there. The analysis results are fed back to the vehicle in real time, and the presence of reckless driving vehicles is notified.

[2104] Examples and prompts

[2105] Specific examples

[2106] 1. Example of high fatigue: The driver is determined to be tired and a voice notification is given saying, "Would you like to talk about relaxing a bit?"

[2107] 2. Example of inattention: The driver is determined to be inattentive and an appropriate voice warning is given.

[2108] 3. Example of a traffic situation-based warning: If an emergency vehicle is approaching from behind, a voice notification will say, "An emergency vehicle is approaching. Please give way."

[2109] Prompt Sentence Examples

[2110] "The facial image and voice data indicate that the driver is tired. The driver's fatigue level is high and their attention level is low. In this case, please generate conversation topic suggestions that are effective in helping the driver relax."

[2111] conclusion

[2112] This system can improve driving safety by monitoring the driver's attention and emotional state in real time and providing appropriate support and warnings.

[2113] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2114] Step 1:

[2115] The device uses an onboard camera and microphone to collect facial and audio information from the driver.

[2116] Input: Driver's face image, voice data

[2117] How it works: The onboard camera captures the driver's facial expressions and gaze information, and the microphone records the driver's speech and tone of voice. These data are acquired in real time.

[2118] Output: Collected facial image data and voice data

[2119] Step 2:

[2120] The terminal transmits the collected face information and voice information to the server.

[2121] Input: collected facial image data, voice data

[2122] Specific operation: The terminal uploads the data stored in the local buffer to the server via the network.

[2123] Output: Facial image data and audio data sent to the server

[2124] Step 3:

[2125] The server analyzes facial and audio information to determine the driver's condition.

[2126] Input: Facial image data, audio data

[2127] How it works: The EmotionEngine on the server analyzes facial image data and infers the driver's fatigue level and emotional state from their facial expressions. A custom voice analysis algorithm is used on the voice data to determine the driver's level of attention through tone and language analysis.

[2128] Output: Driver condition data (fatigue level, attention state, emotional state)

[2129] Step 4:

[2130] The server generates appropriate conversation topics and reminder messages based on the determined condition and emotional state.

[2131] Input: Driver condition data

[2132] Specific operation: Based on the results of the assessment, the server uses an AI model to generate optimal conversation topics and attention-grabbing content for the driver. For example, if the driver's level of fatigue is high, it will suggest topics related to relaxation, and if the driver's attention level is low, it will generate a warning message.

[2133] Output: Conversation topics, warning messages

[2134] Step 5:

[2135] The device notifies the driver by voice of the conversation topic and warning messages received from the server.

[2136] Input: conversation topics, warning messages

[2137] Specific operation: The device uses a speech synthesis library such as pyttsx3 to convert text data into speech data and notify the driver through the car speaker.

[2138] Output: Voice message notified to the driver

[2139] Step 6:

[2140] The terminal collects video data from the vehicle-mounted camera and transmits it to a server.

[2141] Input: Surrounding traffic video data

[2142] Specific operation: The onboard camera captures the surrounding traffic conditions, collects the video data in real time, and sends it to a server via the network.

[2143] Output: Video data sent to the server

[2144] Step 7:

[2145] The server analyzes the video data to detect emergency vehicles and vehicles driving recklessly.

[2146] Input: Video data

[2147] How it works: Image recognition algorithms on the server process video data to detect specific traffic patterns and unusual driving behavior. The algorithms identify emergency vehicles and rapidly approaching vehicles.

[2148] Output: Analysis of surrounding traffic conditions (location information for emergency vehicles and dangerously driving vehicles)

[2149] Step 8:

[2150] The server generates a warning message for the driver based on the analysis results and sends it to the terminal.

[2151] Input: Traffic situation analysis results

[2152] Specific operation: The server generates a warning message for the driver based on the detected emergency vehicle or dangerous driving vehicle information and sends it to the terminal.

[2153] Output: Warning message

[2154] Step 9:

[2155] The terminal notifies the driver of the warning message received from the server by voice.

[2156] Input: warning message

[2157] Specific operation: The device uses a speech synthesis library to convert the warning message into voice data and notify the driver through the car speaker.

[2158] Output: Audio warning message notified to the driver

[2159] This allows drivers to always receive appropriate support in real time that is tailored to the driving situation, enhancing the safe driving environment.

[2160] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[2161] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2162] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[2163] [Fourth embodiment]

[2164] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[2165] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[2166] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[2167] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[2168] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[2169] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[2170] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[2171] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[2172] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[2173] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[2174] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[2175] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[2176] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2177] This invention is a system that uses facial and audio information from the driver to maintain attention and support safe driving. The system works by collecting and analyzing data via an on-board camera and microphone.

[2178] System Configuration

[2179] 1. Collection of facial and voice information (terminal)

[2180] The device uses the onboard camera and microphone to collect facial and audio information from the driver. Facial information includes facial expressions and gaze data, while audio information includes what is being said and tone of voice.

[2181] 2. Condition Analysis (Server)

[2182] The server analyzes the collected facial and voice information to determine the driver's condition. Specifically, it measures the driver's level of fatigue using facial expression analysis and determines the driver's level of attention through voice tone analysis. As a result, the driver's level of fatigue and attention are output as numerical values.

[2183] 3. Providing conversation topics (server)

[2184] The server generates appropriate topics based on the determined condition. For example, if the driver is highly fatigued, it might suggest taking a break, or if the driver is alert, it might suggest talking about an interesting topic.

[2185] 4. Notifications and conversation initiation (device)

[2186] The device will then provide a voice notification to the driver about the topic received from the server, and the conversation will progress naturally based on the driver's responses.

[2187] Analysis of surrounding traffic conditions and alerts

[2188] 1. Video data collection (terminal)

[2189] The device collects video data from the in-vehicle camera, including information on surrounding vehicles and road conditions.

[2190] 2. Traffic situation analysis (server)

[2191] The server analyzes the video data to detect emergency vehicles and vehicles approaching from behind, using image recognition technology to identify specific vehicles and patterns.

[2192] 3. Warning (terminal)

[2193] Based on the analysis results, the device will give the driver appropriate warnings. For example, if an emergency vehicle is approaching, the device will notify the driver, "An emergency vehicle is approaching. Please give way."

[2194] Reckless driving vehicle detection and notification

[2195] 1. Upload data to the cloud (device)

[2196] The video data collected by the device is uploaded to the cloud, where advanced analytical functions enable efficient data processing.

[2197] 2. Cloud analysis (server)

[2198] A cloud server analyzes the uploaded data and detects dangerous driving vehicles using algorithms for pattern recognition and anomaly detection of driving behavior.

[2199] 3. Warning notification (terminal)

[2200] If cloud analysis detects a vehicle driving recklessly, the device will warn the driver, for example, by sending a message saying, "There is a vehicle driving recklessly nearby. Please be careful."

[2201] Specific examples

[2202] Example 1: When a driver becomes tired while driving

[2203] User: The user (driver) is driving for a long time.

[2204] Device: The onboard camera collects the driver's facial expression data, and the microphone collects audio data.

[2205] Server: The server analyzes the collected data and determines the driver's fatigue level as "high."

[2206] Server: The server generates the topic "Would you like to take a break?"

[2207] Device: The device will prompt the driver with a voice message asking, "Would you like to take a break?"

[2208] User: The driver responds, "Yeah, let's take a break," and takes a break.

[2209] Example 2: When an emergency vehicle is approaching

[2210] Terminal: The onboard camera collects video data of the surrounding area.

[2211] Server: The server analyzes the video data and detects emergency vehicles.

[2212] Server: The server generates a warning message saying "Emergency vehicle approaching. Please give way."

[2213] Device: The device will notify the driver via voice message, "An emergency vehicle is approaching. Please give way."

[2214] User: Drivers should check behind them and give way safely.

[2215] Example 3: Reckless driving vehicles nearby

[2216] Terminal: The onboard camera collects video data of the surrounding area.

[2217] Device: The device uploads the video data to the cloud.

[2218] Server (Cloud): The server analyzes the uploaded data and detects dangerous driving vehicles.

[2219] Server (Cloud): The server generates a warning message saying, "There is a dangerously driven vehicle nearby. Please be careful."

[2220] Device: The device will notify the driver by voice, "There is a vehicle driving recklessly nearby. Please be careful."

[2221] User: Drivers should be aware of their surroundings and drive safely.

[2222] This ensures that drivers always receive appropriate support tailored to the driving situation, enhancing the safe driving environment.

[2223] The processing flow will be explained below.

[2224] Chat function

[2225] Program processing

[2226] Step 1:

[2227] The device uses the vehicle's camera and microphone to collect facial and audio information from the driver, including their facial expressions, gaze, and tone of voice.

[2228] Step 2:

[2229] The terminal transmits the collected face information and voice information to the server.

[2230] Step 3:

[2231] The server analyzes the received facial and voice information to determine the driver's condition. Specifically, it measures the driver's fatigue level using facial expression analysis and determines the driver's level of attention through voice tone analysis.

[2232] Step 4:

[2233] Based on the results of the assessment, the server generates conversation topics that correspond to the driver's condition. For example, if the driver is highly fatigued, it will suggest taking a break, and if the driver is alert, it will suggest talking about a topic of interest.

[2234] Step 5:

[2235] The server sends the generated conversation topic to the terminal.

[2236] Step 6:

[2237] The device will notify the driver of the topic of conversation received by voice and begin the conversation.

[2238] Alert function

[2239] Program processing

[2240] Step 1:

[2241] The device uses an onboard camera to collect video data about the surrounding traffic, including other vehicles and road conditions.

[2242] Step 2:

[2243] The terminal transmits the collected video data to the server.

[2244] Step 3:

[2245] The server analyzes the video data and applies image recognition algorithms to detect emergency vehicles and rapidly approaching vehicles from behind.

[2246] Step 4:

[2247] If the server detects an abnormality based on the analysis results, it generates a warning message. For example, in the case of an emergency vehicle, it generates a message saying, "An emergency vehicle is approaching. Please give way."

[2248] Step 5:

[2249] The server generates a warning message and sends it to the terminal.

[2250] Step 6:

[2251] The device will notify the driver of any warning messages it receives via voice.

[2252] Sharing and notifying dangerous driving vehicle information

[2253] Program processing

[2254] Step 1:

[2255] The device uses an onboard camera to collect video data of the surrounding area.

[2256] Step 2:

[2257] The video data collected by the device is uploaded to a cloud server.

[2258] Step 3:

[2259] The cloud server analyzes the uploaded video data to detect dangerous driving vehicles, using algorithms that identify specific driving behaviors.

[2260] Step 4:

[2261] Based on the detection result, the cloud server generates a warning message if there is a reckless driving vehicle, for example, a message saying "There is a reckless driving vehicle nearby. Please be careful."

[2262] Step 5:

[2263] The cloud server generates a warning message and sends it to the terminal.

[2264] Step 6:

[2265] The device will notify the driver of any warning messages it receives via voice.

[2266] Specific examples

[2267] Specific examples of chat functions

[2268] Step 1:

[2269] The device uses the onboard camera and microphone to collect the driver's facial and voice information.

[2270] Step 2:

[2271] The device sends the collected information to the server.

[2272] Step 3:

[2273] The server analyzes the information and determines that the driver is tired.

[2274] Step 4:

[2275] The server generates a conversation topic suggesting a break, such as "Would you like to take a short break?"

[2276] Step 5:

[2277] The server sends the generated topics to the terminal.

[2278] Step 6:

[2279] The device will notify the driver of the topic it receives via voice and suggest, "Would you like to take a short break?"

[2280] Examples of alert functions

[2281] Step 1:

[2282] The device uses an onboard camera to collect information about surrounding traffic conditions.

[2283] Step 2:

[2284] The terminal transmits the collected video data to the server.

[2285] Step 3:

[2286] The server analyzes the video data and detects emergency vehicles.

[2287] Step 4:

[2288] The server generates a warning message saying "Emergency vehicle approaching. Please give way."

[2289] Step 5:

[2290] The server generates a message and sends it to the terminal.

[2291] Step 6:

[2292] The device will then notify the driver via voice of the message received, urging them to give way to the emergency vehicle.

[2293] Specific examples of dangerous driving vehicle information

[2294] Step 1:

[2295] The device uses an onboard camera to collect video data of the surrounding area.

[2296] Step 2:

[2297] The video data collected by the device is uploaded to a cloud server.

[2298] Step 3:

[2299] The cloud server analyzes the video data and detects vehicles driving recklessly.

[2300] Step 4:

[2301] The cloud server generates a warning message saying, "There is a dangerously driven vehicle nearby. Please be careful."

[2302] Step 5:

[2303] The cloud server sends the generated message to the terminal.

[2304] Step 6:

[2305] The device will notify the driver of the received message via voice, urging them to be aware of their surroundings.

[2306] The above is a description of the specific processing steps and operations in the system of the present invention.

[2307] Example 1

[2308] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2309] In today's modern traffic society, there are many accidents that are believed to be caused by driver inattention or fatigue. Furthermore, there is a need for appropriate responses to surrounding traffic conditions and the presence of dangerously driven vehicles. However, conventional systems have had the problem of being unable to accurately grasp the driver's condition and traffic situation in real time and provide appropriate support and warnings.

[2310] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2311] In this invention, the server includes means for collecting face information and voice information of the driver, means for analyzing the face information and voice information of the driver to determine the driver's condition, means for providing a topic of conversation according to the condition, means for notifying the driver of the topic of conversation, means for analyzing the driver's responses and advancing the conversation, means for collecting video data from an in-vehicle camera, means for analyzing the video data to analyze surrounding traffic conditions, means for issuing a warning to the driver based on the traffic conditions, means for uploading the collected video data to a cloud, means for analyzing the video data on the cloud and detecting reckless driving vehicles, and means for issuing a warning to the driver based on the cloud analysis results. This makes it possible to detect a driver's dwindling attention or fatigue in real time, and maintain attention by providing appropriate topics, while also issuing appropriate warnings regarding surrounding traffic conditions and detecting and notifying reckless driving vehicles.

[2312] "Driver's facial information" is data including the driver's facial features such as facial expression and gaze.

[2313] "Voice information" is data that includes voice characteristics such as the driver's tone of voice and what is being said.

[2314] The "means for collecting face information and voice information" refers to a device that acquires the driver's face information and voice information using an in-vehicle camera, microphone, etc.

[2315] The "means for determining the condition" is a device or algorithm that analyzes the collected facial and audio information to determine the driver's level of fatigue and attention.

[2316] A "means for providing conversation topics" is a device or algorithm that generates appropriate conversation topics based on the driver's condition.

[2317] "Means for notifying the driver of the topic of conversation" refers to a device that audibly or visually conveys the generated topic of conversation to the driver through a receiver.

[2318] The "means for analyzing driver responses and advancing the conversation" refers to a device or algorithm that collects responses from the driver, analyzes their content, generates new conversation topics, and advances the conversation.

[2319] The "means for collecting video data from an in-vehicle camera" is a device that acquires video data of the surroundings using a camera mounted on a vehicle.

[2320] The "means for analyzing the surrounding traffic conditions by analyzing the video data" refers to a device or algorithm that analyzes the acquired video data and identifies the traffic conditions of surrounding vehicles, emergency vehicles, etc.

[2321] The "means for issuing a warning to the driver based on the traffic conditions" is a device that issues an appropriate warning to the driver based on the analysis results.

[2322] The "means for uploading video data to the cloud" is a device that transfers collected video data to a cloud environment via the Internet.

[2323] The "means for analyzing video data on the cloud and detecting reckless driving vehicles" refers to a device or algorithm that analyzes video data in a cloud environment and detects signs of reckless driving.

[2324] The "means for issuing a warning to the driver based on the results of cloud analysis" is a device that issues a warning to the driver for safe driving based on the results of cloud analysis.

[2325] This invention is a system that uses facial and audio information of a driver to maintain attention and support safe driving. This system operates by collecting and analyzing data via an on-board camera and microphone. Specific embodiments of this system are described below.

[2326] System Configuration

[2327] Collecting facial and audio information

[2328] The device uses an onboard camera (e.g., a network camera) and microphone (e.g., a high-sensitivity microphone) to collect the driver's facial and audio information. Facial information includes the driver's facial expressions and gaze data. Audio information includes what the driver says and the tone of their voice. For example, the camera captures the driver's eye movements and facial expressions, and the microphone records the pitch and speed of their voice.

[2329] Condition Analysis

[2330] The server analyzes the collected facial and audio information to determine the driver's condition. Specifically, it measures the driver's level of fatigue using facial expression analysis (e.g., image processing algorithms) and determines the driver's level of attention through voice tone analysis (e.g., voice recognition software). As a result, the driver's level of fatigue and attention state are output as numerical values. For example, frequent eye closure and a restless voice are detected as signs of fatigue.

[2331] Providing topics for conversation

[2332] The server generates appropriate topics based on the determined condition. For example, if the driver is highly fatigued, it will "suggest a break," and if the driver is alert, it will "talk about a topic of interest." This process uses a generative AI model, which generates, for example, "driver's interests" based on the prompt sentence.

[2333] Notifications and conversation starters

[2334] The device then notifies the driver of the topic received from the server by voice. Specifically, it plays back a message generated using speech synthesis software (e.g., text-to-speech software). The conversation progresses naturally based on the driver's response. For example, the device may say, "Do you want to take a short break?" and the driver may respond, "Yes, let's take a short break."

[2335] Analysis of surrounding traffic conditions and alerts

[2336] The device collects video data from the onboard camera. This camera captures surrounding vehicles and road conditions in real time. The video data is sent to a server, which analyzes it to detect approaching emergency vehicles and rapidly approaching vehicles from behind. Image recognition technology (e.g., deep learning models) is used for the analysis. Based on the analysis results, the device issues appropriate warnings to the driver. For example, it may notify the driver, "An emergency vehicle is approaching. Please give way."

[2337] Reckless driving vehicle detection and notification

[2338] The video data collected by the device is uploaded to the cloud. The cloud environment is equipped with advanced analytical functions; for example, a server on the cloud analyzes the uploaded data to detect vehicles that are driving recklessly. The analysis uses driving behavior pattern recognition and anomaly detection algorithms. If a vehicle is detected as a result of cloud analysis, the device will notify the driver, "There is a vehicle that is driving recklessly nearby. Please be careful."

[2339] Specific examples

[2340] Example 1: When a driver becomes tired while driving

[2341] User: The driver is driving for a long time.

[2342] Device: The onboard camera collects the driver's facial expression data, and the microphone collects audio data.

[2343] Server: The server analyzes the collected data and determines the driver's fatigue level as "high."

[2344] Server: The server generates the message "Would you like to take a break?"

[2345] Device: Your device will say, "Would you like to take a break?"

[2346] User: The driver responds, "Yeah, let's take a break," and takes a break.

[2347] Example 2: When an emergency vehicle is approaching

[2348] Terminal: The onboard camera collects video data of the surrounding area.

[2349] Server: The server analyzes the video data and detects emergency vehicles.

[2350] Server: The server generates the message "Emergency vehicle approaching. Please give way."

[2351] Device: The device will announce "An emergency vehicle is approaching. Please give way."

[2352] User: Drivers should check behind them and give way safely.

[2353] Example 3: Reckless driving vehicles nearby

[2354] Device: The onboard camera collects images of the surrounding area.

[2355] Device: The device uploads the video data to the cloud.

[2356] Server (Cloud): The server analyzes the uploaded data and detects dangerous driving vehicles.

[2357] Server (Cloud): The server generates a message saying, "There is a dangerously driven vehicle nearby. Please be careful."

[2358] Device: The device will announce with a voice message, "There is a dangerously driven vehicle nearby. Please be careful."

[2359] User: The driver remains alert to their surroundings and continues driving safely.

[2360] The system is designed to provide drivers with appropriate support in real time while driving, thereby contributing to an enhanced safe driving environment.

[2361] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2362] Step 1:

[2363] (Data collection begins)

[2364] The device activates the on-board camera and microphone and starts collecting the driver's facial and audio information. The input is real-time data from the on-board camera and microphone, and the output is stored in the device as facial and audio information.

[2365] Step 2:

[2366] (Data import)

[2367] The device uses a camera to capture the driver's facial image and facial expressions, and a microphone to pick up audio. Specifically, the camera recognizes each part of the face (eyes, mouth, etc.), and the microphone records the pitch and speed of the audio. The input is video and audio data captured in real time, and the output is temporary storage of this data and format conversion for processing.

[2368] Step 3:

[2369] (Data transmission)

[2370] The device transmits the collected facial and audio information to the server in real time. The input is the temporarily stored facial and audio information, and the output is a data stream to the server.

[2371] Step 4:

[2372] (Facial expression analysis)

[2373] The server analyzes facial information, extracts facial expression data, and evaluates the driver's level of fatigue and attention. Specifically, the facial expression analysis algorithm detects the frequency of eye opening and closing and the upward and downward movement of the corners of the mouth. The input is the facial information sent to the server, and the output is a numerical evaluation of the driver's level of fatigue and attention.

[2374] Step 5:

[2375] (audio analysis)

[2376] The server analyzes the voice information and evaluates the driver's mental state based on the tone of the voice and the content of what is being said. Specifically, the voice recognition software analyzes changes in the tempo and volume of the voice. The input is the voice information sent to the server, and the output is the evaluation result of the driver's mental state.

[2377] Step 6:

[2378] (comprehensive evaluation)

[2379] The server combines the results of facial expression analysis and voice analysis to quantify the driver's condition. The input is the results of facial expression analysis and voice analysis, and the output is an integrated condition assessment.

[2380] Step 7:

[2381] (Topic generation)

[2382] The server generates appropriate conversation topics based on the condition evaluation results. Using a generative AI model, it generates, for example, "rest suggestions" or "topics of interest" based on the prompt. The input is the condition evaluation results, and the output is the generated conversation topic.

[2383] Step 8:

[2384] (Send message)

[2385] The server sends the generated message to the terminal. The input is the generated conversation topic, and the output is the message data to the terminal.

[2386] Step 9:

[2387] (Notification execution)

[2388] The device receives messages from the server and notifies the driver using a speech synthesis function. Specifically, the message is converted into speech using text-to-speech software. The input is message data from the server, and the output is a voice notification.

[2389] Step 10:

[2390] (Response analysis)

[2391] The device collects the driver's responses with a microphone and sends the audio to a server for analysis. The input is the driver's audio responses, and the output is a data stream to the server for analysis.

[2392] Step 11:

[2393] (Conversation progress)

[2394] The server generates a new topic based on the driver's response and continues the conversation through the device. It uses a generative AI model to generate the next appropriate topic. The input is the response analysis result, and the output is the next conversation topic.

[2395] Step 12:

[2396] (Video data collection)

[2397] The device collects video data from the in-vehicle camera in real time. The input is the surrounding video data, and the output is the video data stored in the device.

[2398] Step 13:

[2399] (Video data transmission)

[2400] The terminal sends the collected video data to the server. The input is the collected video data, and the output is a data stream to the server.

[2401] Step 14:

[2402] (traffic situation analysis)

[2403] The server analyzes the video data and identifies the approach of specific vehicles or emergency vehicles. Specifically, a deep learning model recognizes the lights and sirens of emergency vehicles. The input is the video data sent to the server, and the output is the analysis result of the traffic situation.

[2404] Step 15:

[2405] (Warning message generation)

[2406] The server generates the necessary warning messages based on the analysis results. The input is the traffic situation analysis results, and the output is the generated warning messages.

[2407] Step 16:

[2408] (Alert notice)

[2409] The terminal notifies the driver of the attention message received from the server by voice. The input is the message data from the server, and the output is the voice notification.

[2410] Step 17:

[2411] (Cloud Upload)

[2412] The device uploads the collected video data to the cloud. The input is the collected video data, and the output is a data stream to the cloud.

[2413] Step 18:

[2414] (Cloud analysis)

[2415] The server (cloud) analyzes the uploaded video data and detects dangerous driving vehicles. It uses driving behavior pattern recognition and anomaly detection algorithms. The input is the video data uploaded to the cloud, and the output is the detection results of dangerous driving vehicles.

[2416] Step 19:

[2417] (Send cloud analysis results)

[2418] The server (cloud) generates a warning message based on the analysis results and sends it to the terminal. The input is the cloud analysis result, and the output is the message data to the terminal.

[2419] Step 20:

[2420] (Warning notice)

[2421] The terminal notifies the driver of warning messages received from the server (cloud) by voice. The input is message data from the cloud, and the output is a voice notification.

[2422] (Application example 1)

[2423] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2424] Conventional driver assistance systems have difficulty providing appropriate rest and alerts based on the driver's level of fatigue or reduced attention span. They also lack real-time warnings and notifications to adapt to surrounding traffic conditions. Self-driving vehicles, in particular, are required to appropriately monitor the driver's condition and respond immediately when necessary. Therefore, there is a need for a new system that comprehensively alerts the driver and notifies them of surrounding traffic conditions, supporting safe driving.

[2425] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2426] In this invention, the server includes means for collecting face information and voice information of the driver, means for analyzing the face information and voice information of the driver to determine the condition, means for providing appropriate actions and warnings by synthetic voice based on the determined condition, and means for notifying the driver of traffic conditions by synthetic voice. This makes it possible to analyze the driver's condition in real time, prompt the driver to take a break or be careful at the appropriate time, and support safe driving by providing real-time notifications according to the surrounding traffic conditions.

[2427] "Driver's facial information" is a general term for visual data related to the driver's face, such as facial expression, gaze position, and whether the eyes are open or closed.

[2428] "Voice information" is a general term for data related to the driver's voice, such as what they say, their tone of voice, volume, and voice patterns.

[2429] "Condition" refers to the psychological and physiological state that affects driving, such as the driver's level of fatigue, attention, and physical condition.

[2430] "Synthetic speech" refers to artificial speech generated by converting text data into speech data.

[2431] "Traffic conditions" refers to the state of the external environment while driving, such as surrounding vehicles, road congestion, and the location of emergency vehicles.

[2432] "Warning" refers to providing information to encourage or warn drivers to take specific actions.

[2433] "Video data" refers to visual digital data collected via in-vehicle cameras, etc., and includes image information about surrounding conditions and objects.

[2434] "Cloud" refers to a group of data centers and servers accessible via the Internet, providing a platform for storing and analyzing data.

[2435] "Dangerous driving vehicles" refer to vehicles that exhibit unusual driving behavior and may pose a threat to the safety of those around them.

[2436] "Providing appropriate actions or warnings using synthetic voice" refers to the act of using synthetic voice to communicate messages generated based on the driver's state.

[2437] "HMD" is an abbreviation for head-mounted display, a display device worn on the user's head that displays visual information.

[2438] The present invention is a system that uses facial information and voice information of the driver to maintain the driver's attention and support safe driving. Specific embodiments for carrying out the present invention are described below.

[2439] Hardware Configuration

[2440] Collecting driver's facial and voice information

[2441] In-vehicle camera: An HD camera is used to collect the driver's facial information, which allows for the acquisition of facial expression and gaze data.

[2442] Microphone: The driver's voice information is collected using an in-vehicle microphone or a microphone built into the head-mounted display (HMD).

[2443] Notifications and Interactions

[2444] Head-mounted display (HMD): Functions as a display and provides visual notifications to the driver. It also provides audio notifications using a built-in speaker.

[2445] Software and Data Processing

[2446] OpenCV: A library for image processing and facial recognition. It analyzes video data from an in-vehicle camera and extracts facial landmarks.

[2447] dlib: A library for detecting facial landmarks. Used to analyze facial expressions and gaze.

[2448] speech_recognition: A library that performs speech recognition and analyzes the driver's voice information.

[2449] pyttsx3: A library that performs speech synthesis and notifies the driver via voice of appropriate warnings and actions.

[2450] System Operation

[2451] 1. Collecting facial and audio information

[2452] The system uses an onboard camera and microphone to collect the driver's facial and voice information in real time. Facial information includes facial expressions and eye gaze position, while voice information includes what the driver says and the tone of their voice.

[2453] 2. Driver status analysis

[2454] The server analyzes the collected facial and voice information to determine the driver's condition, specifically quantifying the driver's level of fatigue and attention from facial expressions, eye movements, and voice tone.

[2455] 3. Traffic situation notifications and alerts

[2456] The system analyzes surrounding video data collected by the in-vehicle camera to detect emergency vehicles and vehicles engaged in reckless driving. This analysis uses image recognition technology. Based on the analysis results, a visual notification is displayed on the HMD and a synthesized voice is used to warn the driver.

[2457] Specific examples

[2458] Example 1: If a driver has been driving for a long time, the server determines that the driver's fatigue level is "high" based on facial information. As a result, a message saying "You need to take a break" is displayed on the HMD and a voice notification is also given.

[2459] Example 2: If the server determines that the driver's attention is declining, it generates a message saying "Please concentrate," displays it on the HMD, and also notifies the driver by voice.

[2460] Example 3: Surrounding traffic conditions are collected from an in-car camera. If an emergency vehicle is approaching, the server analyzes the video data and displays a warning message on the HMD saying, "An emergency vehicle is approaching. Please give way.", and also provides an audio notification.

[2461] Example prompts for generative AI models

[2462] Design a system that analyzes the driver's facial and audio information in real time to quantify their level of fatigue and attention. This system should collect data using an on-board camera and microphone, and based on the analysis results, display a message on the HMD and provide an audio notification to encourage the driver to take appropriate action.

[2463] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2464] Step 1:

[2465] The device uses an onboard camera and microphone to collect the driver's facial and audio information in real time.

[2466] Input: Video data from the dashcam and audio data from the microphone.

[2467] Data processing: Face detection and landmark extraction are performed on video data, and voice recognition is performed on audio data.

[2468] Output: Driver's facial expression data, gaze data, and speech-to-text information.

[2469] Step 2:

[2470] The server analyzes the collected facial and audio information to determine the driver's condition.

[2471] ...

Claims

1. a means for collecting facial and voice information of a driver; a means for analyzing face information and voice information of the driver to determine the condition; means for providing a conversation topic according to the condition; A system including:

2. means for collecting video data from an in-vehicle camera; means for analyzing the video data to analyze surrounding traffic conditions; a means for issuing a warning to a driver based on the traffic conditions; The system of claim 1 , comprising:

3. A means of uploading the collected video data to the cloud, A means for analyzing the video data on the cloud and detecting reckless driving vehicles; A means of notifying the driver of a warning based on the results of cloud analysis, The system of claim 1 , comprising:

Images