System

Abstract

To provide a system.SOLUTION: A system includes: image recognition means for analyzing the face and the movement of eyes of a driver; data analyzing means for analyzing the speed and acceleration of a vehicle; means for monitoring a driving condition of the vehicle based on the analysis result of the data analyzing means; means for detecting drowsiness and reckless driving of the driver based on the analysis result by the image recognition means; means for talking to the driver based on the detected drowsiness and reckless driving; and means for providing proper advice to the driver by using an interactive type AI.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] More than 300,000 traffic accidents occur annually, resulting in more than 2,600 deaths and more than 360,000 injuries. Many of these accidents are caused by driver drowsiness and dangerous driving. [Means for solving the problem]

[0005] This invention provides a drive recorder equipped with conversational AI. This system speaks to the driver when it detects drowsiness or dangerous driving. Furthermore, the conversational AI generates appropriate dialogue based on the driver's response and issues instructions to encourage safe driving. This reduces driver drowsiness and dangerous driving, preventing traffic accidents. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] One form of this invention is a drive recorder equipped with a conversational AI. This drive recorder monitors the vehicle's driving conditions and detects drowsiness or dangerous driving. If this is detected, the conversational AI will speak to the driver. For example, if the driver appears sleepy, the AI ​​will suggest, "Why don't you take a short break?" Furthermore, if dangerous driving is detected, the AI ​​will advise, "Slow down and drive safely."

[0029] "Example 2"

[0030] Furthermore, conversational AI generates appropriate conversations based on the driver's responses. For example, if the driver replies, "I'm still OK," the AI ​​will respond, "However, driving for long periods of time is dangerous. We recommend that you take appropriate breaks." This will help increase the driver's safety awareness.

[0031] "Example 3"

[0032] Conversational AI also provides instructions to encourage safe driving. For example, when there is heavy traffic or bad weather, it will give instructions such as "Pay attention to surrounding vehicles" or "It's raining and visibility is poor, so please reduce your speed" depending on the driving environment. This helps the driver drive safely.

[0033] The processing flow of each embodiment will be described below.

[0034] "Example 1"

[0035] Step 1: A dashcam equipped with conversational AI monitors the vehicle's driving conditions.

[0036] Step 2: The dashcam detects drowsiness and dangerous driving.

[0037] Step 3: If detected, the conversational AI will speak to the driver. For example, if the driver appears sleepy, the AI ​​will suggest, "Would you like to take a short break?"

[0038] "Example 2"

[0039] Step 1: Conversational AI analyzes the driver's responses.

[0040] Step 2: Based on the driver's response, the AI ​​generates an appropriate conversation. For example, if the driver replies, "I'm still OK," the AI ​​will respond, "However, driving for long periods of time is dangerous. We recommend that you take appropriate breaks."

[0041] "Example 3"

[0042] Step 1: Conversational AI analyzes the driving environment.

[0043] Step 2: Depending on the driving environment, the AI ​​will give instructions to encourage the driver to drive safely. For example, if there is heavy traffic or bad weather, the AI ​​will give instructions such as "Pay attention to surrounding vehicles" or "It's raining and visibility is poor, so please reduce your speed."

[0044] Example 1

[0045] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0046] Conventional dashcams simply record video footage and lack the ability to detect driver drowsiness or dangerous driving in real time and provide appropriate advice. This makes it difficult to promote safe driving and increases the risk of accidents.

[0047] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0048] In this invention, the server includes means for monitoring the driving status of the vehicle, means for detecting drowsiness or dangerous driving, means for speaking to the driver based on the detected drowsiness or dangerous driving, image recognition means for analyzing the driver's facial and eye movements, data analysis means for analyzing the vehicle's speed and acceleration, and means for providing appropriate advice to the driver using conversational AI. This makes it possible to promote safe driving by monitoring the driver's condition and driving status in real time and providing appropriate advice.

[0049] "Means for monitoring the driving status of a vehicle" refers to a device that uses cameras and sensors mounted on the vehicle to observe the driver's behavior and the vehicle's movements in real time.

[0050] "Means for detecting drowsiness and dangerous driving" refers to a device that analyzes collected data and identifies driver drowsiness and dangerous driving behavior such as sudden acceleration and braking.

[0051] "Means for speaking to the driver based on detected drowsiness or dangerous driving" refers to a device that provides warnings and advice to the driver through audio and visual means in response to detected abnormalities.

[0052] "Image recognition means for analyzing the driver's face and eye movements" is software that analyzes images captured by a camera and detects whether the driver's eyes are open or closed, the direction of their face, etc.

[0053] "Data analysis means for analyzing vehicle speed and acceleration" refers to software that analyzes speed and acceleration data obtained from sensors installed in the vehicle and evaluates the driving situation.

[0054] "Means for providing appropriate advice to drivers using conversational AI" refers to a device that uses artificial intelligence to generate appropriate advice and warnings based on the driver's condition and driving situation, and provides them via voice.

[0055] MODE FOR CARRYING OUT THE INVENTION

[0056] This invention relates to a drive recorder equipped with a conversational AI, which is a system that monitors the driving conditions of a vehicle and detects drowsiness and dangerous driving. Specific embodiments of this system are described below.

[0057] 1. Program Generation

[0058] The server generates a program for the drive recorder equipped with conversational AI, which monitors the vehicle's driving status and has the ability to detect drowsiness and dangerous driving.

[0059] 2. Program processing explanation

[0060] The terminal (drive recorder) is installed in the vehicle and processes and calculates data using the following hardware and software.

[0061] Hardware:

[0062] Camera: Monitors the driver's face and eye movements.

[0063] Sensors: Measure the vehicle's speed and acceleration.

[0064] Microphone: Recognizes the driver's voice.

[0065] software:

[0066] Image recognition software: Analyzes video data from the camera to detect driver drowsiness.

[0067] Data analysis software: Analyzes data from sensors to detect risky driving.

[0068] Conversational AI (e.g., GPT-4 (registered trademark)): Provides appropriate advice to drivers.

[0069] The specific processing flow is as follows:

[0070] 1. Surveillance:

[0071] Cameras and sensors collect data in real time.

[0072] Image recognition software analyzes the driver's facial and eye movements to detect signs of drowsiness.

[0073] Data analysis software analyzes vehicle speed and acceleration to detect dangerous driving.

[0074] 2. Detection:

[0075] If drowsiness is detected, the conversational AI will ask the driver, "Would you like to take a short break?"

[0076] If dangerous driving is detected, the conversational AI will advise the driver to "slow down and drive safely."

[0077] 3. Examples and prompts

[0078] As a concrete example, the following scenario can be considered.

[0079] Scenario 1: The driver has been driving for a long time and the camera detects that the driver's eyes are closing more frequently.

[0080] Conversational AI: "Would you like to take a break?"

[0081] Scenario 2: The vehicle is repeatedly accelerating and braking suddenly on the highway.

[0082] Conversational AI: "Please slow down and drive safely"

[0083] An example of a prompt sentence to input to the generative AI model is as follows:

[0084] Prompt 1: "How would you talk to a driver if they seemed sleepy?"

[0085] Expected output: "Would you like to take a short break?"

[0086] Prompt 2: "What advice would you give if a vehicle was being driven dangerously?"

[0087] Expected output: "Please slow down and drive safely."

[0088] The above is a form for implementing the invention of a dashcam equipped with conversational AI. This system monitors the driver's condition and driving situation in real time and provides appropriate advice, thereby promoting safe driving.

[0089] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0090] Program processing flow

[0091] Step 1: Data collection

[0092] Input: Real-time data from cameras and sensors onboard the vehicle

[0093] Output: Video data of the driver's face and eyes, vehicle speed and acceleration data

[0094] Specific behavior:

[0095] The device uses the vehicle's on-board camera to capture the driver's facial and eye movements, and sensors to measure the vehicle's speed and acceleration, collecting this data in real time and sending it to the next step.

[0096] Step 2: Data analysis

[0097] Input: Video data and velocity / acceleration data collected in step 1

[0098] Output: Driver drowsiness state, vehicle driving status

[0099] Specific behavior:

[0100] The device uses image recognition software to analyze the driver's face and eye movements, detecting how often their eyes are closed and the direction of their face. At the same time, data analysis software is used to analyze fluctuations in speed and acceleration to detect dangerous driving such as sudden acceleration or braking. The analysis results are sent to the next step as an indication of the driver's drowsiness level and driving situation.

[0101] Step 3: Assess the situation

[0102] Input: Driver drowsiness state and driving situation obtained in Step 2

[0103] Output: Advice for the driver

[0104] Specific behavior:

[0105] The device judges the driver's state and driving situation based on the analysis results. If drowsiness is detected, it flags the driver as "drowsy," and if dangerous driving is detected, it flags the driver as "dangerous driving." This determines the appropriate advice to give to the driver.

[0106] Step 4: Notify the driver

[0107] Input: Advice determined in Step 3

[0108] Output: Voice advice to the driver

[0109] Specific behavior:

[0110] The device uses conversational AI to provide appropriate advice to the driver. If drowsiness is detected, the conversational AI will ask the driver, "Why don't you take a short break?" If dangerous driving is detected, the conversational AI will advise, "Slow down and drive safely." This advice is communicated directly to the driver via the voice output device.

[0111] The above is the specific processing flow of the program of this system.

[0112] (Application example 1)

[0113] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0114] Conventional dashcams only record driving conditions, but lack the ability to detect driver drowsiness or dangerous driving in real time and prompt appropriate action. Another issue is that it is difficult to accurately grasp the driver's condition and provide voice messages at the appropriate time. This results in the problem of being unable to effectively support the driver's safe driving.

[0115] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0116] In this invention, the server is a drive recorder equipped with a conversational AI, and includes means for detecting drowsiness or dangerous driving, means for speaking to the driver based on the detected drowsiness or dangerous driving, means for acquiring camera footage using a smartphone, means for detecting the state of the driver's face and eyes from the acquired camera footage, means for generating a voice message based on the detected state, and means for conveying the generated voice message to the driver. This makes it possible to monitor the driver's state in real time and provide the voice message at an appropriate time.

[0117] "Conversational AI" is artificial intelligence that can converse with humans using natural language processing technology.

[0118] A "drive recorder" is a device that records the driving conditions of a vehicle.

[0119] "Means for detecting drowsiness and dangerous driving" refers to technology that monitors the driver's state and driving behavior and detects drowsiness and dangerous driving in real time.

[0120] "Means of speaking to the driver" refers to technology for conveying audio messages to the driver based on detected information.

[0121] A "smartphone" is a mobile information terminal that has mobile phone functions and is capable of connecting to the Internet and running applications.

[0122] "Means for acquiring camera footage" refers to technology that acquires footage in real time using a smartphone camera.

[0123] "Means for detecting the state of the driver's face and eyes" refers to technology that analyzes the state of the driver's face and eyes from captured camera footage to detect drowsiness or distraction.

[0124] The "means for generating a voice message" is a technology for creating an appropriate voice message based on the detected information.

[0125] The "means for conveying a voice message to the driver" is a technology for making the generated voice message heard by the driver.

[0126] The system for carrying out this invention is mainly composed of a drive recorder equipped with a conversational AI. Specific embodiments are described below.

[0127] System Configuration

[0128] The system consists of the following main components:

[0129] 1. Conversational AI: Artificial intelligence that uses natural language processing technology to converse with the driver.

[0130] 2. Drive recorder: A device that records the driving conditions of a vehicle.

[0131] 3. Smartphone: A mobile information device equipped with a camera and microphone that can connect to the Internet and run applications.

[0132] 4. Camera image acquisition method: Technology that acquires images in real time using a smartphone camera.

[0133] 5. Facial and eye condition detection: Technology that analyzes the driver's facial and eye condition from captured camera footage to detect drowsiness or distraction.

[0134] 6. Voice message generation means: Technology that creates appropriate voice messages based on detected information.

[0135] 7. Voice message delivery means: Technology for delivering generated voice messages to the driver.

[0136] Hardware and Software

[0137] Hardware: Smartphone (camera, microphone), dashcam installed in the vehicle.

[0138] Software: Python, OpenCV, Google® Cloud Text-to-Speech API.

[0139] Data processing and calculation

[0140] 1. Camera image acquisition: Images of the interior of the vehicle are acquired in real time using a smartphone camera. The image data is processed using OpenCV.

[0141] 2. Facial and eye detection: The system analyzes the driver's facial and eye conditions from the acquired video data, thereby detecting driver drowsiness and distraction.

[0142] 3. Generate voice message: Based on the detected information, an appropriate voice message is generated using the Google Cloud Text-to-Speech API.

[0143] 4. Transmitting voice messages: The generated voice messages are transmitted to the driver through the smartphone speaker.

[0144] Specific examples

[0145] For example, if the driver appears drowsy, the system will:

[0146] 1. The smartphone camera captures the driver's face and acquires video data.

[0147] 2. Using OpenCV, the driver's eye opening and closing status is analyzed to detect drowsiness.

[0148] 3. If drowsiness is detected, generate a voice message using the Google Cloud Text-to-Speech API saying, "Would you like to take a break?"

[0149] 4. The generated voice message is transmitted to the driver through the smartphone speaker.

[0150] Prompt Sentence Examples

[0151] "If a driver appears sleepy, how should I talk to them?"

[0152] In this way, the functionality of a dashcam equipped with conversational AI can be applied to a smartphone application, making it possible to monitor the driver's condition in real time and provide voice messages at the appropriate time.

[0153] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0154] Step 1:

[0155] The smartphone camera captures images of the inside of the car in real time. The input is the camera image, and the output is video data. Specifically, the smartphone camera captures images including the driver's face and eye condition.

[0156] Step 2:

[0157] The device processes the captured video data using OpenCV to detect the driver's face and eye status. The input is video data, and the output is the analysis results on the face and eye status. Specifically, a facial recognition algorithm is used to identify the driver's face and analyze whether their eyes are open or closed.

[0158] Step 3:

[0159] The device detects driver drowsiness or distraction based on the analysis results. The input is the analysis results related to the state of the face and eyes, and the output is the detection result of drowsiness or distraction. Specifically, it analyzes the frequency of eye opening and closing and blinking patterns to determine signs of drowsiness.

[0160] Step 4:

[0161] The server uses the Google Cloud Text-to-Speech API to generate an appropriate voice message based on the detection results. The input is the detection result of drowsiness or distraction, and the output is a voice message. Specifically, it generates messages such as "Would you like to take a short break?" or "Please reduce your speed and drive safely."

[0162] Step 5:

[0163] The terminal transmits the generated voice message to the driver through the smartphone speaker. The input is the voice message and the output is the voice output. Specifically, the voice message is played from the smartphone speaker so that the driver can hear it.

[0164] Step 6:

[0165] The user responds to the voice message and takes appropriate action. The input is the voice message, and the output is the driver's action, such as taking a break or slowing down.

[0166] Example 2

[0167] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0168] While conventional dashcams have the ability to detect drowsiness and dangerous driving, they have had the problem of being unable to provide specific advice or warnings to drivers at the appropriate time. Furthermore, they are unable to generate appropriate conversations in real time based on the driver's reactions, limiting their effectiveness in encouraging safe driving.

[0169] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0170] In this invention, the server includes a means for recognizing speech, a means for converting the recognized speech into text data, a means for analyzing the text data and generating an appropriate response, a means for converting the generated response into speech data, and a means for outputting the converted speech data. This makes it possible to provide an appropriate response to a driver's utterances in real time and encourage safe driving.

[0171] A "voice recognition means" is a device or software that captures a user's speech and converts it into digital voice data.

[0172] The "means for converting recognized speech into text data" refers to a device or software that analyzes digital speech data and converts it into corresponding text data.

[0173] The "means for analyzing text data and generating an appropriate response" is an algorithm or software for generating an appropriate response based on the input text data.

[0174] The "means for converting the generated response into voice data" is a device or software for converting text data into voice data.

[0175] The "means for outputting converted audio data" refers to a device or software for playing back audio data through an output device such as a speaker.

[0176] A "generative AI model" is an artificial intelligence model that generates responses in natural language based on input data.

[0177] "Speech synthesis software" is software for converting text data into voice data.

[0178] This invention provides a system that provides an appropriate response to a driver's speech in real time to promote safe driving. The system includes a means for recognizing speech, a means for converting the recognized speech into text data, a means for analyzing the text data and generating an appropriate response, a means for converting the generated response into speech data, and a means for outputting the converted speech data.

[0179] Hardware and software used

[0180] Hardware: On-board computer, microphone, speaker

[0181] Software: Generative AI models (e.g., GPT-4), speech recognition software (e.g., Google Speech-to-Text), and speech synthesis software (e.g., Google Text-to-Speech)

[0182] Program processing

[0183] 1. User speaks:

[0184] The user (driver) says "It's still okay" in the car.

[0185] This speech is captured by a microphone connected to the vehicle's onboard computer.

[0186] 2. The device recognizes your voice:

[0187] The terminal (on-board computer) sends the captured voice data to voice recognition software.

[0188] Speech recognition software converts the voice data into text data.

[0189] For example, the speech "It's still okay" is converted into text "It's still okay."

[0190] 3. The server processes the text data:

[0191] The terminal transmits the converted text data to the server.

[0192] The server inputs the received text data into the generative AI model.

[0193] The generative AI model generates appropriate responses based on the input text.

[0194] For example, if the input text is "You're still okay," the generative AI model will generate the response "However, driving for long periods of time is dangerous. We recommend taking regular breaks."

[0195] 4. The device will speak the response:

[0196] The server sends the generated response text to the terminal.

[0197] The terminal inputs the received response text into the speech synthesis software.

[0198] Speech synthesis software converts text data into voice data.

[0199] The terminal transmits the converted voice data to the user through a speaker.

[0200] For example, a voice may play from the speaker saying, "However, driving for long periods of time is still dangerous. We recommend that you take regular breaks."

[0201] Specific examples

[0202] Prompt Sentence Examples

[0203] Input prompt: "Generate an appropriate response if the driver says, 'I'm still OK.'"

[0204] Generated response: "However, driving for long periods of time is dangerous. We recommend taking regular breaks."

[0205] This system can provide appropriate responses to users' comments in real time, increasing their safety awareness.

[0206] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0207] Step 1:

[0208] The user speaks.

[0209] Input: User (driver) utterance (e.g., "It's still okay").

[0210] Specific action: The user speaks in the car.

[0211] Output: A microphone captures the audio data.

[0212] Step 2:

[0213] The device recognizes the voice.

[0214] Input: The captured audio data.

[0215] Specific operation: The terminal (on-board computer) sends voice data to the voice recognition software.

[0216] Output: Speech recognition software converts the speech data into text (e.g., "It's still okay").

[0217] Step 3:

[0218] The server processes the text data.

[0219] Input: The converted text data (e.g., "It's still okay").

[0220] Specific operation: The device sends text data to the server. The server inputs the text data into the generative AI model. The generative AI model generates an appropriate response.

[0221] Output: The generated response text (e.g., "However, driving for long periods of time is dangerous. We recommend that you take regular breaks.").

[0222] Step 4:

[0223] The device will speak the response.

[0224] Input: Generated response text (e.g., "However, driving for long periods of time is dangerous. We recommend taking regular breaks.").

[0225] Specific operations: The server sends the response text to the device. The device inputs the response text into the speech synthesis software. The speech synthesis software converts the text data into speech data.

[0226] Output: The converted audio data is played through the speaker.

[0227] (Application example 2)

[0228] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0229] Although conventional dashcams have the ability to detect drowsiness and dangerous driving, they lack the ability to generate appropriate dialogue based on the driver's reactions and raise safety awareness, making it difficult to encourage drivers to take breaks at appropriate times and to drive safely.

[0230] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for detecting drowsiness or dangerous driving, means for speaking to the driver based on the detected drowsiness or dangerous driving, means for generating appropriate dialogue based on the driver's response, and means for generating prompt sentences using a generative AI model to enhance the driver's safety awareness. This makes it possible to generate appropriate dialogue in response to the driver's response and enhance the driver's safety awareness.

[0231] "Conversational AI" is an artificial intelligence system that uses natural language processing technology to converse with humans.

[0232] A "drive recorder" is a device that records the driving status of a vehicle and has the ability to record video and audio.

[0233] "Means for detecting drowsiness and dangerous driving" refers to a system that includes sensors and algorithms for monitoring the driver's state and driving behavior and detecting drowsiness and dangerous driving.

[0234] "Means to speak to the driver" refers to a system that includes a speaker or voice synthesis technology to provide audio warnings and advice to the driver based on detected information.

[0235] The "means for generating appropriate dialogue based on the driver's responses" refers to a system that includes algorithms and software for analyzing the driver's responses and actions and generating appropriate dialogue accordingly.

[0236] A "generative AI model" is a model that uses artificial intelligence techniques to generate text and speech, and is particularly used in natural language processing.

[0237] A "prompt" is text that is input to a generative AI model and contains instructions or questions to achieve a specific output.

[0238] "Means to raise driver safety awareness" refers to a system that provides drivers with information and advice to encourage safe driving.

[0239] As an embodiment of the present invention, a drive recorder system equipped with a conversational AI will be described. This system includes means for detecting drowsiness or dangerous driving, means for speaking to the driver based on the detected drowsiness or dangerous driving, means for generating appropriate conversation based on the driver's response, and means for generating prompt sentences using a generative AI model to enhance the driver's safety awareness.

[0240] First, the system uses a vehicle-mounted dashcam, which is equipped with cameras and sensors to monitor the driver's facial expressions and driving behavior, thereby detecting drowsiness and dangerous driving.

[0241] Next, based on the detected information, the system provides the driver with audible warnings and advice. For example, if the driver replies, "I'm still OK," the system might respond, "However, driving for long periods of time is still dangerous. We recommend you take appropriate breaks." In this way, a generative AI model is used to generate appropriate dialogue based on the driver's responses.

[0242] The generative AI model is implemented using the OpenAI (registered trademark) API. This model generates prompts based on the driver's input and generates appropriate responses. Specifically, the following prompts are used:

[0243] Example prompt sentence:

[0244] "If the driver says 'I'm still OK,' generate an appropriate response."

[0245] By inputting this prompt into a generative AI model, the system can generate an appropriate response for the driver, increasing their safety awareness.

[0246] The system is implemented using a smartphone. An application is installed on the smartphone to receive data from the dashcam and generate prompts using the generative AI model. This application is implemented using a Python program and operates by calling OpenAI's API.

[0247] For example, if a driver says, "I'm a little tired," the system will respond, "In that case, I recommend you take a break at the next service area." In this way, the system can generate appropriate conversations based on the driver's reactions, increasing the driver's safety awareness.

[0248] As described above, an embodiment of the present invention uses a drive recorder system equipped with conversational AI to generate appropriate conversations based on the driver's reactions, thereby increasing the driver's safety awareness.

[0249] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0250] Step 1:

[0251] A dashcam uses cameras and sensors inside the vehicle to monitor the driver's facial expressions and driving behavior. The input is real-time data from the cameras and sensors, and the output is data indicating the driver's state (e.g., signs of drowsiness or dangerous driving).

[0252] Step 2:

[0253] The data detected by the drive recorder is analyzed to detect drowsiness and dangerous driving. The input is the output data from Step 1, and the output is the detection result of drowsiness and dangerous driving. Specifically, the system uses image analysis algorithms and machine learning models to evaluate the driver's condition.

[0254] Step 3:

[0255] Based on the detection results, the dashcam provides audio warnings and advice to the driver. The input is the detection results from Step 2, and the output is an audio message for the driver. Specifically, it uses speech synthesis technology to generate an appropriate message and plays it through the speaker.

[0256] Step 4:

[0257] The driver responds to the voice message. The input is the driver's voice response, and the output is the voice data. Specifically, the driver's response is recorded using a microphone.

[0258] Step 5:

[0259] The terminal converts the driver's voice response into text. The input is the voice data from step 4, and the output is text data. Specifically, it uses voice recognition technology to convert the voice into text.

[0260] Step 6:

[0261] The device uses the generative AI model to generate a prompt sentence based on the driver's response. The input is the text data from step 5, and the output is the prompt sentence. Specifically, it generates the following prompt sentence: "If the driver says 'It's still OK,' please generate an appropriate response."

[0262] Step 7:

[0263] The device uses the generative AI model to generate an appropriate response based on the prompt. The input is the prompt from step 6, and the output is the generated response text. Specifically, the device calls the OpenAI API, inputs the prompt into the generative AI model, and obtains a response.

[0264] Step 8:

[0265] The terminal converts the generated response into speech and provides it to the driver. The input is the response text from step 7, and the output is a voice message. Specifically, it uses speech synthesis technology to convert the text into speech and plays it back through the speaker.

[0266] Through these steps, the system can generate appropriate conversations based on the driver's reactions, enhancing the driver's safety awareness.

[0267] Example 3

[0268] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0269] Conventional dashcams and conversational AI systems were limited to detecting driver drowsiness and dangerous driving, and were unable to provide specific safe driving instructions in real time according to the driving environment. This made it difficult to quickly respond to changes in traffic volume, weather, etc. and encourage safe driving.

[0270] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0271] In this invention, the server includes means for collecting information on the driving environment, such as traffic volume data and weather data, means for analyzing the collected data and evaluating the driving environment, means for inputting prompt sentences to the generative AI model based on the analysis results, means for the generative AI model to receive the prompt sentences and generate appropriate driving instructions, means for transmitting the generated instructions to the terminal, and means for the terminal to convey the received instructions to the driver using speech synthesis software and text display software, thereby making it possible to provide specific safe driving instructions according to the driving environment in real time.

[0272] "Traffic volume data" is information about the number and flow of vehicles on roads.

[0273] "Weather data" is information about current weather conditions.

[0274] The "driving environment" refers to external conditions, such as traffic volume and weather, that affect the driver's driving.

[0275] "Means of collection" refers to methods of obtaining data using traffic sensors, weather APIs, etc.

[0276] "Means of analysis" refers to methods for evaluating the driving environment based on collected data.

[0277] A "generative AI model" is an artificial intelligence model that generates appropriate driving instructions based on input prompts.

[0278] A "prompt sentence" is a sentence that is input into a generative AI model to generate instructions.

[0279] "Driving instructions" are specific instructions to encourage the driver to drive safely.

[0280] "Transmitting means" refers to a method for sending the generated driving instructions to the terminal.

[0281] A "terminal" is a device that receives driving instructions sent from the server and conveys them to the driver.

[0282] "Speech synthesis software" is software that converts text into speech and plays it back.

[0283] "Text display software" is software that displays text on a screen.

[0284] MODE FOR CARRYING OUT THE INVENTION

[0285] This invention is a system that collects and analyzes information about the driving environment, such as traffic volume data and weather data, and generates appropriate driving instructions using a generative AI model to communicate them to the driver. Specific embodiments of this system are described below.

[0286] Server Roles

[0287] The server collects information about the driving environment, such as traffic volume data and weather data. Traffic volume data is obtained from traffic sensors, and weather data is obtained using a weather API. After collecting this data, the server analyzes it and evaluates the current driving environment, such as whether traffic volume is heavy or light, whether the weather is sunny or rainy, etc.

[0288] Next, the server inputs prompt sentences into the generative AI model based on the analysis results. Examples of prompt sentences include "Current traffic volume is heavy. Please generate appropriate instructions for the driver" or "Current weather is rainy. Please generate safe driving instructions for the driver."

[0289] The generative AI model receives the prompt and generates appropriate driving instructions, such as "Watch out for surrounding vehicles" or "It's raining and visibility is poor, so please reduce your speed." The generated instructions are then sent from the server to the device.

[0290] Device Role

[0291] The device receives instructions sent from the server. This is done via Wi-Fi or mobile data communication. The device then communicates the received instructions to the driver using speech synthesis software or text display software. For example, speech synthesis software can be used to say, "Pay attention to surrounding vehicles," and text display software can be used to display the same instruction on the screen.

[0292] User (Driver) Role

[0293] The user drives according to the instructions provided by the terminal. For example, if the user receives an instruction such as "It's raining and visibility is poor, so please slow down," the user will drive at a slower speed. In this way, the user can drive safely by following the system's instructions.

[0294] Specific examples

[0295] 1. When there is heavy traffic

[0296] The server obtains and analyzes real-time traffic data from traffic sensors.

[0297] Based on the analysis results, the prompt sentence "Current traffic volume is heavy. Please generate appropriate instructions for the driver" is input into the generative AI model.

[0298] The generative AI model generates the instruction "Pay attention to surrounding vehicles," and the server sends this instruction to the device.

[0299] The device receives instructions and uses speech synthesis software to communicate them aloud to the driver.

[0300] The user follows these instructions and drives while paying attention to surrounding vehicles.

[0301] 2. Bad weather

[0302] The server uses a weather API to obtain and analyze current weather information.

[0303] Based on the analysis results, the prompt sentence "The current weather is rainy. Please generate safe driving instructions for the driver" is input into the generative AI model.

[0304] The generative AI model generates an instruction such as "It's raining and visibility is poor, so please slow down," and the server sends this instruction to the device.

[0305] The terminal receives the instructions and displays them to the driver using text display software.

[0306] The user follows this instruction and drives at a reduced speed.

[0307] In this way, the system provides real-time instructions to assist the driver in safe driving. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0308] Step 1:

[0309] The server collects traffic and weather data from traffic sensors and weather APIs. Specifically, it obtains real-time vehicle count and flow data from traffic sensors and current weather information from weather APIs. These data are used as input data for evaluating the driving environment.

[0310] Step 2:

[0311] The server analyzes the collected traffic volume data and weather data. Specifically, it analyzes the traffic volume data to determine whether the current traffic volume is "heavy," "normal," or "light," and analyzes the weather data to determine the weather condition, such as "sunny," "rainy," or "snowy." The results of this analysis become the basic data for generating prompt sentences to be input into the generative AI model.

[0312] Step 3:

[0313] The server generates prompt sentences to be input into the generative AI model based on the analysis results. For example, it generates prompt sentences such as "Current traffic volume is heavy. Please generate appropriate instructions for the driver" or "Current weather is rainy. Please generate safe driving instructions for the driver." These prompt sentences are input into the generative AI model.

[0314] Step 4:

[0315] The generative AI model receives prompts from the server and generates appropriate driving instructions, such as "Watch out for surrounding vehicles" or "It's raining and visibility is poor, so please reduce your speed." These generated driving instructions are then returned to the server.

[0316] Step 5:

[0317] The server sends the driving instructions returned by the generative AI model to the terminal via real-time communication over the Internet. These driving instructions are then input into the terminal.

[0318] Step 6:

[0319] The device receives driving instructions sent from the server via Wi-Fi or mobile data communication, and the received driving instructions are input into speech synthesis software and text display software.

[0320] Step 7:

[0321] The device then uses voice synthesis software and text display software to communicate the received driving instructions to the driver. Specifically, the device uses voice synthesis software to say, "Pay attention to surrounding vehicles," and text display software to display the same instruction on the screen. This driving instruction is then communicated to the driver.

[0322] Step 8:

[0323] The user (driver) drives according to the driving instructions provided by the terminal. For example, if the user receives an instruction such as "It's raining and visibility is poor, so please slow down," the user will drive at a slower speed. In this way, the user can drive safely by following the system's instructions.

[0324] (Application example 3)

[0325] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0326] Although conventional dashcams have the ability to detect drowsiness and dangerous driving, they are unable to provide real-time safe driving instructions tailored to the driving environment. Furthermore, they lack the ability to learn the driver's driving patterns and provide individually optimized advice. This makes it difficult to effectively support safe driving.

[0327] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server is a drive recorder equipped with a conversational AI, and includes means for detecting drowsiness or dangerous driving, means for speaking to the driver based on the detected drowsiness or dangerous driving, means for collecting environmental data during driving in real time, means for providing the driver with voice instructions for safe driving based on the collected data, and means for learning the driver's driving patterns and providing individually optimized advice. This makes it possible to provide real-time safe driving instructions according to the driving environment and provide individually optimized advice based on the driver's driving patterns.

[0328] "Conversational AI" is an artificial intelligence system that can converse with humans using natural language processing technology.

[0329] A "drive recorder" is a device that records video and audio while a vehicle is in motion.

[0330] "Means for detecting drowsiness and unsafe driving" refers to a system that includes sensors and algorithms to monitor the driver's state and driving behavior and identify drowsiness or unsafe driving behavior.

[0331] "Means for speaking to the driver" refers to an audio output device or software for providing audio warnings and instructions to the driver based on detected information.

[0332] "Means for collecting environmental data in real time while driving" refers to sensors and communication devices for obtaining data on the driving environment, such as weather, traffic volume, and road conditions, in real time.

[0333] "Means for providing voice instructions to the driver on safe driving based on collected data" refers to a system that analyzes collected environmental data and conveys appropriate voice instructions to the driver on safe driving.

[0334] "Means for learning the driver's driving patterns and providing individually optimized advice" refers to machine learning algorithms and software that analyze the driver's driving history and behavioral patterns and provide driving advice that is optimized for each individual driver.

[0335] As an embodiment of the present invention, a drive recorder system equipped with a conversational AI is constructed. A specific embodiment of the system is shown below.

[0336] System Configuration

[0337] The system includes the following major hardware and software components:

[0338] Hardware:

[0339] Drive recorder body

[0340] Sensors (camera, microphone, GPS, accelerometer, etc.)

[0341] Audio output device (speaker)

[0342] Communication modules (Wi-Fi, Bluetooth, 4G / 5G, etc.)

[0343] software:

[0344] Conversational AI (natural language processing engine)

[0345] Data Collection Module

[0346] Data Analysis Module

[0347] Speech generation module (e.g. gTTS)

[0348] Machine Learning Algorithms

[0349] Processing flow

[0350] 1. Data Collection:

[0351] A dashcam collects environmental data (weather, traffic volume, road conditions, etc.) in real time while driving by using cameras and sensors to capture video, audio, and location information.

[0352] 2. Data Analysis:

[0353] The collected data is analyzed by the data analysis module. For example, weather information is obtained from the weather data API, and traffic volume information is obtained from the traffic volume data API.

[0354] 3. Generate safe driving instructions:

[0355] Based on the analysis results, the conversational AI generates safe driving instructions for the driver, such as "It's raining and visibility is poor, so please slow down" or "Traffic is heavy, so please be careful of surrounding vehicles."

[0356] 4. Audio output:

[0357] The generated instructions are converted into voice using a voice generation module and provided to the driver through a speaker.

[0358] 5. Learning driving patterns:

[0359] The machine learning algorithm learns the driver's driving history and behavioral patterns and provides individually optimized advice.

[0360] Specific examples

[0361] For example, if the conversational AI receives information about "rain" from the weather data API, it will generate the instruction, "It's raining and visibility is poor, so please reduce your speed." Similarly, if the AI ​​receives information about "heavy traffic" from the traffic volume data API, it will generate the instruction, "Heavy traffic, so please be careful of surrounding vehicles."

[0362] Prompt Sentence Examples

[0363] Weather data: Rain

[0364] Traffic data: Heavy traffic

[0365] Generated instructions: Rain and poor visibility, please reduce speed. Traffic is heavy, so please be aware of surrounding vehicles.

[0366] In this way, it is possible to provide real-time safe driving instructions tailored to the driving environment and provide personalized advice based on the driver's driving patterns.

[0367] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0368] Step 1:

[0369] The server collects real-time environmental data while driving. Specifically, it uses cameras and sensors to capture video, audio, and location information. Inputs include camera video, microphone audio, GPS data, and acceleration sensor data. This data is collected and sent to the server.

[0370] Step 2:

[0371] The server analyzes the collected data. Specifically, it obtains weather information from the weather data API and traffic volume information from the traffic volume data API. The input includes the environmental data collected in step 1. The server analyzes this data and understands the driving environment. The output is weather information and traffic volume information as the analysis results.

[0372] Step 3:

[0373] The server generates safe driving instructions based on the analysis results. Specifically, the conversational AI generates instructions such as "It's raining and visibility is poor, so please slow down" or "There's a lot of traffic, so please be careful of surrounding vehicles." The input includes the analysis results obtained in step 2. The server generates appropriate instructions based on these analysis results. The output is the generated safe driving instructions.

[0374] Step 4:

[0375] The server converts the generated instructions into voice and provides them to the driver. Specifically, it uses a voice generation module (such as gTTS) to convert the instructions into voice and transmits them to the driver through a speaker. The input includes the instructions generated in step 3. The server converts these instructions into voice data and sends it to an audio output device. The output is voice-based safe driving instructions.

[0376] Step 5:

[0377] The server learns the driver's driving patterns and provides individually optimized advice. Specifically, it analyzes the driver's driving history and behavioral patterns using a machine learning algorithm and generates driving advice optimized for each individual driver. The input includes the driver's driving history data. The server analyzes this data and generates individually optimized advice. The output is individually optimized driving advice.

[0378] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0379] "Example 1"

[0380] One embodiment of the present invention is a system that combines an emotion engine with a dashcam equipped with conversational AI. This system recognizes the driver's emotions and generates appropriate conversations based on those emotions. For example, if the AI ​​senses that the driver is feeling angry, it will suggest, "I recommend you take a short break."

[0381] "Example 2"

[0382] The emotion engine also provides instructions to encourage safe driving based on the driver's emotions. For example, if the driver feels tired, the AI ​​will advise them to "take a moderate break."

[0383] "Example 3"

[0384] Furthermore, the emotion engine generates appropriate conversations depending on the driver's emotions. For example, if the AI ​​senses that the driver is happy, it will praise them by saying, "Great driving." This will increase the driver's motivation to drive and encourage safe driving.

[0385] The processing flow of each embodiment will be described below.

[0386] "Example 1"

[0387] Step 1: A dashcam equipped with conversational AI captures the driver's face with its camera.

[0388] Step 2: The emotion engine recognizes the driver's emotions from the captured video.

[0389] Step 3: Based on the recognized emotion, the AI ​​generates an appropriate conversation and conveys it to the driver.

[0390] "Example 2"

[0391] Step 1: A dashcam equipped with conversational AI captures the driver's face with its camera.

[0392] Step 2: The emotion engine recognizes the driver's emotions from the captured video.

[0393] Step 3: Based on the recognized emotion, the AI ​​generates and conveys safe driving instructions to the driver.

[0394] "Example 3"

[0395] Step 1: A dashcam equipped with conversational AI captures the driver's face with its camera.

[0396] Step 2: The emotion engine recognizes the driver's emotions from the captured video.

[0397] Step 3: Based on the recognized emotion, the AI ​​generates an appropriate conversation and conveys it to the driver.

[0398] Example 1

[0399] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0400] Conventional dashcams are limited to recording driving conditions and providing evidence in the event of an accident, and lack the ability to detect driver drowsiness or dangerous driving in real time and provide appropriate advice. Furthermore, they are unable to provide advice that takes into account the driver's emotional state, making it difficult to effectively promote safe driving.

[0401] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0402] In this invention, the server includes means for collecting driving situation data, means for analyzing the collected data, means for detecting drowsiness or dangerous driving, means for speaking to the driver based on the detected drowsiness or dangerous driving, means for analyzing the driver's emotions, and means for generating appropriate conversation based on the analyzed emotions. This makes it possible to monitor the driver's driving situation in real time and provide appropriate advice when drowsiness or dangerous driving is detected. Furthermore, providing advice that takes the driver's emotional state into consideration can effectively promote safe driving.

[0403] "Driving situation data" refers to information collected while driving a vehicle, such as speed, steering operation, and the driver's facial expressions.

[0404] "Means for collection" refers to devices or methods for acquiring driving situation data using cameras, sensors, etc.

[0405] The "means of analysis" refers to algorithms and software that process collected driving data in real time and detect drowsiness and dangerous driving.

[0406] "Means for detecting drowsiness or dangerous driving" refers to devices or methods that determine signs of driver drowsiness or dangerous driving based on analyzed data.

[0407] "Speech intervention" refers to a device or method that provides audible advice or warnings to the driver based on detected drowsiness or unsafe driving.

[0408] "Means of emotion analysis" refers to algorithms or software that can determine a driver's emotional state from facial expressions, tone of voice, etc.

[0409] The "means for generating appropriate conversation" is a device or method that provides appropriate advice or suggestions to the driver by voice based on the analyzed emotional state.

[0410] This invention is a system that uses a drive recorder equipped with conversational AI to monitor the driver's driving status and detect drowsiness and dangerous driving. Furthermore, by combining it with an emotion engine, it recognizes the driver's emotions and generates appropriate conversations based on those emotions.

[0411] Hardware and software used

[0412] Hardware: Cameras, speed sensors, and steering sensors mounted on the vehicle

[0413] Software: Image processing algorithms, AI algorithms, emotion analysis engines, speech synthesis technology

[0414] Data collection and analysis

[0415] The device collects driving situation data from cameras and sensors installed in the vehicle. Specifically, the camera captures the driver's facial expressions, and the sensors capture data on speed and steering operation. This data is collected in real time and sent to a server.

[0416] The server analyzes the collected data in real time. It uses image processing algorithms to analyze the frequency of the driver's eye movements and the direction of their face, and analyzes driving patterns from sensor data. The analysis results are temporarily stored and used for subsequent processing.

[0417] Detecting drowsiness and dangerous driving

[0418] The device detects drowsiness or dangerous driving based on the analysis results. For example, if the frequency of eye opening and closing exceeds a certain threshold or if there are repeated sudden steering operations, it will determine that the driver is drowsy or driving dangerously. The server records the detection results and generates an alert if necessary.

[0419] Conversational AI advice

[0420] If the device detects drowsiness, it will use conversational AI to ask the driver, "Why don't you take a short break?" Specifically, it uses speech synthesis technology to provide advice in a natural voice. If it detects dangerous driving, it will advise, "Slow down and drive safely."

[0421] Sentiment analysis and appropriate conversation generation

[0422] The server uses an emotion engine to analyze the driver's emotions. Specifically, it senses anger and stress from facial expressions and tone of voice. If the device recognizes that the driver is feeling angry, it uses conversational AI to suggest, "We recommend you take a short break." Based on the emotion analysis results, it generates appropriate advice and communicates it to the driver using voice synthesis technology.

[0423] Specific examples

[0424] For example, if a driver has been driving for a long time and begins to open and close their eyes more frequently, the device will detect drowsiness. At this time, the conversational AI will suggest to the driver, "Why not take a short break?" Also, if the driver makes a sudden steering maneuver, the device will detect dangerous driving and advise, "Slow down and drive safely."

[0425] Prompt Sentence Examples

[0426] Examples of prompts to be input to a generative AI model include:

[0427] Prompt: Describe how a conversational AI might address a driver if they appear sleepy.

[0428]

[0429] Prompt: Explain how a conversational AI using an emotion engine can respond when a driver is angry.

[0430] In this way, the system provides appropriate advice depending on the driver's condition.

[0431] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0432] Step 1: Data collection

[0433] The device collects driving situation data from cameras and sensors installed in the vehicle. Specifically, the camera captures the driver's facial expressions, and the sensors capture data on speed and steering operation.

[0434] Input: Camera footage, speed data, steering operation data

[0435] Output: Collected driving situation data

[0436] Step 2: Data analysis

[0437] The device analyzes the collected data in real time, using image processing algorithms to analyze the frequency of the driver's eye movements and the direction of their face, and then analyzes driving patterns from the sensor data.

[0438] Input: Collected driving situation data

[0439] Output: Analysis results (eye opening / closing frequency, face direction, driving pattern)

[0440] Step 3: Detecting drowsiness and dangerous driving

[0441] The device uses the analysis results to detect drowsiness or dangerous driving. For example, if the frequency of eye opening and closing exceeds a certain threshold or if there are repeated sudden steering operations, it will determine that the driver is drowsy or driving dangerously.

[0442] Input: Analysis results

[0443] Output: Detection results (presence or absence of drowsiness, dangerous driving)

[0444] Step 4: Conversational AI advice

[0445] If the device detects drowsiness, it will use conversational AI to ask the driver, "Why don't you take a short break?" If it detects dangerous driving, it will advise the driver, "Slow down and drive safely."

[0446] Input: Detection result

[0447] Output: Audio advice

[0448] Step 5: Sentiment Analysis

[0449] The server uses an emotion engine to analyze the driver's emotions, specifically detecting anger and stress from facial expressions and tone of voice.

[0450] Input: Driver's facial expression, tone of voice

[0451] Output: Emotion analysis results (anger, stress)

[0452] Step 6: Emotion-aware conversation generation

[0453] If the device recognizes that the driver is feeling angry, it will use conversational AI to suggest, "We recommend that you take a short break." Based on the results of emotion analysis, the device generates appropriate advice and communicates it to the driver using voice synthesis technology.

[0454] Input: Sentiment analysis results

[0455] Output: Audio advice

[0456] In this way, the system provides appropriate advice depending on the driver's condition.

[0457] (Application example 1)

[0458] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0459] Conventional dashcams only record driving conditions, and have the problem of not being able to take appropriate action even if they detect driver drowsiness or dangerous driving. In addition, they provide uniform advice without taking into account the driver's emotions, making it difficult to effectively promote safe driving.

[0460] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server is a drive recorder equipped with a conversational AI, and includes means for detecting drowsiness or dangerous driving, means for speaking to the driver based on the detected drowsiness or dangerous driving, means for recognizing the driver's emotions, and means for generating appropriate conversation based on the recognized emotions. This makes it possible to provide appropriate advice according to the driver's condition and emotions, and promote safe driving.

[0461] "Conversational AI" is an artificial intelligence system that can converse with humans using natural language processing technology.

[0462] A "drive recorder" is a device for recording the driving conditions of a vehicle.

[0463] "Means for detecting drowsiness and dangerous driving" refers to a function that uses sensors and algorithms to detect signs of driver drowsiness or dangerous driving.

[0464] "Means of speaking to the driver" is a function that conveys messages to the driver via voice or text based on detected information.

[0465] "Means for recognizing the driver's emotions" refers to technology that analyzes and recognizes the driver's emotions from their facial expressions, voice, actions, etc.

[0466] "Means for generating appropriate conversation" is a function that generates appropriate conversation for the driver based on the recognized emotions and situation.

[0467] As an embodiment of the present invention, a drive recorder system equipped with a conversational AI will be described. This system is configured using the following hardware and software.

[0468] Hardware

[0469] Camera: Installed inside the vehicle, it captures the driver's face and driving situation in real time.

[0470] Microphone: Collects the driver's voice and uses it for emotion recognition and speech generation.

[0471] Computer: Processes data and runs AI models.

[0472] software

[0473] OpenCV: An image processing library that analyzes images from the camera and performs face detection.

[0474] dlib: A library for face detection and facial feature point extraction.

[0475] transformers: A library for natural language processing and emotion recognition, specifically using emotion recognition models.

[0476] Generative AI model: A model for generating appropriate dialogue based on the driver's emotions and driving situation.

[0477] Data processing and calculation

[0478] 1. Video capture: Capture video from the camera in real time and convert it to grayscale using OpenCV.

[0479] 2. Face detection: Detect the driver's face using dlib and extract the face region.

[0480] 3. Emotion Recognition: Using the transformers library, we analyze the driver's emotions from the extracted facial region.

[0481] 4. Conversation generation: Using a generative AI model, appropriate conversations are generated based on recognized emotions and driving situations.

[0482] Specific examples

[0483] For example, if the driver is feeling angry, the emotion recognition model will detect "anger." Based on this information, the generative AI model will generate a conversation and tell the driver, "We recommend you take a short break."

[0484] Prompt Sentence Examples

[0485] Use the following prompt for the generative AI model:

[0486] If the driver's emotion is "anger," generate appropriate dialogue.

[0487] In this way, it is possible to provide appropriate advice based on the driver's condition and emotions, promoting safe driving.

[0488] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0489] Step 1:

[0490] The server captures video from the camera in real time. The input is video data from the camera, and the output is the captured video frames. Specifically, the server initializes the camera device and acquires video frames.

[0491] Step 2:

[0492] The server converts the captured video frame to grayscale using OpenCV. The input is the captured video frame, and the output is the video frame converted to grayscale. Specifically, it converts color video to grayscale using OpenCV functions.

[0493] Step 3:

[0494] The server uses dlib to detect the driver's face from the grayscale video. The input is a grayscale video frame, and the output is the coordinate information of the detected face area. Specifically, the server uses dlib's face detector to identify the position of the face.

[0495] Step 4:

[0496] The server extracts the detected face region and performs emotion recognition using the transformers library. The input is image data of the face region, and the output is the driver's emotion label (e.g., anger, drowsiness). Specifically, the server inputs the face region into an emotion recognition model and obtains the emotion label.

[0497] Step 5:

[0498] The server inputs prompts to the generative AI model based on the recognized emotion labels to generate appropriate conversations. The inputs are emotion labels and prompts, and the output is the generated conversation text. Specifically, the following prompts are used for the generative AI model:

[0499] If the driver's emotion is "anger," generate appropriate dialogue.

[0500] Step 6:

[0501] The server converts the generated conversation text into speech using a speech synthesis engine and conveys it to the driver. The input is the generated conversation text and the output is voice data. Specifically, the server converts the text into speech using the speech synthesis engine and plays it back through the speaker.

[0502] In this way, it is possible to provide appropriate advice based on the driver's condition and emotions, promoting safe driving.

[0503] Example 2

[0504] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0505] Conventional dashcams simply record video footage and lack the functionality to detect driver drowsiness or dangerous driving and take appropriate action. Furthermore, there are no systems that analyze the driver's emotional state and provide instructions to encourage safe driving based on that analysis. This makes it difficult to raise drivers' safety awareness, and increases the risk of accidents.

[0506] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0507] In this invention, the server includes means for acquiring voice input, means for converting the acquired voice data into text data, means for analyzing the driver's reaction, means for generating appropriate dialogue, means for capturing the driver's facial expressions, means for analyzing the driver's emotional state, and means for issuing instructions to the driver based on the analysis results. This makes it possible to generate appropriate dialogue and instructions based on the driver's reaction and emotional state, thereby increasing the driver's safety awareness.

[0508] "Means for obtaining voice input" refers to devices or technologies for capturing the driver's voice in real time.

[0509] "Means for converting acquired voice data into text data" refers to the process of converting voice data into text information using voice recognition technology.

[0510] "Means for analyzing driver responses" refers to technology that analyzes the driver's words and actions to understand their intentions and state.

[0511] "Means for generating appropriate conversations" refers to technology for generating appropriate responses and advice for the driver based on the analysis results.

[0512] "Means for capturing the driver's facial expressions" refers to technology that uses cameras and sensors to capture the driver's facial expressions in real time.

[0513] "Means for analyzing the driver's emotional state" refers to technology that analyzes acquired facial expression data and determines the driver's emotions and psychological state.

[0514] "Means for issuing instructions to the driver based on the analysis results" refers to technology that generates instructions to encourage the driver to take appropriate action based on the results of emotion analysis.

[0515] MODE FOR CARRYING OUT THE INVENTION

[0516] This invention is a system for supporting safe driving by acquiring voice input, analyzing the driver's reactions and emotional state, and generating appropriate conversations and instructions. Specific embodiments of this system are described below.

[0517] Hardware and software used

[0518] Hardware: On-board computer, microphone, speaker, camera

[0519] Software: Speech recognition software (e.g., Google Speech-to-Text), conversational AI (e.g., OpenAI's GPT-4), emotion recognition software (e.g., Affectiva's emotion recognition software)

[0520] System configuration

[0521] 1. Acquiring voice input:

[0522] The terminal captures the driver's voice through a microphone connected to the vehicle's computer.

[0523] The acquired voice data is sent to voice recognition software in real time and converted into text data.

[0524] 2. Driver reaction analysis:

[0525] The server receives the text data sent from the voice recognition software.

[0526] The server inputs the received text data into a conversational AI and analyzes the driver's response.

[0527] 3. Generate the right conversation:

[0528] Conversational AI generates appropriate dialogue based on the driver's responses.

[0529] The generated conversation is sent back to the server, which then sends it to the terminal.

[0530] 4. Emotion Analysis:

[0531] The device captures the driver's facial expressions through a camera connected to the vehicle's computer.

[0532] The captured facial expression data is sent to emotion recognition software to analyze the driver's emotional state.

[0533] 5. Safe driving instructions:

[0534] The server receives the analysis results sent from the emotion recognition software.

[0535] The server generates appropriate instructions for the driver based on the analysis results and sends them to the terminal.

[0536] Specific examples

[0537] Capture voice input: When the driver says "I'm still OK," the microphone captures the speech and speech recognition software converts it into text: "I'm still OK."

[0538] Driver response analysis: The server sends the text "It's still okay" to the conversational AI, which then analyzes the content.

[0539] Generating an appropriate conversation: The conversational AI generates a response such as, "However, driving for long periods of time is dangerous. We recommend that you take appropriate breaks," and sends it to the device via the server.

[0540] Emotion analysis: The driver's face is captured on camera, and emotion recognition software analyzes their facial expressions and determines they are "tired."

[0541] Safe driving instructions: The server receives the analysis result that the driver is "tired," generates instructions such as "We recommend that you take appropriate breaks," and sends them to the device.

[0542] Prompt Sentence Examples

[0543] "If the driver responds, 'I'm still OK,' generate an appropriate response."

[0544] "If the emotion engine determines that the driver is tired, generate appropriate instructions."

[0545] This system can generate appropriate conversations and instructions based on the driver's reactions and emotional state, increasing the driver's safety awareness.

[0546] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0547] Step 1:

[0548] Acquiring voice input

[0549] The terminal captures the driver's voice through a microphone connected to the vehicle's computer.

[0550] Input: Driver's voice

[0551] Data processing: The acquired voice data is sent to voice recognition software in real time and converted into text data.

[0552] Output: Text data

[0553] What it does: When the driver says "I'm still OK," the microphone captures the audio and speech recognition software converts it into text: "I'm still OK."

[0554] Step 2:

[0555] Driver reaction analysis

[0556] The server receives the text data sent from the voice recognition software.

[0557] Input: Text data

[0558] Data calculation: The server inputs the received text data into the conversational AI and analyzes the driver's response.

[0559] Output: Analysis results

[0560] Specific operation: The server sends the text "It's still okay" to the conversational AI, which then analyzes the content.

[0561] Step 3:

[0562] Generate the right conversations

[0563] Conversational AI generates appropriate dialogue based on the driver's responses.

[0564] Input: Analysis results

[0565] Data calculation: The generated conversation is sent back to the server, which then sends it to the terminal.

[0566] Output: Generated conversation

[0567] Specific operation: The conversational AI generates a response saying, "However, driving for long periods of time is dangerous. We recommend that you take appropriate breaks," and sends it to the device via the server.

[0568] Step 4:

[0569] Emotion Analysis

[0570] The device captures the driver's facial expressions through a camera connected to the vehicle's computer.

[0571] Input: Driver's facial expression data

[0572] Data calculation: The captured facial expression data is sent to emotion recognition software to analyze the driver's emotional state.

[0573] Output: Emotion analysis results

[0574] How it works: The driver's face is captured on camera, and emotion-recognition software analyzes their facial expressions and determines they are "tired."

[0575] Step 5:

[0576] Safe driving instructions

[0577] The server receives the analysis results sent from the emotion recognition software.

[0578] Input: Sentiment analysis results

[0579] Data calculation: Based on the analysis results, appropriate instructions are generated for the driver and sent to the device.

[0580] Output: Safe driving instructions

[0581] Specific operation: The server receives the analysis result that the user is "tired," generates an instruction that says "We recommend that you take a moderate break," and sends it to the device.

[0582] (Application example 2)

[0583] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0584] Although conventional dashcams have the ability to detect drowsiness and dangerous driving, they are unable to generate appropriate conversations or instructions based on the driver's emotions and reactions. This has resulted in a lack of effective communication to raise driver safety awareness. There was also a need for a system that could detect driver fatigue and stress in real time and encourage appropriate rest.

[0585] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0586] In this invention, the server is a drive recorder equipped with a conversational AI, and includes means for detecting drowsiness or dangerous driving, means for speaking to the driver based on the detected drowsiness or dangerous driving, means for recognizing the driver's voice, means for analyzing the driver's emotions, means for generating appropriate conversation based on the driver's emotions, and means for issuing instructions to encourage the driver to drive safely. This makes it possible to generate appropriate conversation and instructions in real time based on the driver's emotions and reactions, thereby increasing the driver's safety awareness.

[0587] "Conversational AI" is an artificial intelligence system that uses natural language processing technology to converse with humans.

[0588] A "drive recorder" is a device that records the driving conditions of a vehicle.

[0589] "Means for detecting drowsiness and dangerous driving" refers to a function that uses sensors and algorithms to detect driver drowsiness and dangerous driving behavior.

[0590] "Means of speaking to the driver" is a function that provides voice warnings and advice to the driver based on detected information.

[0591] "Means for recognizing driver voice" refers to technology that receives the driver's speech as voice input and converts it into text data.

[0592] "Means for analyzing driver emotions" refers to algorithms or models that determine the driver's emotional state from their voice and facial expressions.

[0593] "Means for generating appropriate conversations" refers to the ability to generate natural dialogue based on the driver's emotions and situation.

[0594] "Means for issuing instructions to encourage safe driving" is a function that provides specific instructions and advice to the driver to encourage safe driving.

[0595] A system for implementing this invention uses a drive recorder equipped with conversational AI. The system includes means for detecting drowsiness and dangerous driving, means for speaking to the driver, means for recognizing the driver's voice, means for analyzing the driver's emotions, means for generating appropriate conversations, and means for issuing instructions to encourage safe driving.

[0596] Hardware and Software Configuration

[0597] Hardware: microphones, cameras, smartphones

[0598] Software: Python, SpeechRecognition library, Transformers library

[0599] Processing flow

[0600] 1. Speech recognition: The driver's voice is collected by a microphone and converted into text data using the SpeechRecognition library.

[0601] 2. Sentiment Analysis: The converted text data is analyzed using a sentiment analysis model from the Transformers library to determine the driver's emotional state.

[0602] 3. Conversation generation: Based on the driver's emotional state and text data, appropriate conversations are generated using conversational AI models from the Transformers library.

[0603] 4. Instructions: The generated conversation is provided to the driver as voice output to encourage safe driving.

[0604] Specific examples

[0605] For example, if the driver says, "I'm still OK," the speech recognition means converts this utterance into text data. If the emotion analysis means analyzes this text data and determines that the driver is tired, the conversation generation means generates a conversation saying, "Even so, driving for long periods of time is dangerous. We recommend that you take appropriate breaks." This generated conversation is provided to the driver as a voice output.

[0606] Prompt Sentence Examples

[0607] Driver: It's still okay.

[0608] AI: However, driving for long periods of time is still dangerous, so we recommend taking regular breaks.

[0609] In this way, appropriate conversations and instructions can be generated in real time based on the driver's emotions and reactions, increasing driver safety awareness.

[0610] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0611] Step 1:

[0612] The server uses a microphone to collect the driver's voice. The input is the driver's voice and the output is voice data. This voice data is converted into text data using the SpeechRecognition library. Specifically, the microphone captures the voice and the voice data is sent to the server.

[0613] Step 2:

[0614] The server inputs the converted text data into a sentiment analysis model. The input is text data, and the output is an emotional state (e.g., fatigue, stress, etc.). The sentiment analysis model in the Transformers library is used to analyze the driver's emotions from the text data. Specifically, the text data is passed to the sentiment analysis model, and the emotional state is returned.

[0615] Step 3:

[0616] The server inputs prompts to the conversational AI model based on the emotional state and text data. The input is the emotional state and text data, and the output is the generated conversation. An appropriate conversation is generated using a conversational AI model from the Transformers library. Specifically, the emotional state and text data are passed to the conversational AI model as prompts, and the generated conversation is returned.

[0617] Step 4:

[0618] The server provides the generated conversation to the driver as voice output. The input is the generated conversation, and the output is voice data. Specifically, the generated conversation is passed to a speech synthesis engine, which outputs it as voice data. This voice data is then transmitted to the driver through the speaker.

[0619] Step 5:

[0620] The user (driver) receives the voice output from the server and takes appropriate action. The input is voice data, and the output is the driver's actions. Specifically, the driver listens to the voice instructions and performs actions such as taking a break based on them.

[0621] Example 3

[0622] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0623] Conventional dashcams and conversational AI systems are limited to detecting driver drowsiness and dangerous driving, and are unable to provide appropriate instructions or feedback based on the driving environment and the driver's emotions. Therefore, there is a need for systems that more effectively support safe driving and increase driver motivation.

[0624] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0625] In this invention, the server includes a means for collecting driving data and environmental data, a means for preprocessing the collected data, a means including a generative AI model for generating instructions based on the preprocessed data, a means for analyzing the driver's emotions, and a means for providing the generated instructions and the results of the emotion analysis to the driver, thereby making it possible to provide appropriate instructions and feedback according to the driving environment and the driver's emotions.

[0626] "Driving data" refers to driving-related information collected from vehicle sensors, cameras, GPS, etc.

[0627] "Environmental data" is information related to the driving environment, such as traffic volume and weather.

[0628] "Preprocessing" refers to the process of converting collected data into a format suitable for analysis, such as by removing noise or normalizing the data.

[0629] A "generative AI model" is an artificial intelligence model that generates appropriate instructions based on collected data.

[0630] "Emotion analysis" is a process that analyzes the driver's facial expressions and tone of voice to detect their emotions.

[0631] "Instructions" are driving assistance messages that the generative AI model provides to the driver based on the driving environment.

[0632] "Feedback" refers to providing generated instructions and the results of sentiment analysis to the driver.

[0633] This invention is a system for assisting drivers in safe driving, which collects driving data and environmental data, generates appropriate instructions using a generative AI model, and provides them to the driver. Specific embodiments of this system are described below.

[0634] 1. Generate the system program

[0635] The system generates programs using Python, and the generative AI model is built using machine learning libraries such as TENSORFLOW® and PyTorch. Sentiment analysis uses software such as OpenCV and the Google Cloud Speech-to-Text API.

[0636] 2. Explain the generated program's processing in natural language

[0637] The server collects driving and environmental data from the vehicle's hardware, such as sensors, cameras, and GPS. For example, traffic data is obtained from the vehicle's radar sensor, and weather data is obtained from the vehicle's camera and external weather API.

[0638] The collected data is preprocessed on the server. Specifically, noise removal and data normalization are performed. For example, image data acquired from the camera is denoised using OpenCV, and GPS data undergoes coordinate transformation.

[0639] The preprocessed data is then fed into a generative AI model, which generates instructions based on the driving environment. For example, if there is heavy traffic, the model might generate an instruction such as "Pay attention to surrounding vehicles." This instruction is then communicated to the driver via the in-car display or voice assistant.

[0640] The server also collects the driver's facial expressions and tone of voice from the in-car camera and microphone, and analyzes them in the emotion engine. The emotion engine uses OpenCV and the Google Cloud Speech-to-Text API to detect the driver's emotions in real time. For example, if the driver is smiling, it is determined that they are feeling happy.

[0641] The generated instructions and the results of the emotion analysis are provided to the driver via the device. Specifically, the instructions are displayed on the in-car display, and the voice assistant delivers instructions or words of praise aloud. For example, the voice assistant may deliver words of praise such as, "Great driving!"

[0642] 3. Add examples to your description

[0643] Example 1: Heavy traffic

[0644] The server collects traffic data from vehicle sensors.

[0645] The generative AI model determines that traffic is heavy and generates the instruction "Pay attention to surrounding vehicles."

[0646] These instructions will be displayed on the in-car display and communicated to the driver by the voice assistant.

[0647] Example 2: Bad weather

[0648] The server collects weather data from the vehicle's camera and weather API.

[0649] The generative AI model determines that the weather is bad and generates the instruction, "It's raining and visibility is poor, so please reduce your speed."

[0650] These instructions will be displayed on the in-car display and communicated to the driver by the voice assistant.

[0651] Example 3: When the driver is happy

[0652] The emotion engine analyzes the driver's facial expressions and tone of voice from the in-car camera and microphone.

[0653] If the emotion engine determines that the driver is happy, it generates a compliment such as "Great driving."

[0654] The compliment will be displayed on the in-car display and relayed to the driver by the voice assistant.

[0655] Example prompts to input to the generative AI model

[0656] "What instructions should be given to drivers when traffic is heavy?"

[0657] "What instructions should be given to drivers if the weather is bad?"

[0658] "What praise words should be generated if the driver is happy?"

[0659] In this way, the system supports the driver in safe driving and increases his motivation to drive. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0660] Step 1: Data collection

[0661] The server collects driving and environmental data from hardware such as sensors, cameras, and GPS installed in the vehicle. For example, traffic data is obtained from the vehicle's radar sensor, and weather data is obtained from the vehicle's camera or an external weather API. The input is raw data from the sensors and cameras, and the output is the collected driving and environmental data.

[0662] Step 2: Data Preprocessing

[0663] The server preprocesses the collected data. Specifically, it performs noise removal and data normalization. For example, image data acquired from a camera is denoised using OpenCV, and GPS data is subjected to coordinate transformation. The input is the collected raw data, and the output is the preprocessed data.

[0664] Step 3: Generate instructions

[0665] The server inputs the preprocessed data into a generative AI model. The generative AI model is built using TensorFlow and PyTorch and generates instructions based on the driving environment. For example, if there is heavy traffic, it generates the instruction "Pay attention to surrounding vehicles." The input is the preprocessed data, and the output is the generated instruction.

[0666] Step 4: Sentiment Analysis

[0667] The server collects the driver's facial expressions and tone of voice from the in-car camera and microphone, and analyzes them with the emotion engine. The emotion engine uses OpenCV and the Google Cloud Speech-to-Text API to detect the driver's emotions in real time. For example, if the driver is smiling, it is determined that they are feeling happy. The input is raw data from the camera and microphone, and the output is analyzed emotion data.

[0668] Step 5: Provide feedback

[0669] The device provides the generated instructions and the results of emotion analysis to the driver. Specifically, the instructions are displayed on the in-car display, and the voice assistant speaks instructions or words of praise. For example, the voice assistant may say, "Great driving!" The input is the generated instructions and emotion data, and the output is the feedback provided to the driver.

[0670] (Application example 3)

[0671] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0672] While conventional dashcam systems have the ability to detect drowsiness and dangerous driving, they lack the ability to generate appropriate instructions and conversations based on the driving environment and the driver's emotions. This makes it difficult to effectively support safe driving. Furthermore, they lack the ability to display instructions and conversations in real time via smartphones or head-mounted displays, making it difficult to attract the driver's attention and encourage safe driving.

[0673] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0674] In this invention, the server is a drive recorder equipped with a conversational AI, and includes means for detecting drowsiness and dangerous driving, means for speaking to the driver based on the detected drowsiness and dangerous driving, means for issuing instructions according to the driving environment, means for generating appropriate conversation according to the driver's emotions, and means for displaying the instructions and conversation through a smartphone or a head-mounted display. This makes it possible to provide appropriate instructions and conversation according to the driving environment and emotions of the driver in real time, effectively supporting safe driving.

[0675] "Conversational AI" is an artificial intelligence system that can converse with humans using natural language processing technology.

[0676] A "dashcam" is a device installed in a vehicle that records video and audio while driving.

[0677] "Means for detecting drowsiness and unsafe driving" refers to a system that includes sensors and algorithms to monitor the driver's state and driving behavior and identify drowsiness or unsafe driving behavior.

[0678] "Means of speaking to the driver based on detected drowsiness or dangerous driving" is a function that warns or urges caution to the driver via voice or text based on information on detected drowsiness or dangerous driving.

[0679] "Means for issuing instructions according to the driving environment" refers to a function that analyzes the driving environment, such as traffic conditions and weather, and provides the driver with appropriate driving instructions accordingly.

[0680] "Means for generating appropriate conversation in accordance with the driver's emotions" is a function that senses the driver's emotional state and generates appropriate dialogue content in accordance with that emotion.

[0681] "Means for displaying instructions and conversations via a smartphone or head-mounted display" refers to a function that visually displays instructions and conversation content to the driver in real time using a smartphone or head-mounted display.

[0682] The system for carrying out this invention is centered around a drive recorder equipped with conversational AI. The system includes means for detecting drowsiness or dangerous driving, means for speaking to the driver based on the detected drowsiness or dangerous driving, means for issuing instructions according to the driving environment, means for generating appropriate conversation according to the driver's emotions, and means for displaying instructions and conversation via a smartphone or head-mounted display.

[0683] Program processing explanation

[0684] Hardware and Software

[0685] Hardware: Dashcam, smartphone, head-mounted display, various sensors (camera, microphone, accelerometer, etc.)

[0686] Software: Conversational AI, emotion recognition engine, driving environment analysis algorithm, generative AI model

[0687] Data processing and calculation

[0688] 1. Drowsiness and dangerous driving detection:

[0689] The cameras and sensors installed in the drive recorder monitor the driver's face and driving behavior in real time.

[0690] The collected data is analyzed by algorithms that detect drowsiness and dangerous driving.

[0691] 2. Driving environment analysis:

[0692] It collects GPS data from drive recorders and smartphones, weather information, traffic information, etc., and analyzes the driving environment.

[0693] Based on the analysis results, instructions are generated that are appropriate for the driving environment.

[0694] 3. Emotion recognition:

[0695] The driver's voice and facial expression data are analyzed using an emotion recognition engine to identify the driver's emotional state.

[0696] To generate appropriate conversation content according to emotional state.

[0697] 4. Instructions and dialogue generation:

[0698] Using a generative AI model, instructions and conversations are generated based on the driving environment and the driver's emotions.

[0699] The generated instructions and conversations are displayed to the driver via a smartphone or head-mounted display.

[0700] Specific examples

[0701] For example, if the traffic is heavy while driving and the driver is happy, the system will input the following prompt sentence into the generative AI model:

[0702] Prompt: "The current driving environment is busy and the driver seems happy. Generate appropriate commands and emotional responses."

[0703] Based on this prompt, the generative AI model will generate the following output:

[0704] Instructions: "Watch out for surrounding vehicles."

[0705] Emotional response: "Great driving."

[0706] In this way, it is possible to provide appropriate instructions and conversations in real time that are tailored to the driver's driving environment and emotions, effectively supporting safe driving.

[0707] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0708] Step 1:

[0709] The drive recorder uses cameras and sensors to monitor the driver's face and driving behavior in real time.

[0710] Input: Driver's face image, driving behavior data

[0711] Data processing: A facial recognition algorithm analyzes the driver's eye status and facial expressions to analyze driving behavior data.

[0712] Output: Drowsiness and dangerous driving detection results

[0713] Step 2:

[0714] The server collects GPS data, weather information, and traffic information from smartphones and dashcams.

[0715] Input: GPS data, weather information, traffic information

[0716] Data processing: The driving environment analysis algorithm analyzes the collected data and identifies the current driving environment.

[0717] Output: Analysis results of the driving environment

[0718] Step 3:

[0719] The drive recorder analyzes the driver's voice and facial expression data using an emotion recognition engine.

[0720] Input: Driver's voice data, facial expression data

[0721] Data processing: Voice and facial expression recognition algorithms identify the driver's emotional state.

[0722] Output: Driver's emotional state

[0723] Step 4:

[0724] The server inputs prompt sentences into the generative AI model based on the analysis of the driving environment and the driver's emotional state.

[0725] Input: Analysis results of driving environment, driver's emotional state

[0726] Data processing: Input prompt sentences into the generative AI model to generate appropriate instructions and emotional responses.

[0727] Output: Generated instructions and emotional responses

[0728] Step 5:

[0729] A smartphone or head-mounted display will display the generated instructions and emotional responses to the driver.

[0730] Input: Generated instructions and emotional responses

[0731] Data processing: Converting data into a format suitable for the display device and displaying it visually.

[0732] Output: Display of driver instructions and emotional responses

[0733] In this way, the system can provide appropriate instructions and conversations in real time that are tailored to the driver's driving environment and emotions, effectively supporting safe driving.

[0734] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0735] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0736] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0737] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0738] [Second embodiment]

[0739] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0740] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0741] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32.

[0742] The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0743] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0744] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0745] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0746] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0747] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0748] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0749] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0750] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0751] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0752] "Example 1"

[0753] One form of this invention is a drive recorder equipped with a conversational AI. This drive recorder monitors the vehicle's driving conditions and detects drowsiness or dangerous driving. If this is detected, the conversational AI will speak to the driver. For example, if the driver appears sleepy, the AI ​​will suggest, "Why don't you take a short break?" Furthermore, if dangerous driving is detected, the AI ​​will advise, "Slow down and drive safely."

[0754] "Example 2"

[0755] Furthermore, conversational AI generates appropriate conversations based on the driver's responses. For example, if the driver replies, "I'm still OK," the AI ​​will respond, "However, driving for long periods of time is dangerous. We recommend that you take appropriate breaks." This will help increase the driver's safety awareness.

[0756] "Example 3"

[0757] Conversational AI also provides instructions to encourage safe driving. For example, when there is heavy traffic or bad weather, it will give instructions such as "Pay attention to surrounding vehicles" or "It's raining and visibility is poor, so please reduce your speed" depending on the driving environment. This helps the driver drive safely.

[0758] The processing flow of each embodiment will be described below.

[0759] "Example 1"

[0760] Step 1: A dashcam equipped with conversational AI monitors the vehicle's driving conditions.

[0761] Step 2: The dashcam detects drowsiness and dangerous driving.

[0762] Step 3: If detected, the conversational AI will speak to the driver. For example, if the driver appears sleepy, the AI ​​will suggest, "Would you like to take a short break?"

[0763] "Example 2"

[0764] Step 1: Conversational AI analyzes the driver's responses.

[0765] Step 2: Based on the driver's response, the AI ​​generates an appropriate conversation. For example, if the driver replies, "I'm still OK," the AI ​​will respond, "However, driving for long periods of time is dangerous. We recommend that you take appropriate breaks."

[0766] "Example 3"

[0767] Step 1: Conversational AI analyzes the driving environment.

[0768] Step 2: Depending on the driving environment, the AI ​​will give instructions to encourage the driver to drive safely. For example, if there is heavy traffic or bad weather, the AI ​​will give instructions such as "Pay attention to surrounding vehicles" or "It's raining and visibility is poor, so please reduce your speed."

[0769] Example 1

[0770] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0771] Conventional dashcams simply record video footage and lack the ability to detect driver drowsiness or dangerous driving in real time and provide appropriate advice. This makes it difficult to promote safe driving and increases the risk of accidents.

[0772] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0773] In this invention, the server includes means for monitoring the driving status of the vehicle, means for detecting drowsiness or dangerous driving, means for speaking to the driver based on the detected drowsiness or dangerous driving, image recognition means for analyzing the driver's facial and eye movements, data analysis means for analyzing the vehicle's speed and acceleration, and means for providing appropriate advice to the driver using conversational AI. This makes it possible to promote safe driving by monitoring the driver's condition and driving status in real time and providing appropriate advice.

[0774] "Means for monitoring the driving status of a vehicle" refers to a device that uses cameras and sensors mounted on the vehicle to observe the driver's behavior and the vehicle's movements in real time.

[0775] "Means for detecting drowsiness and dangerous driving" refers to a device that analyzes collected data and identifies driver drowsiness and dangerous driving behavior such as sudden acceleration and braking.

[0776] "Means for speaking to the driver based on detected drowsiness or dangerous driving" refers to a device that provides warnings and advice to the driver through audio and visual means in response to detected abnormalities.

[0777] "Image recognition means for analyzing the driver's face and eye movements" is software that analyzes images captured by a camera and detects whether the driver's eyes are open or closed, the direction of their face, etc.

[0778] "Data analysis means for analyzing vehicle speed and acceleration" refers to software that analyzes speed and acceleration data obtained from sensors installed in the vehicle and evaluates the driving situation.

[0779] "Means for providing appropriate advice to drivers using conversational AI" refers to a device that uses artificial intelligence to generate appropriate advice and warnings based on the driver's condition and driving situation, and provides them via voice.

[0780] MODE FOR CARRYING OUT THE INVENTION

[0781] This invention relates to a drive recorder equipped with a conversational AI, which is a system that monitors the driving conditions of a vehicle and detects drowsiness and dangerous driving. Specific embodiments of this system are described below.

[0782] 1. Program Generation

[0783] The server generates a program for the drive recorder equipped with conversational AI, which monitors the vehicle's driving status and has the ability to detect drowsiness and dangerous driving.

[0784] 2. Program processing explanation

[0785] The terminal (drive recorder) is installed in the vehicle and processes and calculates data using the following hardware and software.

[0786] Hardware:

[0787] Camera: Monitors the driver's face and eye movements.

[0788] Sensors: Measure the vehicle's speed and acceleration.

[0789] Microphone: Recognizes the driver's voice.

[0790] software:

[0791] Image recognition software: Analyzes video data from the camera to detect driver drowsiness.

[0792] Data analysis software: Analyzes data from sensors to detect risky driving.

[0793] Conversational AI (e.g., GPT-4): Providing appropriate advice to drivers.

[0794] The specific processing flow is as follows:

[0795] 1. Surveillance:

[0796] Cameras and sensors collect data in real time.

[0797] Image recognition software analyzes the driver's facial and eye movements to detect signs of drowsiness.

[0798] Data analysis software analyzes vehicle speed and acceleration to detect dangerous driving.

[0799] 2. Detection:

[0800] If drowsiness is detected, the conversational AI will ask the driver, "Would you like to take a short break?"

[0801] If dangerous driving is detected, the conversational AI will advise the driver to "slow down and drive safely."

[0802] 3. Examples and prompts

[0803] As a concrete example, the following scenario can be considered.

[0804] Scenario 1: The driver has been driving for a long time and the camera detects that the driver's eyes are closing more frequently.

[0805] Conversational AI: "Would you like to take a break?"

[0806] Scenario 2: The vehicle is repeatedly accelerating and braking suddenly on the highway.

[0807] Conversational AI: "Please slow down and drive safely"

[0808] An example of a prompt sentence to input to the generative AI model is as follows:

[0809] Prompt 1: "How would you talk to a driver if they seemed sleepy?"

[0810] Expected output: "Would you like to take a short break?"

[0811] Prompt 2: "What advice would you give if a vehicle was being driven dangerously?"

[0812] Expected output: "Please slow down and drive safely."

[0813] The above is a description of the embodiment of the invention of a dashcam equipped with conversational AI. This system monitors the driver's condition and driving situation in real time and provides appropriate advice, thereby promoting safe driving.

[0814] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0815] Program processing flow

[0816] Step 1: Data collection

[0817] Input: Real-time data from cameras and sensors onboard the vehicle

[0818] Output: Video data of the driver's face and eyes, vehicle speed and acceleration data

[0819] Specific behavior:

[0820] The device uses the vehicle's on-board camera to capture the driver's facial and eye movements, and sensors to measure the vehicle's speed and acceleration, collecting this data in real time and sending it to the next step.

[0821] Step 2: Data analysis

[0822] Input: Video data and velocity / acceleration data collected in step 1

[0823] Output: Driver drowsiness state, vehicle driving status

[0824] Specific behavior:

[0825] The device uses image recognition software to analyze the driver's face and eye movements, detecting how often their eyes are closed and the direction of their face. At the same time, data analysis software is used to analyze fluctuations in speed and acceleration to detect dangerous driving such as sudden acceleration or braking. The analysis results are sent to the next step as an indication of the driver's drowsiness level and driving situation.

[0826] Step 3: Assess the situation

[0827] Input: Driver drowsiness state and driving situation obtained in Step 2

[0828] Output: Advice for the driver

[0829] Specific behavior:

[0830] The device judges the driver's condition and driving situation based on the analysis results. If drowsiness is detected, it flags the driver as "drowsy," and if dangerous driving is detected, it flags the driver as "dangerous driving." This determines the appropriate advice to give to the driver.

[0831] Step 4: Notify the driver

[0832] Input: Advice determined in Step 3

[0833] Output: Voice advice to the driver

[0834] Specific behavior:

[0835] The device uses conversational AI to provide appropriate advice to the driver. If drowsiness is detected, the conversational AI will ask the driver, "Why don't you take a short break?" If dangerous driving is detected, the conversational AI will advise, "Slow down and drive safely." This advice is communicated directly to the driver via the voice output device.

[0836] The above is the specific processing flow of the program of this system.

[0837] (Application example 1)

[0838] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0839] Conventional dashcams only record driving conditions, but lack the ability to detect driver drowsiness or dangerous driving in real time and prompt appropriate action. Another issue is that it is difficult to accurately grasp the driver's condition and provide voice messages at the appropriate time. This results in the problem of being unable to effectively support the driver's safe driving.

[0840] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0841] In this invention, the server is a drive recorder equipped with a conversational AI, and includes means for detecting drowsiness or dangerous driving, means for speaking to the driver based on the detected drowsiness or dangerous driving, means for acquiring camera footage using a smartphone, means for detecting the state of the driver's face and eyes from the acquired camera footage, means for generating a voice message based on the detected state, and means for conveying the generated voice message to the driver. This makes it possible to monitor the driver's state in real time and provide the voice message at an appropriate time.

[0842] "Conversational AI" is artificial intelligence that can converse with humans using natural language processing technology.

[0843] A "drive recorder" is a device that records the driving conditions of a vehicle.

[0844] "Means for detecting drowsiness and dangerous driving" refers to technology that monitors the driver's state and driving behavior and detects drowsiness and dangerous driving in real time.

[0845] "Means of speaking to the driver" refers to technology for conveying audio messages to the driver based on detected information.

[0846] A "smartphone" is a mobile information terminal that has mobile phone functions and is capable of connecting to the Internet and running applications.

[0847] "Means for acquiring camera footage" refers to technology that acquires footage in real time using a smartphone camera.

[0848] "Means for detecting the state of the driver's face and eyes" refers to technology that analyzes the state of the driver's face and eyes from captured camera footage to detect drowsiness or distraction.

[0849] The "means for generating a voice message" is a technology for creating an appropriate voice message based on the detected information.

[0850] The "means for conveying a voice message to the driver" is a technology for making the generated voice message heard by the driver.

[0851] The system for carrying out this invention is mainly composed of a drive recorder equipped with a conversational AI. Specific embodiments are described below.

[0852] System Configuration

[0853] The system consists of the following main components:

[0854] 1. Conversational AI: Artificial intelligence that uses natural language processing technology to converse with the driver.

[0855] 2. Drive recorder: A device that records the driving conditions of a vehicle.

[0856] 3. Smartphone: A mobile information device equipped with a camera and microphone that can connect to the Internet and run applications.

[0857] 4. Camera image acquisition method: Technology that acquires images in real time using a smartphone camera.

[0858] 5. Facial and eye condition detection: Technology that analyzes the driver's facial and eye condition from captured camera footage to detect drowsiness or distraction.

[0859] 6. Voice message generation means: Technology that creates appropriate voice messages based on detected information.

[0860] 7. Voice message delivery means: Technology for delivering generated voice messages to the driver.

[0861] Hardware and Software

[0862] Hardware: Smartphone (camera, microphone), dashcam installed in the vehicle.

[0863] Software: Python, OpenCV, Google Cloud Text-to-Speech API.

[0864] Data processing and calculation

[0865] 1. Camera image acquisition: Images of the interior of the vehicle are acquired in real time using a smartphone camera. The image data is processed using OpenCV.

[0866] 2. Facial and eye detection: The system analyzes the driver's facial and eye conditions from the acquired video data, thereby detecting driver drowsiness and distraction.

[0867] 3. Generate voice message: Based on the detected information, an appropriate voice message is generated using the Google Cloud Text-to-Speech API.

[0868] 4. Transmitting voice messages: The generated voice messages are transmitted to the driver through the smartphone speaker.

[0869] Specific examples

[0870] For example, if the driver appears drowsy, the system will:

[0871] 1. The smartphone camera captures the driver's face and acquires video data.

[0872] 2. Using OpenCV, the driver's eye opening and closing status is analyzed to detect drowsiness.

[0873] 3. If drowsiness is detected, generate a voice message using the Google Cloud Text-to-Speech API saying, "Would you like to take a break?"

[0874] 4. The generated voice message is transmitted to the driver through the smartphone speaker.

[0875] Prompt Sentence Examples

[0876] "If a driver appears sleepy, how should I talk to them?"

[0877] In this way, the functionality of a dashcam equipped with conversational AI can be applied to a smartphone application, making it possible to monitor the driver's condition in real time and provide voice messages at the appropriate time.

[0878] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0879] Step 1:

[0880] The smartphone camera captures images of the inside of the car in real time. The input is the camera image, and the output is video data. Specifically, the smartphone camera captures images including the driver's face and eye condition.

[0881] Step 2:

[0882] The device processes the captured video data using OpenCV to detect the driver's face and eye status. The input is video data, and the output is the analysis results on the face and eye status. Specifically, a facial recognition algorithm is used to identify the driver's face and analyze whether their eyes are open or closed.

[0883] Step 3:

[0884] The device detects driver drowsiness or distraction based on the analysis results. The input is the analysis results related to the state of the face and eyes, and the output is the detection result of drowsiness or distraction. Specifically, it analyzes the frequency of eye opening and closing and blinking patterns to determine signs of drowsiness.

[0885] Step 4:

[0886] The server uses the Google Cloud Text-to-Speech API to generate an appropriate voice message based on the detection results. The input is the detection result of drowsiness or distraction, and the output is a voice message. Specifically, it generates messages such as "Would you like to take a short break?" or "Please reduce your speed and drive safely."

[0887] Step 5:

[0888] The terminal transmits the generated voice message to the driver through the smartphone speaker. The input is the voice message and the output is the voice output. Specifically, the voice message is played from the smartphone speaker so that the driver can hear it.

[0889] Step 6:

[0890] The user responds to the voice message and takes appropriate action. The input is the voice message, and the output is the driver's action, such as taking a break or slowing down.

[0891] Example 2

[0892] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0893] While conventional dashcams have the ability to detect drowsiness and dangerous driving, they have had the problem of being unable to provide specific advice or warnings to drivers at the appropriate time. Furthermore, they are unable to generate appropriate conversations in real time based on the driver's reactions, limiting their effectiveness in encouraging safe driving.

[0894] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0895] In this invention, the server includes a means for recognizing speech, a means for converting the recognized speech into text data, a means for analyzing the text data and generating an appropriate response, a means for converting the generated response into speech data, and a means for outputting the converted speech data. This makes it possible to provide an appropriate response to a driver's utterances in real time and encourage safe driving.

[0896] A "voice recognition means" is a device or software that captures a user's speech and converts it into digital voice data.

[0897] The "means for converting recognized speech into text data" refers to a device or software that analyzes digital speech data and converts it into corresponding text data.

[0898] The "means for analyzing text data and generating an appropriate response" is an algorithm or software for generating an appropriate response based on the input text data.

[0899] The "means for converting the generated response into voice data" is a device or software for converting text data into voice data.

[0900] The "means for outputting converted audio data" refers to a device or software for playing back audio data through an output device such as a speaker.

[0901] A "generative AI model" is an artificial intelligence model that generates responses in natural language based on input data.

[0902] "Speech synthesis software" is software for converting text data into voice data.

[0903] This invention provides a system that provides an appropriate response to a driver's speech in real time to promote safe driving. The system includes a means for recognizing speech, a means for converting the recognized speech into text data, a means for analyzing the text data and generating an appropriate response, a means for converting the generated response into speech data, and a means for outputting the converted speech data.

[0904] Hardware and software used

[0905] Hardware: On-board computer, microphone, speaker

[0906] Software: Generative AI models (e.g., GPT-4), speech recognition software (e.g., Google Speech-to-Text), and speech synthesis software (e.g., Google Text-to-Speech)

[0907] Program processing

[0908] 1. User speaks:

[0909] The user (driver) says "It's still okay" in the car.

[0910] This speech is captured by a microphone connected to the vehicle's onboard computer.

[0911] 2. The device recognizes your voice:

[0912] The terminal (on-board computer) sends the captured voice data to voice recognition software.

[0913] Speech recognition software converts the voice data into text data.

[0914] For example, the speech "It's still okay" is converted into text "It's still okay."

[0915] 3. The server processes the text data:

[0916] The terminal transmits the converted text data to the server.

[0917] The server inputs the received text data into the generative AI model.

[0918] The generative AI model generates appropriate responses based on the input text.

[0919] For example, if the input text is "You're still okay," the generative AI model will generate the response "However, driving for long periods of time is dangerous. We recommend taking regular breaks."

[0920] 4. The device will speak the response:

[0921] The server sends the generated response text to the terminal.

[0922] The terminal inputs the received response text into the speech synthesis software.

[0923] Speech synthesis software converts text data into voice data.

[0924] The terminal transmits the converted voice data to the user through a speaker.

[0925] For example, a voice may play from the speaker saying, "However, driving for long periods of time is still dangerous. We recommend that you take regular breaks."

[0926] Specific examples

[0927] Prompt Sentence Examples

[0928] Input prompt: "Generate an appropriate response if the driver says, 'I'm still OK.'"

[0929] Generated response: "However, driving for long periods of time is dangerous. We recommend taking regular breaks."

[0930] This system can provide appropriate responses to users' comments in real time, increasing their safety awareness.

[0931] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0932] Step 1:

[0933] The user speaks.

[0934] Input: User (driver) utterance (e.g., "It's still okay").

[0935] Specific action: The user speaks in the car.

[0936] Output: A microphone captures the audio data.

[0937] Step 2:

[0938] The device recognizes the voice.

[0939] Input: The captured audio data.

[0940] Specific operation: The terminal (on-board computer) sends voice data to the voice recognition software.

[0941] Output: Speech recognition software converts the speech data into text (e.g., "It's still okay").

[0942] Step 3:

[0943] The server processes the text data.

[0944] Input: The converted text data (e.g., "It's still okay").

[0945] Specific operation: The device sends text data to the server. The server inputs the text data into the generative AI model. The generative AI model generates an appropriate response.

[0946] Output: The generated response text (e.g., "However, driving for long periods of time is dangerous. We recommend that you take regular breaks.").

[0947] Step 4:

[0948] The device will speak the response.

[0949] Input: Generated response text (e.g., "However, driving for long periods of time is dangerous. We recommend taking regular breaks.").

[0950] Specific operations: The server sends the response text to the device. The device inputs the response text into the speech synthesis software. The speech synthesis software converts the text data into speech data.

[0951] Output: The converted audio data is played through the speaker.

[0952] (Application example 2)

[0953] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0954] Although conventional dashcams have the ability to detect drowsiness and dangerous driving, they lack the ability to generate appropriate dialogue based on the driver's reactions and raise safety awareness, making it difficult to encourage drivers to take breaks at appropriate times and to drive safely.

[0955] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for detecting drowsiness or dangerous driving, means for speaking to the driver based on the detected drowsiness or dangerous driving, means for generating appropriate dialogue based on the driver's response, and means for generating prompt sentences using a generative AI model to enhance the driver's safety awareness. This makes it possible to generate appropriate dialogue in response to the driver's response and enhance the driver's safety awareness.

[0956] "Conversational AI" is an artificial intelligence system that uses natural language processing technology to converse with humans.

[0957] A "drive recorder" is a device that records the driving status of a vehicle and has the ability to record video and audio.

[0958] "Means for detecting drowsiness and dangerous driving" refers to a system that includes sensors and algorithms for monitoring the driver's state and driving behavior and detecting drowsiness and dangerous driving.

[0959] "Means to speak to the driver" refers to a system that includes a speaker or voice synthesis technology to provide audio warnings and advice to the driver based on detected information.

[0960] The "means for generating appropriate dialogue based on the driver's responses" refers to a system that includes algorithms and software for analyzing the driver's responses and actions and generating appropriate dialogue accordingly.

[0961] A "generative AI model" is a model that uses artificial intelligence technology to generate text and speech, and is particularly used in natural language processing.

[0962] A "prompt" is text that is input to a generative AI model and contains instructions or questions to achieve a specific output.

[0963] "Means to raise driver safety awareness" refers to a system that provides drivers with information and advice to encourage safe driving.

[0964] As an embodiment of the present invention, a drive recorder system equipped with a conversational AI will be described. This system includes means for detecting drowsiness or dangerous driving, means for speaking to the driver based on the detected drowsiness or dangerous driving, means for generating appropriate conversation based on the driver's response, and means for generating prompt sentences using a generative AI model to enhance the driver's safety awareness.

[0965] First, the system uses a vehicle-mounted dashcam, which is equipped with cameras and sensors to monitor the driver's facial expressions and driving behavior, thereby detecting drowsiness and dangerous driving.

[0966] Next, based on the detected information, the system provides the driver with audible warnings and advice. For example, if the driver replies, "I'm still OK," the system might respond, "However, driving for long periods of time is still dangerous. We recommend you take appropriate breaks." In this way, a generative AI model is used to generate appropriate dialogue based on the driver's responses.

[0967] The generative AI model is implemented using OpenAI's API. The model generates prompts based on the driver's input and generates appropriate responses. Specifically, the following prompts are used:

[0968] Example prompt sentence:

[0969] "If the driver says 'I'm still OK,' generate an appropriate response."

[0970] By inputting this prompt into a generative AI model, the system can generate an appropriate response for the driver, increasing their safety awareness.

[0971] The system is implemented using a smartphone. An application is installed on the smartphone to receive data from the dashcam and generate prompts using the generative AI model. This application is implemented using a Python program and operates by calling OpenAI's API.

[0972] For example, if a driver says, "I'm a little tired," the system will respond, "In that case, I recommend you take a break at the next service area." In this way, the system can generate appropriate conversations based on the driver's reactions, increasing the driver's safety awareness.

[0973] As described above, an embodiment of the present invention uses a drive recorder system equipped with conversational AI to generate appropriate conversations based on the driver's reactions, thereby increasing the driver's safety awareness.

[0974] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0975] Step 1:

[0976] A dashcam uses cameras and sensors inside the vehicle to monitor the driver's facial expressions and driving behavior. The input is real-time data from the cameras and sensors, and the output is data indicating the driver's state (e.g., signs of drowsiness or dangerous driving).

[0977] Step 2:

[0978] The data detected by the drive recorder is analyzed to detect drowsiness and dangerous driving. The input is the output data from Step 1, and the output is the detection result of drowsiness and dangerous driving. Specifically, the system uses image analysis algorithms and machine learning models to evaluate the driver's condition.

[0979] Step 3:

[0980] Based on the detection results, the dashcam provides audio warnings and advice to the driver. The input is the detection results from Step 2, and the output is an audio message for the driver. Specifically, it uses speech synthesis technology to generate an appropriate message and plays it through the speaker.

[0981] Step 4:

[0982] The driver responds to the voice message. The input is the driver's voice response, and the output is the voice data. Specifically, the driver's response is recorded using a microphone.

[0983] Step 5:

[0984] The terminal converts the driver's voice response into text. The input is the voice data from step 4, and the output is text data. Specifically, it uses voice recognition technology to convert the voice into text.

[0985] Step 6:

[0986] The device uses the generative AI model to generate a prompt sentence based on the driver's response. The input is the text data from step 5, and the output is the prompt sentence. Specifically, it generates the following prompt sentence: "If the driver says 'It's still OK,' please generate an appropriate response."

[0987] Step 7:

[0988] The device uses the generative AI model to generate an appropriate response based on the prompt. The input is the prompt from step 6, and the output is the generated response text. Specifically, the device calls the OpenAI API, inputs the prompt into the generative AI model, and obtains a response.

[0989] Step 8:

[0990] The terminal converts the generated response into speech and provides it to the driver. The input is the response text from step 7, and the output is a voice message. Specifically, it uses speech synthesis technology to convert the text into speech and plays it back through the speaker.

[0991] Through these steps, the system can generate appropriate conversations based on the driver's reactions, enhancing the driver's safety awareness.

[0992] Example 3

[0993] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0994] Conventional dashcams and conversational AI systems were limited to detecting driver drowsiness and dangerous driving, and were unable to provide specific safe driving instructions in real time according to the driving environment. This made it difficult to quickly respond to changes in traffic volume, weather, etc. and encourage safe driving.

[0995] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0996] In this invention, the server includes means for collecting information on the driving environment, such as traffic volume data and weather data, means for analyzing the collected data and evaluating the driving environment, means for inputting prompt sentences to the generative AI model based on the analysis results, means for the generative AI model to receive the prompt sentences and generate appropriate driving instructions, means for transmitting the generated instructions to the terminal, and means for the terminal to convey the received instructions to the driver using speech synthesis software and text display software, thereby making it possible to provide specific safe driving instructions according to the driving environment in real time.

[0997] "Traffic volume data" is information about the number and flow of vehicles on roads.

[0998] "Weather data" is information about current weather conditions.

[0999] The "driving environment" refers to external conditions, such as traffic volume and weather, that affect the driver's driving.

[1000] "Means of collection" refers to methods of obtaining data using traffic sensors, weather APIs, etc.

[1001] "Means of analysis" refers to methods for evaluating the driving environment based on collected data.

[1002] A "generative AI model" is an artificial intelligence model that generates appropriate driving instructions based on input prompts.

[1003] A "prompt sentence" is a sentence that is input into a generative AI model to generate instructions.

[1004] "Driving instructions" are specific instructions to encourage the driver to drive safely.

[1005] "Transmitting means" refers to a method for sending the generated driving instructions to the terminal.

[1006] A "terminal" is a device that receives driving instructions sent from the server and conveys them to the driver.

[1007] "Speech synthesis software" is software that converts text into speech and plays it back.

[1008] "Text display software" is software that displays text on a screen.

[1009] MODE FOR CARRYING OUT THE INVENTION

[1010] This invention is a system that collects and analyzes information about the driving environment, such as traffic volume data and weather data, and generates appropriate driving instructions using a generative AI model to communicate them to the driver. Specific embodiments of this system are described below.

[1011] Server Roles

[1012] The server collects information about the driving environment, such as traffic volume data and weather data. Traffic volume data is obtained from traffic sensors, and weather data is obtained using a weather API. After collecting this data, the server analyzes it and evaluates the current driving environment, such as whether traffic volume is heavy or light, whether the weather is sunny or rainy, etc.

[1013] Next, the server inputs prompt sentences into the generative AI model based on the analysis results. Examples of prompt sentences include "Current traffic volume is heavy. Please generate appropriate instructions for the driver" or "Current weather is rainy. Please generate safe driving instructions for the driver."

[1014] The generative AI model receives the prompt and generates appropriate driving instructions, such as "Watch out for surrounding vehicles" or "It's raining and visibility is poor, so please reduce your speed." The generated instructions are then sent from the server to the device.

[1015] Device Role

[1016] The device receives instructions sent from the server. This is done via Wi-Fi or mobile data communication. The device then communicates the received instructions to the driver using speech synthesis software or text display software. For example, speech synthesis software can be used to say, "Pay attention to surrounding vehicles," and text display software can be used to display the same instruction on the screen.

[1017] User (Driver) Role

[1018] The user drives according to the instructions provided by the terminal. For example, if the user receives an instruction such as "It's raining and visibility is poor, so please slow down," the user will drive at a slower speed. In this way, the user can drive safely by following the system's instructions.

[1019] Specific examples

[1020] 1. When there is heavy traffic

[1021] The server obtains and analyzes real-time traffic data from traffic sensors.

[1022] Based on the analysis results, a prompt sentence such as "Current traffic volume is heavy. Please generate appropriate instructions for the driver" is input into the generative AI model.

[1023] The generative AI model generates the instruction "Pay attention to surrounding vehicles," and the server sends this instruction to the device.

[1024] The device receives instructions and uses speech synthesis software to communicate them aloud to the driver.

[1025] The user follows these instructions and drives while paying attention to surrounding vehicles.

[1026] 2. Bad weather

[1027] The server uses a weather API to obtain and analyze current weather information.

[1028] Based on the analysis results, the prompt sentence "The current weather is rainy. Please generate safe driving instructions for the driver" is input into the generative AI model.

[1029] The generative AI model generates an instruction such as "It's raining and visibility is poor, so please slow down," and the server sends this instruction to the device.

[1030] The terminal receives the instructions and displays them to the driver using text display software.

[1031] The user follows this instruction and drives at a reduced speed.

[1032] In this way, the system provides real-time instructions to assist the driver in safe driving. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1033] Step 1:

[1034] The server collects traffic and weather data from traffic sensors and weather APIs. Specifically, it obtains real-time vehicle count and flow data from traffic sensors and current weather information from weather APIs. These data are used as input data for evaluating the driving environment.

[1035] Step 2:

[1036] The server analyzes the collected traffic volume data and weather data. Specifically, it analyzes the traffic volume data to determine whether the current traffic volume is "heavy," "normal," or "light," and analyzes the weather data to determine the weather condition, such as "sunny," "rainy," or "snowy." The results of this analysis become the basic data for generating prompt sentences to be input into the generative AI model.

[1037] Step 3:

[1038] The server generates prompt sentences to be input into the generative AI model based on the analysis results. For example, it generates prompt sentences such as "Current traffic volume is heavy. Please generate appropriate instructions for the driver" or "Current weather is rainy. Please generate safe driving instructions for the driver." These prompt sentences are input into the generative AI model.

[1039] Step 4:

[1040] The generative AI model receives prompts from the server and generates appropriate driving instructions, such as "Watch out for surrounding vehicles" or "It's raining and visibility is poor, so please reduce your speed." These generated driving instructions are then returned to the server.

[1041] Step 5:

[1042] The server sends the driving instructions returned by the generative AI model to the terminal via real-time communication over the Internet. These driving instructions are then input into the terminal.

[1043] Step 6:

[1044] The device receives driving instructions sent from the server via Wi-Fi or mobile data communication, and the received driving instructions are input into speech synthesis software and text display software.

[1045] Step 7:

[1046] The device then uses voice synthesis software and text display software to communicate the received driving instructions to the driver. Specifically, the device uses voice synthesis software to say, "Pay attention to surrounding vehicles," and text display software to display the same instruction on the screen. This driving instruction is then communicated to the driver.

[1047] Step 8:

[1048] The user (driver) drives according to the driving instructions provided by the terminal. For example, if the user receives an instruction such as "It's raining and visibility is poor, so please slow down," the user will drive at a slower speed. In this way, the user can drive safely by following the system's instructions.

[1049] (Application example 3)

[1050] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1051] Although conventional dashcams have the ability to detect drowsiness and dangerous driving, they are unable to provide real-time safe driving instructions tailored to the driving environment. Furthermore, they lack the ability to learn the driver's driving patterns and provide individually optimized advice. This makes it difficult to effectively support safe driving.

[1052] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server is a drive recorder equipped with a conversational AI, and includes means for detecting drowsiness or dangerous driving, means for speaking to the driver based on the detected drowsiness or dangerous driving, means for collecting environmental data during driving in real time, means for providing the driver with voice instructions for safe driving based on the collected data, and means for learning the driver's driving patterns and providing individually optimized advice. This makes it possible to provide real-time safe driving instructions according to the driving environment and provide individually optimized advice based on the driver's driving patterns.

[1053] "Conversational AI" is an artificial intelligence system that can converse with humans using natural language processing technology.

[1054] A "drive recorder" is a device that records video and audio while a vehicle is in motion.

[1055] "Means for detecting drowsiness and unsafe driving" refers to a system that includes sensors and algorithms to monitor the driver's state and driving behavior and identify drowsiness or unsafe driving behavior.

[1056] "Means for speaking to the driver" refers to an audio output device or software for providing audio warnings and instructions to the driver based on detected information.

[1057] "Means for collecting environmental data in real time while driving" refers to sensors and communication devices for obtaining data on the driving environment, such as weather, traffic volume, and road conditions, in real time.

[1058] "Means for providing voice instructions to the driver on safe driving based on collected data" refers to a system that analyzes collected environmental data and conveys appropriate voice instructions to the driver on safe driving.

[1059] "Means for learning the driver's driving patterns and providing individually optimized advice" refers to machine learning algorithms and software that analyze the driver's driving history and behavioral patterns and provide driving advice that is optimized for each individual driver.

[1060] As an embodiment of the present invention, a drive recorder system equipped with a conversational AI is constructed. A specific embodiment of the system is shown below.

[1061] System Configuration

[1062] The system includes the following major hardware and software components:

[1063] Hardware:

[1064] Drive recorder body

[1065] Sensors (camera, microphone, GPS, accelerometer, etc.)

[1066] Audio output device (speaker)

[1067] Communication modules (Wi-Fi, Bluetooth, 4G / 5G, etc.)

[1068] software:

[1069] Conversational AI (natural language processing engine)

[1070] Data Collection Module

[1071] Data Analysis Module

[1072] Speech generation module (e.g. gTTS)

[1073] Machine Learning Algorithms

[1074] Processing flow

[1075] 1. Data Collection:

[1076] A dashcam collects real-time environmental data (weather, traffic volume, road conditions, etc.) while driving by using cameras and sensors to capture video, audio, and location information.

[1077] 2. Data Analysis:

[1078] The collected data is analyzed by a data analysis module. For example, weather information is obtained from a weather data API, and traffic volume information is obtained from a traffic volume data API.

[1079] 3. Generate safe driving instructions:

[1080] Based on the analysis results, the conversational AI generates safe driving instructions for the driver, such as "It's raining and visibility is poor, so please slow down" or "Traffic is heavy, so please be careful of surrounding vehicles."

[1081] 4. Audio output:

[1082] The generated instructions are converted into voice using a voice generation module and provided to the driver through a speaker.

[1083] 5. Learning driving patterns:

[1084] The machine learning algorithm learns the driver's driving history and behavioral patterns and provides individually optimized advice.

[1085] Specific examples

[1086] For example, if the conversational AI receives information about "rain" from the weather data API, it will generate the instruction, "It's raining and visibility is poor, so please reduce your speed." Similarly, if the AI ​​receives information about "heavy traffic" from the traffic volume data API, it will generate the instruction, "Heavy traffic, so please be careful of surrounding vehicles."

[1087] Prompt Sentence Examples

[1088] Weather data: Rain

[1089] Traffic data: Heavy traffic

[1090] Generated instructions: Rain and poor visibility, please reduce speed. Traffic is heavy, so please be aware of surrounding vehicles.

[1091] In this way, it is possible to provide real-time safe driving instructions tailored to the driving environment and provide personalized advice based on the driver's driving patterns.

[1092] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1093] Step 1:

[1094] The server collects real-time environmental data while driving. Specifically, it uses cameras and sensors to capture video, audio, and location information. Inputs include camera video, microphone audio, GPS data, and acceleration sensor data. This data is collected and sent to the server.

[1095] Step 2:

[1096] The server analyzes the collected data. Specifically, it obtains weather information from the weather data API and traffic volume information from the traffic volume data API. The input includes the environmental data collected in step 1. The server analyzes this data and understands the driving environment. The output is weather information and traffic volume information as the analysis results.

[1097] Step 3:

[1098] The server generates safe driving instructions based on the analysis results. Specifically, the conversational AI generates instructions such as "It's raining and visibility is poor, so please slow down" or "There's a lot of traffic, so please be careful of surrounding vehicles." The input includes the analysis results obtained in step 2. The server generates appropriate instructions based on these analysis results. The output is the generated safe driving instructions.

[1099] Step 4:

[1100] The server converts the generated instructions into voice and provides them to the driver. Specifically, it uses a voice generation module (such as gTTS) to convert the instructions into voice and transmits them to the driver through a speaker. The input includes the instructions generated in step 3. The server converts these instructions into voice data and sends it to an audio output device. The output is voice-based safe driving instructions.

[1101] Step 5:

[1102] The server learns the driver's driving patterns and provides individually optimized advice. Specifically, it analyzes the driver's driving history and behavioral patterns using a machine learning algorithm and generates driving advice optimized for each individual driver. The input includes the driver's driving history data. The server analyzes this data and generates individually optimized advice. The output is individually optimized driving advice.

[1103] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1104] "Example 1"

[1105] One embodiment of the present invention is a system that combines an emotion engine with a dashcam equipped with conversational AI. This system recognizes the driver's emotions and generates appropriate conversations based on those emotions. For example, if the AI ​​senses that the driver is feeling angry, it will suggest, "I recommend you take a short break."

[1106] "Example 2"

[1107] The emotion engine also provides instructions to encourage safe driving based on the driver's emotions. For example, if the driver feels tired, the AI ​​will advise them to "take a moderate break."

[1108] "Example 3"

[1109] Furthermore, the emotion engine generates appropriate conversations depending on the driver's emotions. For example, if the AI ​​senses that the driver is happy, it will praise them by saying, "Great driving." This will increase the driver's motivation to drive and encourage safe driving.

[1110] The processing flow of each embodiment will be described below.

[1111] "Example 1"

[1112] Step 1: A dashcam equipped with conversational AI captures the driver's face with its camera.

[1113] Step 2: The emotion engine recognizes the driver's emotions from the captured video.

[1114] Step 3: Based on the recognized emotion, the AI ​​generates an appropriate conversation and conveys it to the driver.

[1115] "Example 2"

[1116] Step 1: A dashcam equipped with conversational AI captures the driver's face with its camera.

[1117] Step 2: The emotion engine recognizes the driver's emotions from the captured video.

[1118] Step 3: Based on the recognized emotion, the AI ​​generates and conveys safe driving instructions to the driver.

[1119] "Example 3"

[1120] Step 1: A dashcam equipped with conversational AI captures the driver's face with its camera.

[1121] Step 2: The emotion engine recognizes the driver's emotions from the captured video.

[1122] Step 3: Based on the recognized emotion, the AI ​​generates an appropriate conversation and conveys it to the driver.

[1123] Example 1

[1124] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1125] Conventional dashcams are limited to recording driving conditions and providing evidence in the event of an accident, and lack the ability to detect driver drowsiness or dangerous driving in real time and provide appropriate advice. Furthermore, they are unable to provide advice that takes into account the driver's emotional state, making it difficult to effectively promote safe driving.

[1126] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1127] In this invention, the server includes means for collecting driving situation data, means for analyzing the collected data, means for detecting drowsiness or dangerous driving, means for speaking to the driver based on the detected drowsiness or dangerous driving, means for analyzing the driver's emotions, and means for generating appropriate conversation based on the analyzed emotions. This makes it possible to monitor the driver's driving situation in real time and provide appropriate advice when drowsiness or dangerous driving is detected. Furthermore, providing advice that takes the driver's emotional state into consideration can effectively promote safe driving.

[1128] "Driving situation data" refers to information collected while driving a vehicle, such as speed, steering operation, and the driver's facial expressions.

[1129] "Means for collection" refers to devices or methods for acquiring driving situation data using cameras, sensors, etc.

[1130] The "means of analysis" refers to algorithms and software that process collected driving data in real time and detect drowsiness and dangerous driving.

[1131] "Means for detecting drowsiness or dangerous driving" refers to devices or methods that determine signs of driver drowsiness or dangerous driving based on analyzed data.

[1132] "Speech intervention" refers to a device or method that provides audible advice or warnings to the driver based on detected drowsiness or unsafe driving.

[1133] "Means of emotion analysis" refers to algorithms or software that can determine a driver's emotional state from facial expressions, tone of voice, etc.

[1134] The "means for generating appropriate conversation" is a device or method that provides appropriate advice or suggestions to the driver by voice based on the analyzed emotional state.

[1135] This invention is a system that uses a drive recorder equipped with conversational AI to monitor the driver's driving status and detect drowsiness and dangerous driving. Furthermore, by combining it with an emotion engine, it recognizes the driver's emotions and generates appropriate conversations based on those emotions.

[1136] Hardware and software used

[1137] Hardware: Cameras, speed sensors, and steering sensors mounted on the vehicle

[1138] Software: Image processing algorithms, AI algorithms, emotion analysis engines, speech synthesis technology

[1139] Data collection and analysis

[1140] The device collects driving situation data from cameras and sensors installed in the vehicle. Specifically, the camera captures the driver's facial expressions, and the sensors capture data on speed and steering operation. This data is collected in real time and sent to a server.

[1141] The server analyzes the collected data in real time. It uses image processing algorithms to analyze the frequency of the driver's eye movements and the direction of their face, and analyzes driving patterns from the sensor data. The analysis results are temporarily stored and used for subsequent processing.

[1142] Detecting drowsiness and dangerous driving

[1143] The device detects drowsiness or dangerous driving based on the analysis results. For example, if the frequency of eye opening and closing exceeds a certain threshold or if there are repeated sudden steering operations, it will determine that the driver is drowsy or driving dangerously. The server records the detection results and generates an alert if necessary.

[1144] Conversational AI advice

[1145] If the device detects drowsiness, it will use conversational AI to ask the driver, "Why don't you take a short break?" Specifically, it uses speech synthesis technology to provide advice in a natural voice. If it detects dangerous driving, it will advise, "Slow down and drive safely."

[1146] Sentiment analysis and appropriate conversation generation

[1147] The server uses an emotion engine to analyze the driver's emotions. Specifically, it senses anger and stress from facial expressions and tone of voice. If the device recognizes that the driver is feeling angry, it uses conversational AI to suggest, "We recommend you take a short break." Based on the emotion analysis results, it generates appropriate advice and communicates it to the driver using voice synthesis technology.

[1148] Specific examples

[1149] For example, if a driver has been driving for a long time and begins to open and close their eyes more frequently, the device will detect drowsiness. At this time, the conversational AI will suggest to the driver, "Why not take a short break?" Also, if the driver makes a sudden steering maneuver, the device will detect dangerous driving and advise, "Slow down and drive safely."

[1150] Prompt Sentence Examples

[1151] Examples of prompts to be input to a generative AI model include:

[1152] Prompt: Describe how a conversational AI might address a driver if they appear sleepy.

[1153]

[1154] Prompt: Explain how a conversational AI using an emotion engine can respond when a driver is angry.

[1155] In this way, the system provides appropriate advice depending on the driver's condition.

[1156] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1157] Step 1: Data collection

[1158] The device collects driving situation data from cameras and sensors installed in the vehicle. Specifically, the camera captures the driver's facial expressions, and the sensors capture data on speed and steering operation.

[1159] Input: Camera footage, speed data, steering operation data

[1160] Output: Collected driving situation data

[1161] Step 2: Data analysis

[1162] The device analyzes the collected data in real time, using image processing algorithms to analyze the frequency of the driver's eye movements and the direction of their face, and then analyzes driving patterns from the sensor data.

[1163] Input: Collected driving situation data

[1164] Output: Analysis results (eye opening / closing frequency, face direction, driving pattern)

[1165] Step 3: Detecting drowsiness and dangerous driving

[1166] The device uses the analysis results to detect drowsiness or dangerous driving. For example, if the frequency of eye opening and closing exceeds a certain threshold or if there are repeated sudden steering operations, it will determine that the driver is drowsy or driving dangerously.

[1167] Input: Analysis results

[1168] Output: Detection results (presence or absence of drowsiness, dangerous driving)

[1169] Step 4: Conversational AI advice

[1170] If the device detects drowsiness, it will use conversational AI to ask the driver, "Why don't you take a short break?" If it detects dangerous driving, it will advise the driver, "Slow down and drive safely."

[1171] Input: Detection result

[1172] Output: Audio advice

[1173] Step 5: Sentiment Analysis

[1174] The server uses an emotion engine to analyze the driver's emotions, specifically detecting anger and stress from facial expressions and tone of voice.

[1175] Input: Driver's facial expression, tone of voice

[1176] Output: Emotion analysis results (anger, stress)

[1177] Step 6: Emotion-aware conversation generation

[1178] If the device recognizes that the driver is feeling angry, it will use conversational AI to suggest, "We recommend that you take a short break." Based on the results of emotion analysis, the device generates appropriate advice and conveys it to the driver using voice synthesis technology.

[1179] Input: Sentiment analysis results

[1180] Output: Audio advice

[1181] In this way, the system provides appropriate advice depending on the driver's condition.

[1182] (Application example 1)

[1183] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1184] Conventional dashcams only record driving conditions, and have the problem of not being able to take appropriate action even if they detect driver drowsiness or dangerous driving. In addition, they provide uniform advice without taking into account the driver's emotions, making it difficult to effectively promote safe driving.

[1185] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server is a drive recorder equipped with a conversational AI, and includes means for detecting drowsiness or dangerous driving, means for speaking to the driver based on the detected drowsiness or dangerous driving, means for recognizing the driver's emotions, and means for generating appropriate conversation based on the recognized emotions. This makes it possible to provide appropriate advice according to the driver's condition and emotions, and promote safe driving.

[1186] "Conversational AI" is an artificial intelligence system that can converse with humans using natural language processing technology.

[1187] A "drive recorder" is a device for recording the driving conditions of a vehicle.

[1188] "Means for detecting drowsiness and dangerous driving" refers to a function that uses sensors and algorithms to detect signs of driver drowsiness or dangerous driving.

[1189] "Means of speaking to the driver" is a function that conveys messages to the driver via voice or text based on detected information.

[1190] "Means for recognizing the driver's emotions" refers to technology that analyzes and recognizes the driver's emotions from their facial expressions, voice, actions, etc.

[1191] "Means for generating appropriate conversation" is a function that generates appropriate conversation for the driver based on the recognized emotions and situation.

[1192] As an embodiment of the present invention, a drive recorder system equipped with a conversational AI will be described. This system is configured using the following hardware and software.

[1193] Hardware

[1194] Camera: Installed inside the vehicle, it captures the driver's face and driving situation in real time.

[1195] Microphone: Collects the driver's voice and uses it for emotion recognition and speech generation.

[1196] Computer: Processes data and runs AI models.

[1197] software

[1198] OpenCV: An image processing library that analyzes images from the camera and performs face detection.

[1199] dlib: A library for face detection and facial feature point extraction.

[1200] transformers: A library for natural language processing and emotion recognition, specifically using emotion recognition models.

[1201] Generative AI model: A model for generating appropriate dialogue based on the driver's emotions and driving situation.

[1202] Data processing and calculation

[1203] 1. Video capture: Capture video from the camera in real time and convert it to grayscale using OpenCV.

[1204] 2. Face detection: Detect the driver's face using dlib and extract the face region.

[1205] 3. Emotion Recognition: Using the transformers library, we analyze the driver's emotions from the extracted facial region.

[1206] 4. Conversation generation: Using a generative AI model, appropriate conversations are generated based on recognized emotions and driving situations.

[1207] Specific examples

[1208] For example, if the driver is feeling angry, the emotion recognition model will detect "anger." Based on this information, the generative AI model will generate a conversation and tell the driver, "We recommend you take a short break."

[1209] Prompt Sentence Examples

[1210] Use the following prompt for the generative AI model:

[1211] If the driver's emotion is "anger," generate appropriate dialogue.

[1212] In this way, it is possible to provide appropriate advice based on the driver's condition and emotions, promoting safe driving.

[1213] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1214] Step 1:

[1215] The server captures video from the camera in real time. The input is video data from the camera, and the output is the captured video frames. Specifically, the server initializes the camera device and acquires video frames.

[1216] Step 2:

[1217] The server converts the captured video frame to grayscale using OpenCV. The input is the captured video frame, and the output is the video frame converted to grayscale. Specifically, it converts color video to grayscale using OpenCV functions.

[1218] Step 3:

[1219] The server uses dlib to detect the driver's face from the grayscale video. The input is a grayscale video frame, and the output is the coordinate information of the detected face area. Specifically, the server uses dlib's face detector to identify the position of the face.

[1220] Step 4:

[1221] The server extracts the detected face region and performs emotion recognition using the transformers library. The input is image data of the face region, and the output is the driver's emotion label (e.g., anger, drowsiness). Specifically, the server inputs the face region into an emotion recognition model and obtains the emotion label.

[1222] Step 5:

[1223] The server inputs prompts to the generative AI model based on the recognized emotion labels to generate appropriate conversations. The inputs are emotion labels and prompts, and the output is the generated conversation text. Specifically, the following prompts are used for the generative AI model:

[1224] If the driver's emotion is "anger," generate appropriate dialogue.

[1225] Step 6:

[1226] The server converts the generated conversation text into speech using a speech synthesis engine and conveys it to the driver. The input is the generated conversation text and the output is voice data. Specifically, the server converts the text into speech using the speech synthesis engine and plays it back through the speaker.

[1227] In this way, it is possible to provide appropriate advice based on the driver's condition and emotions, promoting safe driving.

[1228] Example 2

[1229] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1230] Conventional dashcams simply record video footage and lack the functionality to detect driver drowsiness or dangerous driving and take appropriate action. Furthermore, there are no systems that analyze the driver's emotional state and provide instructions to encourage safe driving based on that analysis. This makes it difficult to raise drivers' safety awareness, and increases the risk of accidents.

[1231] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1232] In this invention, the server includes means for acquiring voice input, means for converting the acquired voice data into text data, means for analyzing the driver's reaction, means for generating appropriate dialogue, means for capturing the driver's facial expressions, means for analyzing the driver's emotional state, and means for issuing instructions to the driver based on the analysis results. This makes it possible to generate appropriate dialogue and instructions based on the driver's reaction and emotional state, thereby increasing the driver's safety awareness.

[1233] "Means for obtaining voice input" refers to devices or technologies for capturing the driver's voice in real time.

[1234] "Means for converting acquired voice data into text data" refers to the process of converting voice data into text information using voice recognition technology.

[1235] "Means for analyzing driver responses" refers to technology that analyzes the driver's words and actions to understand their intentions and state.

[1236] "Means for generating appropriate conversation" refers to technology for generating appropriate responses and advice for the driver based on the analysis results.

[1237] "Means for capturing the driver's facial expressions" refers to technology that uses cameras and sensors to capture the driver's facial expressions in real time.

[1238] "Means for analyzing the driver's emotional state" refers to technology that analyzes acquired facial expression data and determines the driver's emotions and psychological state.

[1239] "Means for issuing instructions to the driver based on the analysis results" refers to technology that generates instructions to encourage the driver to take appropriate action based on the results of emotion analysis.

[1240] MODE FOR CARRYING OUT THE INVENTION

[1241] This invention is a system for supporting safe driving by acquiring voice input, analyzing the driver's reactions and emotional state, and generating appropriate conversations and instructions. Specific embodiments of this system are described below.

[1242] Hardware and software used

[1243] Hardware: On-board computer, microphone, speaker, camera

[1244] Software: speech recognition software (e.g., Google Speech-to-Text), conversational AI (e.g., OpenAI's GPT-4), emotion recognition software (e.g., Affectiva's emotion recognition software)

[1245] System configuration

[1246] 1. Acquiring voice input:

[1247] The terminal captures the driver's voice through a microphone connected to the vehicle's computer.

[1248] The acquired voice data is sent to voice recognition software in real time and converted into text data.

[1249] 2. Driver reaction analysis:

[1250] The server receives the text data sent from the voice recognition software.

[1251] The server inputs the received text data into a conversational AI and analyzes the driver's response.

[1252] 3. Generate the right conversation:

[1253] Conversational AI generates appropriate dialogue based on the driver's responses.

[1254] The generated conversation is sent back to the server, which then sends it to the terminal.

[1255] 4. Emotion Analysis:

[1256] The device captures the driver's facial expressions through a camera connected to the vehicle's computer.

[1257] The captured facial expression data is sent to emotion recognition software to analyze the driver's emotional state.

[1258] 5. Safe driving instructions:

[1259] The server receives the analysis results sent from the emotion recognition software.

[1260] The server generates appropriate instructions for the driver based on the analysis results and sends them to the terminal.

[1261] Specific examples

[1262] Capture voice input: When the driver says "I'm still OK," the microphone captures the speech and speech recognition software converts it into text: "I'm still OK."

[1263] Driver response analysis: The server sends the text "It's still okay" to the conversational AI, which then analyzes the content.

[1264] Generating an appropriate conversation: The conversational AI generates a response such as, "However, driving for long periods of time is dangerous. We recommend that you take appropriate breaks," and sends it to the device via the server.

[1265] Emotion analysis: The driver's face is captured on camera, and emotion recognition software analyzes their facial expressions to determine that they are "tired."

[1266] Safe driving instructions: The server receives the analysis result that the driver is "tired," generates instructions such as "We recommend that you take appropriate breaks," and sends them to the device.

[1267] Prompt Sentence Examples

[1268] "If the driver responds, 'I'm still OK,' generate an appropriate response."

[1269] "If the emotion engine determines that the driver is tired, generate appropriate instructions."

[1270] This system can generate appropriate conversations and instructions based on the driver's reactions and emotional state, increasing the driver's safety awareness.

[1271] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1272] Step 1:

[1273] Acquiring voice input

[1274] The terminal captures the driver's voice through a microphone connected to the vehicle's computer.

[1275] Input: Driver's voice

[1276] Data processing: The acquired voice data is sent to voice recognition software in real time and converted into text data.

[1277] Output: Text data

[1278] What it does: When the driver says "I'm still OK," the microphone captures the audio and speech recognition software converts it into text: "I'm still OK."

[1279] Step 2:

[1280] Driver reaction analysis

[1281] The server receives the text data sent from the voice recognition software.

[1282] Input: Text data

[1283] Data calculation: The server inputs the received text data into the conversational AI and analyzes the driver's response.

[1284] Output: Analysis results

[1285] Specific operation: The server sends the text "It's still okay" to the conversational AI, which then analyzes the content.

[1286] Step 3:

[1287] Generate the right conversations

[1288] Conversational AI generates appropriate dialogue based on the driver's responses.

[1289] Input: Analysis results

[1290] Data calculation: The generated conversation is sent back to the server, which then sends it to the terminal.

[1291] Output: Generated conversation

[1292] Specific operation: The conversational AI generates a response saying, "However, driving for long periods of time is dangerous. We recommend that you take appropriate breaks," and sends it to the device via the server.

[1293] Step 4:

[1294] Emotion Analysis

[1295] The device captures the driver's facial expressions through a camera connected to the vehicle's computer.

[1296] Input: Driver's facial expression data

[1297] Data calculation: The captured facial expression data is sent to emotion recognition software to analyze the driver's emotional state.

[1298] Output: Emotion analysis results

[1299] How it works: The driver's face is captured on camera, and emotion-recognition software analyzes their facial expressions and determines they are "tired."

[1300] Step 5:

[1301] Safe driving instructions

[1302] The server receives the analysis results sent from the emotion recognition software.

[1303] Input: Sentiment analysis results

[1304] Data calculation: Based on the analysis results, appropriate instructions are generated for the driver and sent to the device.

[1305] Output: Safe driving instructions

[1306] Specific operation: The server receives the analysis result that the user is "tired," generates an instruction that says "We recommend that you take a moderate break," and sends it to the device.

[1307] (Application example 2)

[1308] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1309] Although conventional dashcams have the ability to detect drowsiness and dangerous driving, they are unable to generate appropriate conversations or instructions based on the driver's emotions and reactions. This has resulted in a lack of effective communication to raise driver safety awareness. There was also a need for a system that could detect driver fatigue and stress in real time and encourage appropriate rest.

[1310] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1311] In this invention, the server is a drive recorder equipped with a conversational AI, and includes means for detecting drowsiness or dangerous driving, means for speaking to the driver based on the detected drowsiness or dangerous driving, means for recognizing the driver's voice, means for analyzing the driver's emotions, means for generating appropriate conversation based on the driver's emotions, and means for issuing instructions to encourage the driver to drive safely. This makes it possible to generate appropriate conversation and instructions in real time based on the driver's emotions and reactions, thereby increasing the driver's safety awareness.

[1312] "Conversational AI" is an artificial intelligence system that uses natural language processing technology to converse with humans.

[1313] A "drive recorder" is a device that records the driving conditions of a vehicle.

[1314] "Means for detecting drowsiness and dangerous driving" refers to a function that uses sensors and algorithms to detect driver drowsiness and dangerous driving behavior.

[1315] "Means of speaking to the driver" is a function that provides voice warnings and advice to the driver based on detected information.

[1316] "Means for recognizing driver voice" refers to technology that receives the driver's speech as voice input and converts it into text data.

[1317] "Means for analyzing driver emotions" refers to algorithms or models that determine the driver's emotional state from their voice and facial expressions.

[1318] "Means for generating appropriate conversations" refers to the ability to generate natural dialogue based on the driver's emotions and situation.

[1319] "Means for issuing instructions to encourage safe driving" is a function that provides specific instructions and advice to the driver to encourage safe driving.

[1320] A system for implementing this invention uses a drive recorder equipped with conversational AI. The system includes means for detecting drowsiness and dangerous driving, means for speaking to the driver, means for recognizing the driver's voice, means for analyzing the driver's emotions, means for generating appropriate conversations, and means for issuing instructions to encourage safe driving.

[1321] Hardware and Software Configuration

[1322] Hardware: microphones, cameras, smartphones

[1323] Software: Python, SpeechRecognition library, Transformers library

[1324] Processing flow

[1325] 1. Speech recognition: The driver's voice is collected by a microphone and converted into text data using the SpeechRecognition library.

[1326] 2. Sentiment Analysis: The converted text data is analyzed using a sentiment analysis model from the Transformers library to determine the driver's emotional state.

[1327] 3. Conversation generation: Based on the driver's emotional state and text data, appropriate conversations are generated using conversational AI models from the Transformers library.

[1328] 4. Instructions: The generated conversation is provided to the driver as voice output to encourage safe driving.

[1329] Specific examples

[1330] For example, if the driver says, "I'm still OK," the speech recognition means converts this utterance into text data. If the emotion analysis means analyzes this text data and determines that the driver is tired, the conversation generation means generates a conversation saying, "Even so, driving for long periods of time is dangerous. We recommend that you take appropriate breaks." This generated conversation is provided to the driver as a voice output.

[1331] Prompt Sentence Examples

[1332] Driver: It's still okay.

[1333] AI: However, driving for long periods of time is still dangerous, so we recommend taking regular breaks.

[1334] In this way, appropriate conversations and instructions can be generated in real time based on the driver's emotions and reactions, increasing driver safety awareness.

[1335] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1336] Step 1:

[1337] The server uses a microphone to collect the driver's voice. The input is the driver's voice and the output is voice data. This voice data is converted into text data using the SpeechRecognition library. Specifically, the microphone captures the voice and the voice data is sent to the server.

[1338] Step 2:

[1339] The server inputs the converted text data into a sentiment analysis model. The input is text data, and the output is an emotional state (e.g., fatigue, stress, etc.). The sentiment analysis model in the Transformers library is used to analyze the driver's emotions from the text data. Specifically, the text data is passed to the sentiment analysis model, and the emotional state is returned.

[1340] Step 3:

[1341] The server inputs prompts to the conversational AI model based on the emotional state and text data. The input is the emotional state and text data, and the output is the generated conversation. An appropriate conversation is generated using a conversational AI model from the Transformers library. Specifically, the emotional state and text data are passed to the conversational AI model as prompts, and the generated conversation is returned.

[1342] Step 4:

[1343] The server provides the generated conversation to the driver as voice output. The input is the generated conversation, and the output is voice data. Specifically, the generated conversation is passed to a speech synthesis engine, which outputs it as voice data. This voice data is then transmitted to the driver through the speaker.

[1344] Step 5:

[1345] The user (driver) receives the voice output from the server and takes appropriate action. The input is voice data, and the output is the driver's actions. Specifically, the driver listens to the voice instructions and performs actions such as taking a break based on them.

[1346] Example 3

[1347] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1348] Conventional dashcams and conversational AI systems are limited to detecting driver drowsiness and dangerous driving, and are unable to provide appropriate instructions or feedback based on the driving environment and the driver's emotions. Therefore, there is a need for systems that more effectively support safe driving and increase driver motivation.

[1349] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1350] In this invention, the server includes a means for collecting driving data and environmental data, a means for preprocessing the collected data, a means including a generative AI model for generating instructions based on the preprocessed data, a means for analyzing the driver's emotions, and a means for providing the generated instructions and the results of the emotion analysis to the driver, thereby making it possible to provide appropriate instructions and feedback according to the driving environment and the driver's emotions.

[1351] "Driving data" refers to driving-related information collected from vehicle sensors, cameras, GPS, etc.

[1352] "Environmental data" is information related to the driving environment, such as traffic volume and weather.

[1353] "Preprocessing" refers to the process of converting collected data into a format suitable for analysis, such as by removing noise or normalizing the data.

[1354] A "generative AI model" is an artificial intelligence model that generates appropriate instructions based on collected data.

[1355] "Emotion analysis" is a process that analyzes the driver's facial expressions and tone of voice to detect their emotions.

[1356] "Instructions" are driving assistance messages that the generative AI model provides to the driver based on the driving environment.

[1357] "Feedback" refers to providing generated instructions and the results of sentiment analysis to the driver.

[1358] This invention is a system for assisting drivers in safe driving, which collects driving data and environmental data, generates appropriate instructions using a generative AI model, and provides them to the driver. Specific embodiments of this system are described below.

[1359] 1. Generate the system program

[1360] The system generates programs using Python, and the generative AI model is built using machine learning libraries such as TensorFlow and PyTorch. For sentiment analysis, it uses software such as OpenCV and the Google Cloud Speech-to-Text API.

[1361] 2. Explain the generated program's processing in natural language

[1362] The server collects driving and environmental data from the vehicle's hardware, such as sensors, cameras, and GPS. For example, traffic data is obtained from the vehicle's radar sensor, and weather data is obtained from the vehicle's camera and external weather API.

[1363] The collected data is preprocessed on the server. Specifically, noise removal and data normalization are performed. For example, image data acquired from the camera is denoised using OpenCV, and GPS data undergoes coordinate transformation.

[1364] The preprocessed data is then fed into a generative AI model, which generates instructions based on the driving environment. For example, if there is heavy traffic, the model might generate an instruction such as "Pay attention to surrounding vehicles." This instruction is then communicated to the driver via the in-car display or voice assistant.

[1365] The server also collects the driver's facial expressions and tone of voice from the in-car camera and microphone, and analyzes them in the emotion engine. The emotion engine uses OpenCV and the Google Cloud Speech-to-Text API to detect the driver's emotions in real time. For example, if the driver is smiling, it is determined that they are feeling happy.

[1366] The generated instructions and the results of the emotion analysis are provided to the driver via the device. Specifically, the instructions are displayed on the in-car display, and the voice assistant delivers instructions or words of praise aloud. For example, the voice assistant may deliver words of praise such as, "Great driving!"

[1367] 3. Add examples to your description

[1368] Example 1: Heavy traffic

[1369] The server collects traffic data from vehicle sensors.

[1370] The generative AI model determines that traffic is heavy and generates the instruction "Pay attention to surrounding vehicles."

[1371] These instructions will be displayed on the in-car display and communicated to the driver by the voice assistant.

[1372] Example 2: Bad weather

[1373] The server collects weather data from the vehicle's camera and weather API.

[1374] The generative AI model determines that the weather is bad and generates the instruction, "It's raining and visibility is poor, so please reduce your speed."

[1375] These instructions will be displayed on the in-car display and communicated to the driver by the voice assistant.

[1376] Example 3: When the driver is happy

[1377] The emotion engine analyzes the driver's facial expressions and tone of voice from the in-car camera and microphone.

[1378] If the emotion engine determines that the driver is happy, it generates a compliment such as "Great driving."

[1379] The compliment will be displayed on the in-car display and relayed to the driver by the voice assistant.

[1380] Example prompts to input to the generative AI model

[1381] "What instructions should be given to drivers when traffic is heavy?"

[1382] "What instructions should be given to drivers if the weather is bad?"

[1383] "What praise words should be generated if the driver is happy?"

[1384] In this way, the system supports the driver in safe driving and increases his motivation to drive. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1385] Step 1: Data collection

[1386] The server collects driving and environmental data from hardware such as sensors, cameras, and GPS installed in the vehicle. For example, traffic data is obtained from the vehicle's radar sensor, and weather data is obtained from the vehicle's camera or an external weather API. The input is raw data from the sensors and cameras, and the output is the collected driving and environmental data.

[1387] Step 2: Data Preprocessing

[1388] The server preprocesses the collected data. Specifically, it performs noise removal and data normalization. For example, image data acquired from a camera is denoised using OpenCV, and GPS data is subjected to coordinate transformation. The input is the collected raw data, and the output is the preprocessed data.

[1389] Step 3: Generate instructions

[1390] The server inputs the preprocessed data into a generative AI model. The generative AI model is built using TensorFlow and PyTorch and generates instructions based on the driving environment. For example, if there is heavy traffic, it generates the instruction "Pay attention to surrounding vehicles." The input is the preprocessed data, and the output is the generated instruction.

[1391] Step 4: Sentiment Analysis

[1392] The server collects the driver's facial expressions and tone of voice from the in-car camera and microphone, and analyzes them with the emotion engine. The emotion engine uses OpenCV and the Google Cloud Speech-to-Text API to detect the driver's emotions in real time. For example, if the driver is smiling, it is determined that they are feeling happy. The input is raw data from the camera and microphone, and the output is analyzed emotion data.

[1393] Step 5: Provide feedback

[1394] The device provides the generated instructions and the results of emotion analysis to the driver. Specifically, the instructions are displayed on the in-car display, and the voice assistant speaks instructions or words of praise. For example, the voice assistant may say, "Great driving!" The input is the generated instructions and emotion data, and the output is the feedback provided to the driver.

[1395] (Application example 3)

[1396] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1397] While conventional dashcam systems have the ability to detect drowsiness and dangerous driving, they lack the ability to generate appropriate instructions and conversations based on the driving environment and the driver's emotions. This makes it difficult to effectively support safe driving. Furthermore, they lack the ability to display instructions and conversations in real time via smartphones or head-mounted displays, making it difficult to attract the driver's attention and encourage safe driving.

[1398] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1399] In this invention, the server is a drive recorder equipped with a conversational AI, and includes means for detecting drowsiness and dangerous driving, means for speaking to the driver based on the detected drowsiness and dangerous driving, means for issuing instructions according to the driving environment, means for generating appropriate conversation according to the driver's emotions, and means for displaying the instructions and conversation through a smartphone or a head-mounted display. This makes it possible to provide appropriate instructions and conversation according to the driving environment and emotions of the driver in real time, effectively supporting safe driving.

[1400] "Conversational AI" is an artificial intelligence system that can converse with humans using natural language processing technology.

[1401] A "dashcam" is a device installed in a vehicle that records video and audio while driving.

[1402] "Means for detecting drowsiness and unsafe driving" refers to a system that includes sensors and algorithms to monitor the driver's state and driving behavior and identify drowsiness or unsafe driving behavior.

[1403] "Means of speaking to the driver based on detected drowsiness or dangerous driving" is a function that warns or urges caution to the driver via voice or text based on information on detected drowsiness or dangerous driving.

[1404] "Means for issuing instructions according to the driving environment" refers to a function that analyzes the driving environment, such as traffic conditions and weather, and provides the driver with appropriate driving instructions accordingly.

[1405] "Means for generating appropriate conversation in accordance with the driver's emotions" is a function that senses the driver's emotional state and generates appropriate dialogue content in accordance with that emotion.

[1406] "Means for displaying instructions and conversations via a smartphone or head-mounted display" refers to a function that visually displays instructions and conversation content to the driver in real time using a smartphone or head-mounted display.

[1407] The system for carrying out this invention is centered around a drive recorder equipped with conversational AI. The system includes means for detecting drowsiness or dangerous driving, means for speaking to the driver based on the detected drowsiness or dangerous driving, means for issuing instructions according to the driving environment, means for generating appropriate conversation according to the driver's emotions, and means for displaying instructions and conversation via a smartphone or head-mounted display.

[1408] Program processing explanation

[1409] Hardware and Software

[1410] Hardware: Dashcam, smartphone, head-mounted display, various sensors (camera, microphone, accelerometer, etc.)

[1411] Software: Conversational AI, emotion recognition engine, driving environment analysis algorithm, generative AI model

[1412] Data processing and calculation

[1413] 1. Drowsiness and dangerous driving detection:

[1414] The cameras and sensors installed in the drive recorder monitor the driver's face and driving behavior in real time.

[1415] The collected data is analyzed by algorithms that detect drowsiness and dangerous driving.

[1416] 2. Driving environment analysis:

[1417] It collects GPS data from drive recorders and smartphones, weather information, traffic information, etc., and analyzes the driving environment.

[1418] Based on the analysis results, instructions are generated that are appropriate for the driving environment.

[1419] 3. Emotion recognition:

[1420] The driver's voice and facial expression data are analyzed using an emotion recognition engine to identify the driver's emotional state.

[1421] To generate appropriate conversation content according to emotional state.

[1422] 4. Instructions and dialogue generation:

[1423] Using a generative AI model, instructions and conversations are generated based on the driving environment and the driver's emotions.

[1424] The generated instructions and conversations are displayed to the driver via a smartphone or head-mounted display.

[1425] Specific examples

[1426] For example, if the traffic is heavy while driving and the driver is happy, the system will input the following prompt sentence into the generative AI model:

[1427] Prompt: "The current driving environment is busy and the driver seems happy. Generate appropriate commands and emotional responses."

[1428] Based on this prompt, the generative AI model will generate the following output:

[1429] Instructions: "Watch out for surrounding vehicles."

[1430] Emotional response: "Great driving."

[1431] In this way, it is possible to provide appropriate instructions and conversations in real time that are tailored to the driver's driving environment and emotions, effectively supporting safe driving.

[1432] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1433] Step 1:

[1434] The drive recorder uses cameras and sensors to monitor the driver's face and driving behavior in real time.

[1435] Input: Driver's face image, driving behavior data

[1436] Data processing: A facial recognition algorithm analyzes the driver's eye status and facial expressions to analyze driving behavior data.

[1437] Output: Drowsiness and dangerous driving detection results

[1438] Step 2:

[1439] The server collects GPS data, weather information, and traffic information from smartphones and dashcams.

[1440] Input: GPS data, weather information, traffic information

[1441] Data processing: The driving environment analysis algorithm analyzes the collected data and identifies the current driving environment.

[1442] Output: Analysis results of the driving environment

[1443] Step 3:

[1444] The drive recorder analyzes the driver's voice and facial expression data using an emotion recognition engine.

[1445] Input: Driver's voice data, facial expression data

[1446] Data processing: Voice and facial expression recognition algorithms identify the driver's emotional state.

[1447] Output: Driver's emotional state

[1448] Step 4:

[1449] The server inputs prompt sentences into the generative AI model based on the analysis of the driving environment and the driver's emotional state.

[1450] Input: Analysis results of driving environment, driver's emotional state

[1451] Data processing: Input prompt sentences into the generative AI model to generate appropriate instructions and emotional responses.

[1452] Output: Generated instructions and emotional responses

[1453] Step 5:

[1454] A smartphone or head-mounted display will display the generated instructions and emotional responses to the driver.

[1455] Input: Generated instructions and emotional responses

[1456] Data processing: Converting data into a format suitable for the display device and displaying it visually.

[1457] Output: Display of driver instructions and emotional responses

[1458] In this way, the system can provide appropriate instructions and conversations in real time that are tailored to the driver's driving environment and emotions, effectively supporting safe driving.

[1459] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1460] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1461] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are listed.

[1462] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1463] [Third embodiment]

[1464] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1465] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1466] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1467] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1468] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1469] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1470] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1471] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1472] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1473] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1474] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1475] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1476] "Example 1"

[1477] One form of this invention is a drive recorder equipped with a conversational AI. This drive recorder monitors the vehicle's driving conditions and detects drowsiness or dangerous driving. If this is detected, the conversational AI will speak to the driver. For example, if the driver appears sleepy, the AI ​​will suggest, "Why don't you take a short break?" Furthermore, if dangerous driving is detected, the AI ​​will advise, "Slow down and drive safely."

[1478] "Example 2"

[1479] Furthermore, conversational AI generates appropriate conversations based on the driver's responses. For example, if the driver replies, "I'm still OK," the AI ​​will respond, "However, driving for long periods of time is dangerous. We recommend that you take appropriate breaks." This will help increase the driver's safety awareness.

[1480] "Example 3"

[1481] Conversational AI also provides instructions to encourage safe driving. For example, when there is heavy traffic or bad weather, it will give instructions such as "Pay attention to surrounding vehicles" or "It's raining and visibility is poor, so please reduce your speed" depending on the driving environment. This helps the driver drive safely.

[1482] The processing flow of each embodiment will be described below.

[1483] "Example 1"

[1484] Step 1: A dashcam equipped with conversational AI monitors the vehicle's driving conditions.

[1485] Step 2: The dashcam detects drowsiness and dangerous driving.

[1486] Step 3: If detected, the conversational AI will speak to the driver. For example, if the driver appears sleepy, the AI ​​will suggest, "Would you like to take a short break?"

[1487] "Example 2"

[1488] Step 1: Conversational AI analyzes the driver's responses.

[1489] Step 2: Based on the driver's response, the AI ​​generates an appropriate conversation. For example, if the driver replies, "I'm still OK," the AI ​​will respond, "However, driving for long periods of time is dangerous. We recommend that you take appropriate breaks."

[1490] "Example 3"

[1491] Step 1: Conversational AI analyzes the driving environment.

[1492] Step 2: Depending on the driving environment, the AI ​​will give instructions to encourage the driver to drive safely. For example, if there is heavy traffic or bad weather, the AI ​​will give instructions such as "Pay attention to surrounding vehicles" or "It's raining and visibility is poor, so please reduce your speed."

[1493] Example 1

[1494] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1495] Conventional dashcams simply record video footage and lack the ability to detect driver drowsiness or dangerous driving in real time and provide appropriate advice. This makes it difficult to promote safe driving and increases the risk of accidents.

[1496] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1497] In this invention, the server includes means for monitoring the driving status of the vehicle, means for detecting drowsiness or dangerous driving, means for speaking to the driver based on the detected drowsiness or dangerous driving, image recognition means for analyzing the driver's facial and eye movements, data analysis means for analyzing the vehicle's speed and acceleration, and means for providing appropriate advice to the driver using conversational AI. This makes it possible to promote safe driving by monitoring the driver's condition and driving status in real time and providing appropriate advice.

[1498] "Means for monitoring the driving status of a vehicle" refers to a device that uses cameras and sensors mounted on the vehicle to observe the driver's behavior and the vehicle's movements in real time.

[1499] "Means for detecting drowsiness and dangerous driving" refers to a device that analyzes collected data and identifies driver drowsiness and dangerous driving behavior such as sudden acceleration and braking.

[1500] "Means for speaking to the driver based on detected drowsiness or dangerous driving" refers to a device that provides warnings and advice to the driver through audio and visual means in response to detected abnormalities.

[1501] "Image recognition means for analyzing the driver's face and eye movements" is software that analyzes images captured by a camera and detects whether the driver's eyes are open or closed, the direction of their face, etc.

[1502] "Data analysis means for analyzing vehicle speed and acceleration" refers to software that analyzes speed and acceleration data obtained from sensors installed in the vehicle and evaluates the driving situation.

[1503] "Means for providing appropriate advice to drivers using conversational AI" refers to a device that uses artificial intelligence to generate appropriate advice and warnings based on the driver's condition and driving situation, and provides them via voice.

[1504] MODE FOR CARRYING OUT THE INVENTION

[1505] This invention relates to a drive recorder equipped with a conversational AI, which is a system that monitors the driving conditions of a vehicle and detects drowsiness and dangerous driving. Specific embodiments of this system are described below.

[1506] 1. Program Generation

[1507] The server generates a program for the drive recorder equipped with conversational AI, which monitors the vehicle's driving status and has the ability to detect drowsiness and dangerous driving.

[1508] 2. Program processing explanation

[1509] The terminal (drive recorder) is installed in the vehicle and processes and calculates data using the following hardware and software.

[1510] Hardware:

[1511] Camera: Monitors the driver's face and eye movements.

[1512] Sensors: Measure the vehicle's speed and acceleration.

[1513] Microphone: Recognizes the driver's voice.

[1514] software:

[1515] Image recognition software: Analyzes video data from the camera to detect driver drowsiness.

[1516] Data analysis software: Analyzes data from sensors to detect risky driving.

[1517] Conversational AI (e.g., GPT-4): Providing appropriate advice to drivers.

[1518] The specific processing flow is as follows:

[1519] 1. Surveillance:

[1520] Cameras and sensors collect data in real time.

[1521] Image recognition software analyzes the driver's facial and eye movements to detect signs of drowsiness.

[1522] Data analysis software analyzes vehicle speed and acceleration to detect dangerous driving.

[1523] 2. Detection:

[1524] If drowsiness is detected, the conversational AI will ask the driver, "Would you like to take a short break?"

[1525] If dangerous driving is detected, the conversational AI will advise the driver to "slow down and drive safely."

[1526] 3. Examples and prompts

[1527] As a concrete example, the following scenario can be considered.

[1528] Scenario 1: The driver has been driving for a long time and the camera detects that the driver's eyes are closing more frequently.

[1529] Conversational AI: "Would you like to take a break?"

[1530] Scenario 2: The vehicle is repeatedly accelerating and braking suddenly on the highway.

[1531] Conversational AI: "Please slow down and drive safely"

[1532] An example of a prompt sentence to input to the generative AI model is as follows:

[1533] Prompt 1: "How would you talk to a driver if they seemed sleepy?"

[1534] Expected output: "Would you like to take a short break?"

[1535] Prompt 2: "What advice would you give if a vehicle was being driven dangerously?"

[1536] Expected output: "Please slow down and drive safely."

[1537] The above is a form for implementing the invention of a dashcam equipped with conversational AI. This system monitors the driver's condition and driving situation in real time and provides appropriate advice, thereby promoting safe driving.

[1538] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1539] Program processing flow

[1540] Step 1: Data collection

[1541] Input: Real-time data from cameras and sensors onboard the vehicle

[1542] Output: Video data of the driver's face and eyes, vehicle speed and acceleration data

[1543] Specific behavior:

[1544] The device uses the vehicle's on-board camera to capture the driver's facial and eye movements, and sensors to measure the vehicle's speed and acceleration, collecting this data in real time and sending it to the next step.

[1545] Step 2: Data analysis

[1546] Input: Video data and velocity / acceleration data collected in step 1

[1547] Output: Driver drowsiness state, vehicle driving status

[1548] Specific behavior:

[1549] The device uses image recognition software to analyze the driver's face and eye movements, detecting how often their eyes are closed and the direction of their face. At the same time, data analysis software is used to analyze fluctuations in speed and acceleration to detect dangerous driving such as sudden acceleration or braking. The analysis results are sent to the next step as an indication of the driver's drowsiness level and driving situation.

[1550] Step 3: Assess the situation

[1551] Input: Driver drowsiness state and driving situation obtained in Step 2

[1552] Output: Advice for the driver

[1553] Specific behavior:

[1554] The device judges the driver's condition and driving situation based on the analysis results. If drowsiness is detected, it flags the driver as "drowsy," and if dangerous driving is detected, it flags the driver as "dangerous driving." This determines the appropriate advice to give to the driver.

[1555] Step 4: Notify the driver

[1556] Input: Advice determined in Step 3

[1557] Output: Voice advice to the driver

[1558] Specific behavior:

[1559] The device uses conversational AI to provide appropriate advice to the driver. If drowsiness is detected, the conversational AI will ask the driver, "Why don't you take a short break?" If dangerous driving is detected, the conversational AI will advise, "Slow down and drive safely." This advice is communicated directly to the driver via the voice output device.

[1560] The above is the specific processing flow of the program of this system.

[1561] (Application example 1)

[1562] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1563] Conventional dashcams only record driving conditions, but lack the ability to detect driver drowsiness or dangerous driving in real time and prompt appropriate action. Another issue is that it is difficult to accurately grasp the driver's condition and provide voice messages at the appropriate time. This results in the problem of being unable to effectively support the driver's safe driving.

[1564] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1565] In this invention, the server is a drive recorder equipped with a conversational AI, and includes means for detecting drowsiness or dangerous driving, means for speaking to the driver based on the detected drowsiness or dangerous driving, means for acquiring camera footage using a smartphone, means for detecting the state of the driver's face and eyes from the acquired camera footage, means for generating a voice message based on the detected state, and means for conveying the generated voice message to the driver. This makes it possible to monitor the driver's state in real time and provide the voice message at an appropriate time.

[1566] "Conversational AI" is artificial intelligence that can converse with humans using natural language processing technology.

[1567] A "drive recorder" is a device that records the driving conditions of a vehicle.

[1568] "Means for detecting drowsiness and dangerous driving" refers to technology that monitors the driver's state and driving behavior and detects drowsiness and dangerous driving in real time.

[1569] "Means of speaking to the driver" refers to technology for conveying audio messages to the driver based on detected information.

[1570] A "smartphone" is a mobile information terminal that has mobile phone functions and is capable of connecting to the Internet and running applications.

[1571] "Means for acquiring camera footage" refers to technology that acquires footage in real time using a smartphone camera.

[1572] "Means for detecting the state of the driver's face and eyes" refers to technology that analyzes the state of the driver's face and eyes from captured camera footage to detect drowsiness or distraction.

[1573] The "means for generating a voice message" is a technology for creating an appropriate voice message based on the detected information.

[1574] The "means for conveying a voice message to the driver" is a technology for making the generated voice message heard by the driver.

[1575] The system for carrying out this invention is mainly composed of a drive recorder equipped with a conversational AI. Specific embodiments are described below.

[1576] System Configuration

[1577] The system consists of the following main components:

[1578] 1. Conversational AI: Artificial intelligence that uses natural language processing technology to converse with the driver.

[1579] 2. Drive recorder: A device that records the driving conditions of a vehicle.

[1580] 3. Smartphone: A mobile information device equipped with a camera and microphone that can connect to the Internet and run applications.

[1581] 4. Camera image acquisition method: Technology that acquires images in real time using a smartphone camera.

[1582] 5. Facial and eye condition detection: Technology that analyzes the driver's facial and eye condition from captured camera footage to detect drowsiness or distraction.

[1583] 6. Voice message generation means: Technology that creates appropriate voice messages based on detected information.

[1584] 7. Voice message delivery means: Technology for delivering generated voice messages to the driver.

[1585] Hardware and Software

[1586] Hardware: Smartphone (camera, microphone), dashcam installed in the vehicle.

[1587] Software: Python, OpenCV, Google Cloud Text-to-Speech API.

[1588] Data processing and calculation

[1589] 1. Camera image acquisition: Images of the interior of the vehicle are acquired in real time using a smartphone camera. The image data is processed using OpenCV.

[1590] 2. Facial and eye detection: The system analyzes the driver's facial and eye conditions from the acquired video data, thereby detecting driver drowsiness and distraction.

[1591] 3. Generate voice message: Based on the detected information, an appropriate voice message is generated using the Google Cloud Text-to-Speech API.

[1592] 4. Transmitting voice messages: The generated voice messages are transmitted to the driver through the smartphone speaker.

[1593] Specific examples

[1594] For example, if the driver appears drowsy, the system will:

[1595] 1. The smartphone camera captures the driver's face and acquires video data.

[1596] 2. Using OpenCV, the driver's eye opening and closing status is analyzed to detect drowsiness.

[1597] 3. If drowsiness is detected, generate a voice message using the Google Cloud Text-to-Speech API saying, "Would you like to take a break?"

[1598] 4. The generated voice message is transmitted to the driver through the smartphone speaker.

[1599] Prompt Sentence Examples

[1600] "If a driver appears sleepy, how should I talk to them?"

[1601] In this way, the functionality of a dashcam equipped with conversational AI can be applied to a smartphone application, making it possible to monitor the driver's condition in real time and provide voice messages at the appropriate time.

[1602] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1603] Step 1:

[1604] The smartphone camera captures images of the inside of the car in real time. The input is the camera image, and the output is video data. Specifically, the smartphone camera captures images including the driver's face and eye condition.

[1605] Step 2:

[1606] The device processes the captured video data using OpenCV to detect the driver's face and eye status. The input is video data, and the output is the analysis results on the face and eye status. Specifically, a facial recognition algorithm is used to identify the driver's face and analyze whether their eyes are open or closed.

[1607] Step 3:

[1608] The device detects driver drowsiness or distraction based on the analysis results. The input is the analysis results related to the state of the face and eyes, and the output is the detection result of drowsiness or distraction. Specifically, it analyzes the frequency of eye opening and closing and blinking patterns to determine signs of drowsiness.

[1609] Step 4:

[1610] The server uses the Google Cloud Text-to-Speech API to generate an appropriate voice message based on the detection results. The input is the detection result of drowsiness or distraction, and the output is a voice message. Specifically, it generates messages such as "Would you like to take a short break?" or "Please reduce your speed and drive safely."

[1611] Step 5:

[1612] The terminal transmits the generated voice message to the driver through the smartphone speaker. The input is the voice message and the output is the voice output. Specifically, the voice message is played from the smartphone speaker so that the driver can hear it.

[1613] Step 6:

[1614] The user responds to the voice message and takes appropriate action. The input is the voice message, and the output is the driver's action, such as taking a break or slowing down.

[1615] Example 2

[1616] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1617] While conventional dashcams have the ability to detect drowsiness and dangerous driving, they have had the problem of being unable to provide specific advice or warnings to drivers at the appropriate time. Furthermore, they are unable to generate appropriate conversations in real time based on the driver's reactions, limiting their effectiveness in encouraging safe driving.

[1618] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1619] In this invention, the server includes a means for recognizing speech, a means for converting the recognized speech into text data, a means for analyzing the text data and generating an appropriate response, a means for converting the generated response into speech data, and a means for outputting the converted speech data. This makes it possible to provide an appropriate response to a driver's utterances in real time and encourage safe driving.

[1620] A "voice recognition means" is a device or software that captures a user's speech and converts it into digital voice data.

[1621] The "means for converting recognized speech into text data" refers to a device or software that analyzes digital speech data and converts it into corresponding text data.

[1622] The "means for analyzing text data and generating an appropriate response" is an algorithm or software for generating an appropriate response based on the input text data.

[1623] The "means for converting the generated response into voice data" is a device or software for converting text data into voice data.

[1624] The "means for outputting converted audio data" refers to a device or software for playing back audio data through an output device such as a speaker.

[1625] A "generative AI model" is an artificial intelligence model that generates responses in natural language based on input data.

[1626] "Speech synthesis software" is software for converting text data into voice data.

[1627] This invention provides a system that provides an appropriate response to a driver's speech in real time to promote safe driving. The system includes a means for recognizing speech, a means for converting the recognized speech into text data, a means for analyzing the text data and generating an appropriate response, a means for converting the generated response into speech data, and a means for outputting the converted speech data.

[1628] Hardware and software used

[1629] Hardware: On-board computer, microphone, speaker

[1630] Software: Generative AI models (e.g., GPT-4), speech recognition software (e.g., Google Speech-to-Text), and speech synthesis software (e.g., Google Text-to-Speech)

[1631] Program processing

[1632] 1. User speaks:

[1633] The user (driver) says "It's still okay" in the car.

[1634] This speech is captured by a microphone connected to the vehicle's onboard computer.

[1635] 2. The device recognizes your voice:

[1636] The terminal (on-board computer) sends the captured voice data to voice recognition software.

[1637] Speech recognition software converts the voice data into text data.

[1638] For example, the speech "It's still okay" is converted into text "It's still okay."

[1639] 3. The server processes the text data:

[1640] The terminal transmits the converted text data to the server.

[1641] The server inputs the received text data into the generative AI model.

[1642] The generative AI model generates appropriate responses based on the input text.

[1643] For example, if the input text is "You're still okay," the generative AI model will generate the response "However, driving for long periods of time is dangerous. We recommend taking regular breaks."

[1644] 4. The device will speak the response:

[1645] The server sends the generated response text to the terminal.

[1646] The terminal inputs the received response text into the speech synthesis software.

[1647] Speech synthesis software converts text data into voice data.

[1648] The terminal transmits the converted voice data to the user through a speaker.

[1649] For example, a voice may play from the speaker saying, "However, driving for long periods of time is still dangerous. We recommend that you take regular breaks."

[1650] Specific examples

[1651] Prompt Sentence Examples

[1652] Input prompt: "Generate an appropriate response if the driver says, 'I'm still OK.'"

[1653] Generated response: "However, driving for long periods of time is dangerous. We recommend taking regular breaks."

[1654] This system can provide appropriate responses to users' comments in real time, increasing their safety awareness.

[1655] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1656] Step 1:

[1657] The user speaks.

[1658] Input: User (driver) utterance (e.g., "It's still okay").

[1659] Specific action: The user speaks in the car.

[1660] Output: A microphone captures the audio data.

[1661] Step 2:

[1662] The device recognizes the voice.

[1663] Input: The captured audio data.

[1664] Specific operation: The terminal (on-board computer) sends voice data to the voice recognition software.

[1665] Output: Speech recognition software converts the speech data into text (e.g., "It's still okay").

[1666] Step 3:

[1667] The server processes the text data.

[1668] Input: The converted text data (e.g., "It's still okay").

[1669] Specific operation: The device sends text data to the server. The server inputs the text data into the generative AI model. The generative AI model generates an appropriate response.

[1670] Output: The generated response text (e.g., "However, driving for long periods of time is dangerous. We recommend that you take regular breaks.").

[1671] Step 4:

[1672] The device will speak the response.

[1673] Input: Generated response text (e.g., "However, driving for long periods of time is dangerous. We recommend taking regular breaks.").

[1674] Specific operations: The server sends the response text to the device. The device inputs the response text into the speech synthesis software. The speech synthesis software converts the text data into speech data.

[1675] Output: The converted audio data is played through the speaker.

[1676] (Application example 2)

[1677] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1678] Although conventional dashcams have the ability to detect drowsiness and dangerous driving, they lack the ability to generate appropriate dialogue based on the driver's reactions and raise safety awareness, making it difficult to encourage drivers to take breaks at appropriate times and to drive safely.

[1679] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for detecting drowsiness or dangerous driving, means for speaking to the driver based on the detected drowsiness or dangerous driving, means for generating appropriate dialogue based on the driver's response, and means for generating prompt sentences using a generative AI model to enhance the driver's safety awareness. This makes it possible to generate appropriate dialogue in response to the driver's response and enhance the driver's safety awareness.

[1680] "Conversational AI" is an artificial intelligence system that uses natural language processing technology to converse with humans.

[1681] A "drive recorder" is a device that records the driving status of a vehicle and has the ability to record video and audio.

[1682] "Means for detecting drowsiness and dangerous driving" refers to a system that includes sensors and algorithms for monitoring the driver's state and driving behavior and detecting drowsiness and dangerous driving.

[1683] "Means to speak to the driver" refers to a system that includes a speaker or voice synthesis technology to provide audio warnings and advice to the driver based on detected information.

[1684] The "means for generating appropriate dialogue based on the driver's responses" refers to a system that includes algorithms and software for analyzing the driver's responses and actions and generating appropriate dialogue accordingly.

[1685] A "generative AI model" is a model that uses artificial intelligence technology to generate text and speech, and is particularly used in natural language processing.

[1686] A "prompt" is text that is input to a generative AI model and contains instructions or questions to achieve a specific output.

[1687] "Means to raise driver safety awareness" refers to a system that provides drivers with information and advice to encourage safe driving.

[1688] As an embodiment of the present invention, a drive recorder system equipped with a conversational AI will be described. This system includes means for detecting drowsiness or dangerous driving, means for speaking to the driver based on the detected drowsiness or dangerous driving, means for generating appropriate conversation based on the driver's response, and means for generating prompt sentences using a generative AI model to enhance the driver's safety awareness.

[1689] First, the system uses a vehicle-mounted dashcam, which is equipped with cameras and sensors to monitor the driver's facial expressions and driving behavior, thereby detecting drowsiness and dangerous driving.

[1690] Next, based on the detected information, the system provides the driver with audible warnings and advice. For example, if the driver replies, "I'm still OK," the system might respond, "However, driving for long periods of time is still dangerous. We recommend you take appropriate breaks." In this way, a generative AI model is used to generate appropriate dialogue based on the driver's responses.

[1691] The generative AI model is implemented using OpenAI's API. The model generates prompts based on the driver's input and generates appropriate responses. Specifically, the following prompts are used:

[1692] Example prompt sentence:

[1693] "If the driver says 'I'm still OK,' generate an appropriate response."

[1694] By inputting this prompt into a generative AI model, the system can generate an appropriate response for the driver, increasing their safety awareness.

[1695] The system is implemented using a smartphone. An application is installed on the smartphone to receive data from the dashcam and generate prompts using the generative AI model. This application is implemented using a Python program and operates by calling OpenAI's API.

[1696] For example, if a driver says, "I'm a little tired," the system will respond, "In that case, I recommend you take a break at the next service area." In this way, the system can generate appropriate conversations based on the driver's reactions, increasing the driver's safety awareness.

[1697] As described above, an embodiment of the present invention uses a drive recorder system equipped with conversational AI to generate appropriate conversations based on the driver's reactions, thereby increasing the driver's safety awareness.

[1698] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1699] Step 1:

[1700] A dashcam uses cameras and sensors inside the vehicle to monitor the driver's facial expressions and driving behavior. The input is real-time data from the cameras and sensors, and the output is data indicating the driver's state (e.g., signs of drowsiness or dangerous driving).

[1701] Step 2:

[1702] The data detected by the drive recorder is analyzed to detect drowsiness and dangerous driving. The input is the output data from Step 1, and the output is the detection result of drowsiness and dangerous driving. Specifically, the system uses image analysis algorithms and machine learning models to evaluate the driver's condition.

[1703] Step 3:

[1704] Based on the detection results, the dashcam provides audio warnings and advice to the driver. The input is the detection results from Step 2, and the output is an audio message for the driver. Specifically, it uses speech synthesis technology to generate an appropriate message and plays it through the speaker.

[1705] Step 4:

[1706] The driver responds to the voice message. The input is the driver's voice response, and the output is the voice data. Specifically, the driver's response is recorded using a microphone.

[1707] Step 5:

[1708] The terminal converts the driver's voice response into text. The input is the voice data from step 4, and the output is text data. Specifically, it uses voice recognition technology to convert the voice into text.

[1709] Step 6:

[1710] The device uses the generative AI model to generate a prompt sentence based on the driver's response. The input is the text data from step 5, and the output is the prompt sentence. Specifically, it generates the following prompt sentence: "If the driver says 'It's still OK,' please generate an appropriate response."

[1711] Step 7:

[1712] The device uses the generative AI model to generate an appropriate response based on the prompt. The input is the prompt from step 6, and the output is the generated response text. Specifically, the device calls the OpenAI API, inputs the prompt into the generative AI model, and obtains a response.

[1713] Step 8:

[1714] The terminal converts the generated response into speech and provides it to the driver. The input is the response text from step 7, and the output is a voice message. Specifically, it uses speech synthesis technology to convert the text into speech and plays it back through the speaker.

[1715] Through these steps, the system can generate appropriate conversations based on the driver's reactions, enhancing the driver's safety awareness.

[1716] Example 3

[1717] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1718] Conventional dashcams and conversational AI systems were limited to detecting driver drowsiness and dangerous driving, and were unable to provide specific safe driving instructions in real time according to the driving environment. This made it difficult to quickly respond to changes in traffic volume, weather, etc. and encourage safe driving.

[1719] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1720] In this invention, the server includes means for collecting information on the driving environment, such as traffic volume data and weather data, means for analyzing the collected data and evaluating the driving environment, means for inputting prompt sentences to the generative AI model based on the analysis results, means for the generative AI model to receive the prompt sentences and generate appropriate driving instructions, means for transmitting the generated instructions to the terminal, and means for the terminal to convey the received instructions to the driver using speech synthesis software and text display software, thereby making it possible to provide specific safe driving instructions according to the driving environment in real time.

[1721] "Traffic volume data" is information about the number and flow of vehicles on roads.

[1722] "Weather data" is information about current weather conditions.

[1723] The "driving environment" refers to external conditions, such as traffic volume and weather, that affect the driver's driving.

[1724] "Means of collection" refers to methods of obtaining data using traffic sensors, weather APIs, etc.

[1725] "Means of analysis" refers to methods for evaluating the driving environment based on collected data.

[1726] A "generative AI model" is an artificial intelligence model that generates appropriate driving instructions based on input prompts.

[1727] A "prompt sentence" is a sentence that is input into a generative AI model to generate instructions.

[1728] "Driving instructions" are specific instructions to encourage the driver to drive safely.

[1729] "Transmitting means" refers to a method for sending the generated driving instructions to the terminal.

[1730] A "terminal" is a device that receives driving instructions sent from the server and conveys them to the driver.

[1731] "Speech synthesis software" is software that converts text into speech and plays it back.

[1732] "Text display software" is software that displays text on a screen.

[1733] MODE FOR CARRYING OUT THE INVENTION

[1734] This invention is a system that collects and analyzes information about the driving environment, such as traffic volume data and weather data, and generates appropriate driving instructions using a generative AI model to communicate them to the driver. Specific embodiments of this system are described below.

[1735] Server Roles

[1736] The server collects information about the driving environment, such as traffic volume data and weather data. Traffic volume data is obtained from traffic sensors, and weather data is obtained using a weather API. After collecting this data, the server analyzes it and evaluates the current driving environment, such as whether traffic volume is heavy or light, whether the weather is sunny or rainy, etc.

[1737] Next, the server inputs prompt sentences into the generative AI model based on the analysis results. Examples of prompt sentences include "Current traffic volume is heavy. Please generate appropriate instructions for the driver" or "Current weather is rainy. Please generate safe driving instructions for the driver."

[1738] The generative AI model receives the prompt and generates appropriate driving instructions, such as "Watch out for surrounding vehicles" or "It's raining and visibility is poor, so please reduce your speed." The generated instructions are then sent from the server to the device.

[1739] Device Role

[1740] The device receives instructions sent from the server. This is done via Wi-Fi or mobile data communication. The device then communicates the received instructions to the driver using speech synthesis software or text display software. For example, speech synthesis software can be used to say, "Pay attention to surrounding vehicles," and text display software can be used to display the same instruction on the screen.

[1741] User (Driver) Role

[1742] The user drives according to the instructions provided by the terminal. For example, if the user receives an instruction such as "It's raining and visibility is poor, so please slow down," the user will drive at a slower speed. In this way, the user can drive safely by following the system's instructions.

[1743] Specific examples

[1744] 1. When there is heavy traffic

[1745] The server obtains and analyzes real-time traffic data from traffic sensors.

[1746] Based on the analysis results, a prompt sentence such as "Current traffic volume is heavy. Please generate appropriate instructions for the driver" is input into the generative AI model.

[1747] The generative AI model generates the instruction "Pay attention to surrounding vehicles," and the server sends this instruction to the device.

[1748] The device receives instructions and uses speech synthesis software to communicate them aloud to the driver.

[1749] The user follows these instructions and drives while paying attention to surrounding vehicles.

[1750] 2. Bad weather

[1751] The server uses a weather API to obtain and analyze current weather information.

[1752] Based on the analysis results, the prompt sentence "The current weather is rainy. Please generate safe driving instructions for the driver" is input into the generative AI model.

[1753] The generative AI model generates an instruction such as "It's raining and visibility is poor, so please slow down," and the server sends this instruction to the device.

[1754] The terminal receives the instructions and displays them to the driver using text display software.

[1755] The user follows this instruction and drives at a reduced speed.

[1756] In this way, the system provides real-time instructions to assist the driver in safe driving. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1757] Step 1:

[1758] The server collects traffic and weather data from traffic sensors and weather APIs. Specifically, it obtains real-time vehicle count and flow data from traffic sensors and current weather information from weather APIs. These data are used as input data for evaluating the driving environment.

[1759] Step 2:

[1760] The server analyzes the collected traffic volume data and weather data. Specifically, it analyzes the traffic volume data to determine whether the current traffic volume is "heavy," "normal," or "light," and analyzes the weather data to determine the weather condition, such as "sunny," "rainy," or "snowy." The results of this analysis become the basic data for generating prompt sentences to be input into the generative AI model.

[1761] Step 3:

[1762] The server generates prompt sentences to be input into the generative AI model based on the analysis results. For example, it generates prompt sentences such as "Current traffic volume is heavy. Please generate appropriate instructions for the driver" or "Current weather is rainy. Please generate safe driving instructions for the driver." These prompt sentences are input into the generative AI model.

[1763] Step 4:

[1764] The generative AI model receives prompts from the server and generates appropriate driving instructions, such as "Watch out for surrounding vehicles" or "It's raining and visibility is poor, so please reduce your speed." These generated driving instructions are then returned to the server.

[1765] Step 5:

[1766] The server sends the driving instructions returned by the generative AI model to the terminal via real-time communication over the Internet. These driving instructions are then input into the terminal.

[1767] Step 6:

[1768] The device receives driving instructions sent from the server via Wi-Fi or mobile data communication, and the received driving instructions are input into speech synthesis software and text display software.

[1769] Step 7:

[1770] The device then uses voice synthesis software and text display software to communicate the received driving instructions to the driver. Specifically, the device uses voice synthesis software to say, "Pay attention to surrounding vehicles," and text display software to display the same instruction on the screen. This driving instruction is then communicated to the driver.

[1771] Step 8:

[1772] The user (driver) drives according to the driving instructions provided by the terminal. For example, if the user receives an instruction such as "It's raining and visibility is poor, so please slow down," the user will drive at a slower speed. In this way, the user can drive safely by following the system's instructions.

[1773] (Application example 3)

[1774] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1775] Although conventional dashcams have the ability to detect drowsiness and dangerous driving, they are unable to provide real-time safe driving instructions tailored to the driving environment. Furthermore, they lack the ability to learn the driver's driving patterns and provide individually optimized advice. This makes it difficult to effectively support safe driving.

[1776] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server is a drive recorder equipped with a conversational AI, and includes means for detecting drowsiness or dangerous driving, means for speaking to the driver based on the detected drowsiness or dangerous driving, means for collecting environmental data during driving in real time, means for providing the driver with voice instructions for safe driving based on the collected data, and means for learning the driver's driving patterns and providing individually optimized advice. This makes it possible to provide real-time safe driving instructions according to the driving environment and provide individually optimized advice based on the driver's driving patterns.

[1777] "Conversational AI" is an artificial intelligence system that can converse with humans using natural language processing technology.

[1778] A "drive recorder" is a device that records video and audio while a vehicle is in motion.

[1779] "Means for detecting drowsiness and unsafe driving" refers to a system that includes sensors and algorithms to monitor the driver's state and driving behavior and identify drowsiness or unsafe driving behavior.

[1780] "Means for speaking to the driver" refers to an audio output device or software for providing audio warnings and instructions to the driver based on detected information.

[1781] "Means for collecting environmental data in real time while driving" refers to sensors and communication devices for obtaining data on the driving environment, such as weather, traffic volume, and road conditions, in real time.

[1782] "Means for providing voice instructions to the driver on safe driving based on collected data" refers to a system that analyzes collected environmental data and conveys appropriate voice instructions to the driver on safe driving.

[1783] "Means for learning the driver's driving patterns and providing individually optimized advice" refers to machine learning algorithms and software that analyze the driver's driving history and behavioral patterns and provide driving advice that is optimized for each individual driver.

[1784] As an embodiment of the present invention, a drive recorder system equipped with a conversational AI is constructed. A specific embodiment of the system is shown below.

[1785] System Configuration

[1786] The system includes the following major hardware and software components:

[1787] Hardware:

[1788] Drive recorder body

[1789] Sensors (camera, microphone, GPS, accelerometer, etc.)

[1790] Audio output device (speaker)

[1791] Communication modules (Wi-Fi, Bluetooth, 4G / 5G, etc.)

[1792] software:

[1793] Conversational AI (natural language processing engine)

[1794] Data Collection Module

[1795] Data Analysis Module

[1796] Speech generation module (e.g. gTTS)

[1797] Machine Learning Algorithms

[1798] Processing flow

[1799] 1. Data Collection:

[1800] A dashcam collects real-time environmental data (weather, traffic volume, road conditions, etc.) while driving by using cameras and sensors to capture video, audio, and location information.

[1801] 2. Data Analysis:

[1802] The collected data is analyzed by a data analysis module. For example, weather information is obtained from a weather data API, and traffic volume information is obtained from a traffic volume data API.

[1803] 3. Generate safe driving instructions:

[1804] Based on the analysis results, the conversational AI generates safe driving instructions for the driver, such as "It's raining and visibility is poor, so please slow down" or "Traffic is heavy, so please be careful of surrounding vehicles."

[1805] 4. Audio output:

[1806] The generated instructions are converted into voice using a voice generation module and provided to the driver through a speaker.

[1807] 5. Learning driving patterns:

[1808] The machine learning algorithm learns the driver's driving history and behavioral patterns and provides individually optimized advice.

[1809] Specific examples

[1810] For example, if the conversational AI receives information about "rain" from the weather data API, it will generate the instruction, "It's raining and visibility is poor, so please reduce your speed." Similarly, if the AI ​​receives information about "heavy traffic" from the traffic volume data API, it will generate the instruction, "Heavy traffic, so please be careful of surrounding vehicles."

[1811] Prompt Sentence Examples

[1812] Weather data: Rain

[1813] Traffic data: Heavy traffic

[1814] Generated instructions: Rain and poor visibility, please reduce speed. Traffic is heavy, so please be aware of surrounding vehicles.

[1815] In this way, it is possible to provide real-time safe driving instructions tailored to the driving environment and provide personalized advice based on the driver's driving patterns.

[1816] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1817] Step 1:

[1818] The server collects real-time environmental data while driving. Specifically, it uses cameras and sensors to capture video, audio, and location information. Inputs include camera video, microphone audio, GPS data, and acceleration sensor data. This data is collected and sent to the server.

[1819] Step 2:

[1820] The server analyzes the collected data. Specifically, it obtains weather information from the weather data API and traffic volume information from the traffic volume data API. The input includes the environmental data collected in step 1. The server analyzes this data and understands the driving environment. The output is weather information and traffic volume information as the analysis results.

[1821] Step 3:

[1822] The server generates safe driving instructions based on the analysis results. Specifically, the conversational AI generates instructions such as "It's raining and visibility is poor, so please slow down" or "There's a lot of traffic, so please be careful of surrounding vehicles." The input includes the analysis results obtained in step 2. The server generates appropriate instructions based on these analysis results. The output is the generated safe driving instructions.

[1823] Step 4:

[1824] The server converts the generated instructions into voice and provides them to the driver. Specifically, it uses a voice generation module (such as gTTS) to convert the instructions into voice and transmits them to the driver through a speaker. The input includes the instructions generated in step 3. The server converts these instructions into voice data and sends it to an audio output device. The output is voice-based safe driving instructions.

[1825] Step 5:

[1826] The server learns the driver's driving patterns and provides individually optimized advice. Specifically, it analyzes the driver's driving history and behavioral patterns using a machine learning algorithm and generates driving advice optimized for each individual driver. The input includes the driver's driving history data. The server analyzes this data and generates individually optimized advice. The output is individually optimized driving advice.

[1827] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1828] "Example 1"

[1829] One embodiment of the present invention is a system that combines an emotion engine with a dashcam equipped with conversational AI. This system recognizes the driver's emotions and generates appropriate conversations based on those emotions. For example, if the AI ​​senses that the driver is feeling angry, it will suggest, "I recommend you take a short break."

[1830] "Example 2"

[1831] The emotion engine also provides instructions to encourage safe driving based on the driver's emotions. For example, if the driver feels tired, the AI ​​will advise them to "take a moderate break."

[1832] "Example 3"

[1833] Furthermore, the emotion engine generates appropriate conversations depending on the driver's emotions. For example, if the AI ​​senses that the driver is happy, it will praise them by saying, "Great driving." This will increase the driver's motivation to drive and encourage safe driving.

[1834] The processing flow of each embodiment will be described below.

[1835] "Example 1"

[1836] Step 1: A dashcam equipped with conversational AI captures the driver's face with its camera.

[1837] Step 2: The emotion engine recognizes the driver's emotions from the captured video.

[1838] Step 3: Based on the recognized emotion, the AI ​​generates an appropriate conversation and conveys it to the driver.

[1839] "Example 2"

[1840] Step 1: A dashcam equipped with conversational AI captures the driver's face with its camera.

[1841] Step 2: The emotion engine recognizes the driver's emotions from the captured video.

[1842] Step 3: Based on the recognized emotion, the AI ​​generates and conveys safe driving instructions to the driver.

[1843] "Example 3"

[1844] Step 1: A dashcam equipped with conversational AI captures the driver's face with its camera.

[1845] Step 2: The emotion engine recognizes the driver's emotions from the captured video.

[1846] Step 3: Based on the recognized emotion, the AI ​​generates an appropriate conversation and conveys it to the driver.

[1847] Example 1

[1848] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1849] Conventional dashcams are limited to recording driving conditions and providing evidence in the event of an accident, and lack the ability to detect driver drowsiness or dangerous driving in real time and provide appropriate advice. Furthermore, they are unable to provide advice that takes into account the driver's emotional state, making it difficult to effectively promote safe driving.

[1850] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1851] In this invention, the server includes means for collecting driving situation data, means for analyzing the collected data, means for detecting drowsiness or dangerous driving, means for speaking to the driver based on the detected drowsiness or dangerous driving, means for analyzing the driver's emotions, and means for generating appropriate conversation based on the analyzed emotions. This makes it possible to monitor the driver's driving situation in real time and provide appropriate advice when drowsiness or dangerous driving is detected. Furthermore, providing advice that takes the driver's emotional state into consideration can effectively promote safe driving.

[1852] "Driving situation data" refers to information collected while driving a vehicle, such as speed, steering operation, and the driver's facial expressions.

[1853] "Means for collection" refers to devices or methods for acquiring driving situation data using cameras, sensors, etc.

[1854] The "means of analysis" refers to algorithms and software that process collected driving data in real time and detect drowsiness and dangerous driving.

[1855] "Means for detecting drowsiness or dangerous driving" refers to devices or methods that determine signs of driver drowsiness or dangerous driving based on analyzed data.

[1856] "Speech intervention" refers to a device or method that provides audible advice or warnings to the driver based on detected drowsiness or unsafe driving.

[1857] "Means of emotion analysis" refers to algorithms or software that can determine a driver's emotional state from facial expressions, tone of voice, etc.

[1858] The "means for generating appropriate conversation" is a device or method that provides appropriate advice or suggestions to the driver by voice based on the analyzed emotional state.

[1859] This invention is a system that uses a drive recorder equipped with conversational AI to monitor the driver's driving status and detect drowsiness and dangerous driving. Furthermore, by combining it with an emotion engine, it recognizes the driver's emotions and generates appropriate conversations based on those emotions.

[1860] Hardware and software used

[1861] Hardware: Cameras, speed sensors, and steering sensors mounted on the vehicle

[1862] Software: Image processing algorithms, AI algorithms, emotion analysis engines, speech synthesis technology

[1863] Data collection and analysis

[1864] The device collects driving situation data from cameras and sensors installed in the vehicle. Specifically, the camera captures the driver's facial expressions, and the sensors capture data on speed and steering operation. This data is collected in real time and sent to a server.

[1865] The server analyzes the collected data in real time. It uses image processing algorithms to analyze the frequency of the driver's eye movements and the direction of their face, and analyzes driving patterns from the sensor data. The analysis results are temporarily stored and used for subsequent processing.

[1866] Detecting drowsiness and dangerous driving

[1867] The device detects drowsiness or dangerous driving based on the analysis results. For example, if the frequency of eye opening and closing exceeds a certain threshold or if there are repeated sudden steering operations, it will determine that the driver is drowsy or driving dangerously. The server records the detection results and generates an alert if necessary.

[1868] Conversational AI advice

[1869] If the device detects drowsiness, it will use conversational AI to ask the driver, "Why don't you take a short break?" Specifically, it uses speech synthesis technology to provide advice in a natural voice. If it detects dangerous driving, it will advise, "Slow down and drive safely."

[1870] Sentiment analysis and appropriate conversation generation

[1871] The server uses an emotion engine to analyze the driver's emotions. Specifically, it senses anger and stress from facial expressions and tone of voice. If the device recognizes that the driver is feeling angry, it uses conversational AI to suggest, "We recommend you take a short break." Based on the emotion analysis results, it generates appropriate advice and communicates it to the driver using voice synthesis technology.

[1872] Specific examples

[1873] For example, if a driver has been driving for a long time and begins to open and close their eyes more frequently, the device will detect drowsiness. At this time, the conversational AI will suggest to the driver, "Why not take a short break?" Also, if the driver makes a sudden steering maneuver, the device will detect dangerous driving and advise, "Slow down and drive safely."

[1874] Prompt Sentence Examples

[1875] Examples of prompts to be input to a generative AI model include:

[1876] Prompt: Describe how a conversational AI might address a driver if they appear sleepy.

[1877]

[1878] Prompt: Explain how a conversational AI using an emotion engine can respond when a driver is angry.

[1879] In this way, the system provides appropriate advice depending on the driver's condition.

[1880] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1881] Step 1: Data collection

[1882] The device collects driving situation data from cameras and sensors installed in the vehicle. Specifically, the camera captures the driver's facial expressions, and the sensors capture data on speed and steering operation.

[1883] Input: Camera footage, speed data, steering operation data

[1884] Output: Collected driving situation data

[1885] Step 2: Data analysis

[1886] The device analyzes the collected data in real time, using image processing algorithms to analyze the frequency of the driver's eye movements and the direction of their face, and then analyzes driving patterns from the sensor data.

[1887] Input: Collected driving situation data

[1888] Output: Analysis results (eye opening / closing frequency, face direction, driving pattern)

[1889] Step 3: Detecting drowsiness and dangerous driving

[1890] The device uses the analysis results to detect drowsiness or dangerous driving. For example, if the frequency of eye opening and closing exceeds a certain threshold or if there are repeated sudden steering operations, it will determine that the driver is drowsy or driving dangerously.

[1891] Input: Analysis results

[1892] Output: Detection results (presence or absence of drowsiness, dangerous driving)

[1893] Step 4: Conversational AI advice

[1894] If the device detects drowsiness, it will use conversational AI to ask the driver, "Why don't you take a short break?" If it detects dangerous driving, it will advise the driver, "Slow down and drive safely."

[1895] Input: Detection result

[1896] Output: Audio advice

[1897] Step 5: Sentiment Analysis

[1898] The server uses an emotion engine to analyze the driver's emotions, specifically detecting anger and stress from facial expressions and tone of voice.

[1899] Input: Driver's facial expression, tone of voice

[1900] Output: Emotion analysis results (anger, stress)

[1901] Step 6: Emotion-aware conversation generation

[1902] If the device recognizes that the driver is feeling angry, it will use conversational AI to suggest, "We recommend that you take a short break." Based on the results of emotion analysis, the device generates appropriate advice and conveys it to the driver using voice synthesis technology.

[1903] Input: Sentiment analysis results

[1904] Output: Audio advice

[1905] In this way, the system provides appropriate advice depending on the driver's condition.

[1906] (Application example 1)

[1907] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1908] Conventional dashcams only record driving conditions, and have the problem of not being able to take appropriate action even if they detect driver drowsiness or dangerous driving. In addition, they provide uniform advice without taking into account the driver's emotions, making it difficult to effectively promote safe driving.

[1909] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server is a drive recorder equipped with a conversational AI, and includes means for detecting drowsiness or dangerous driving, means for speaking to the driver based on the detected drowsiness or dangerous driving, means for recognizing the driver's emotions, and means for generating appropriate conversation based on the recognized emotions. This makes it possible to provide appropriate advice according to the driver's condition and emotions, and promote safe driving.

[1910] "Conversational AI" is an artificial intelligence system that can converse with humans using natural language processing technology.

[1911] A "drive recorder" is a device for recording the driving conditions of a vehicle.

[1912] "Means for detecting drowsiness and dangerous driving" refers to a function that uses sensors and algorithms to detect signs of driver drowsiness or dangerous driving.

[1913] "Means of speaking to the driver" is a function that conveys messages to the driver via voice or text based on detected information.

[1914] "Means for recognizing the driver's emotions" refers to technology that analyzes and recognizes the driver's emotions from their facial expressions, voice, actions, etc.

[1915] "Means for generating appropriate conversation" is a function that generates appropriate conversation for the driver based on the recognized emotions and situation.

[1916] As an embodiment of the present invention, a drive recorder system equipped with a conversational AI will be described. This system is configured using the following hardware and software.

[1917] Hardware

[1918] Camera: Installed inside the vehicle, it captures the driver's face and driving situation in real time.

[1919] Microphone: Collects the driver's voice and uses it for emotion recognition and speech generation.

[1920] Computer: Processes data and runs AI models.

[1921] software

[1922] OpenCV: An image processing library that analyzes images from the camera and performs face detection.

[1923] dlib: A library for face detection and facial feature point extraction.

[1924] transformers: A library for natural language processing and emotion recognition, specifically using emotion recognition models.

[1925] Generative AI model: A model for generating appropriate dialogue based on the driver's emotions and driving situation.

[1926] Data processing and calculation

[1927] 1. Video capture: Capture video from the camera in real time and convert it to grayscale using OpenCV.

[1928] 2. Face detection: Detect the driver's face using dlib and extract the face region.

[1929] 3. Emotion Recognition: Using the transformers library, we analyze the driver's emotions from the extracted facial region.

[1930] 4. Conversation generation: Using a generative AI model, appropriate conversations are generated based on recognized emotions and driving situations.

[1931] Specific examples

[1932] For example, if the driver is feeling angry, the emotion recognition model will detect "anger." Based on this information, the generative AI model will generate a conversation and tell the driver, "We recommend you take a short break."

[1933] Prompt Sentence Examples

[1934] Use the following prompt for the generative AI model:

[1935] If the driver's emotion is "anger," generate appropriate dialogue.

[1936] In this way, it is possible to provide appropriate advice based on the driver's condition and emotions, promoting safe driving.

[1937] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1938] Step 1:

[1939] The server captures video from the camera in real time. The input is video data from the camera, and the output is the captured video frames. Specifically, the server initializes the camera device and acquires video frames.

[1940] Step 2:

[1941] The server converts the captured video frame to grayscale using OpenCV. The input is the captured video frame, and the output is the video frame converted to grayscale. Specifically, it converts color video to grayscale using OpenCV functions.

[1942] Step 3:

[1943] The server uses dlib to detect the driver's face from the grayscale video. The input is a grayscale video frame, and the output is the coordinate information of the detected face area. Specifically, the server uses dlib's face detector to identify the position of the face.

[1944] Step 4:

[1945] The server extracts the detected face region and performs emotion recognition using the transformers library. The input is image data of the face region, and the output is the driver's emotion label (e.g., anger, drowsiness). Specifically, the server inputs the face region into an emotion recognition model and obtains the emotion label.

[1946] Step 5:

[1947] The server inputs prompts to the generative AI model based on the recognized emotion labels to generate appropriate conversations. The inputs are emotion labels and prompts, and the output is the generated conversation text. Specifically, the following prompts are used for the generative AI model:

[1948] If the driver's emotion is "anger," generate appropriate dialogue.

[1949] Step 6:

[1950] The server converts the generated conversation text into speech using a speech synthesis engine and conveys it to the driver. The input is the generated conversation text and the output is voice data. Specifically, the server converts the text into speech using the speech synthesis engine and plays it back through the speaker.

[1951] In this way, it is possible to provide appropriate advice based on the driver's condition and emotions, promoting safe driving.

[1952] Example 2

[1953] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1954] Conventional dashcams simply record video footage and lack the functionality to detect driver drowsiness or dangerous driving and take appropriate action. Furthermore, there are no systems that analyze the driver's emotional state and provide instructions to encourage safe driving based on that analysis. This makes it difficult to raise drivers' safety awareness, and increases the risk of accidents.

[1955] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1956] In this invention, the server includes means for acquiring voice input, means for converting the acquired voice data into text data, means for analyzing the driver's reaction, means for generating appropriate dialogue, means for capturing the driver's facial expressions, means for analyzing the driver's emotional state, and means for issuing instructions to the driver based on the analysis results. This makes it possible to generate appropriate dialogue and instructions based on the driver's reaction and emotional state, thereby increasing the driver's safety awareness.

[1957] "Means for obtaining voice input" refers to devices or technologies for capturing the driver's voice in real time. 【1...

Claims

1. Image recognition means to analyze the driver's face and eye movements; A data analysis means for analyzing a speed and an acceleration of a vehicle; a means for monitoring a driving condition of the vehicle based on an analysis result by the data analysis means; A means for detecting drowsiness or dangerous driving of the driver based on the analysis result by the image recognition means; means for speaking to the driver based on the detected drowsiness or unsafe driving; A means for providing appropriate advice to the driver using a conversational AI; A system including:

2. The system of claim 1 , wherein the conversational AI further comprises means for generating appropriate dialogue based on a driver's response.

3. The system of claim 1 , wherein the conversational AI further comprises means for issuing instructions to encourage the driver to drive safely.

Images