System

Abstract

A system is provided.SOLUTION: A system comprising: means including an electro-oculogram sensor and camera device for real-time detection of eyelid and eyeball movement; means including an artificial intelligence algorithm for analyzing data acquired by said sensor and camera device to detect an instantaneous sleep; light emitting means for executing a visual alert in response to detection of said instantaneous sleep; audio output means for executing an audio alert in response to detection of said instantaneous sleep; communication means for communicating data from said sensor and camera device; and external server means for storing and analyzing data transmitted by said communication means.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] While we strive for a comfortable world free of traffic accidents, the number of car accidents attributed to drowsy driving remains high, particularly in Japan, where it has remained stable, but is on the rise in Europe and the United States. Reducing these unfortunate accidents requires a system that can quickly and effectively warn drivers before they fall into a momentary sleep (microsleep). However, current technology lacks a system that can detect momentary sleep and provide multiple warnings. There is also a need for a system that will popularize this technology by accumulating and analyzing driver driving data and offering incentives such as insurance discounts. [Means for solving the problem]

[0005] To address this issue, the present invention provides a system that includes an electrooculography sensor and a camera device that instantly detects eyelid and eyeball movements. It also includes an artificial intelligence algorithm that analyzes data acquired by the sensor and camera device to detect momentary sleep. In response to this detection, the system includes both a light-emitting device that issues a visual warning and a voice output device that issues an audible warning. It also includes a communication device that transmits data from the sensor and camera device, allowing the transmitted data to be stored and analyzed on an external server. It also includes a function that links with a navigation system to announce navigation and traffic congestion information through the voice output device, providing insights into the driver's driving characteristics. It also works with insurance companies to offer insurance premium discounts based on the driver's driving data, increasing its value to users.

[0006] An "electrooculography sensor" is a device that measures fluctuations in the electrical potential of the eyelids and eyeballs, and is used to detect the movements of the eyelids and eyeballs.

[0007] "Camera device" means a device that captures images or video and is used to monitor eyelid and eyeball movement in real time.

[0008] The "artificial intelligence algorithm" is a computational method that analyzes data obtained from sensors and camera devices, recognizes patterns, and detects momentary sleep.

[0009] The "light emitting means" is a device equipped with a light source for visually informing the user of a warning.

[0010] "Audio output means" refers to a speaker or similar device for audibly conveying warnings and guidance information to the user.

[0011] "Communication means" refers to modules and systems for transmitting data from sensors and camera devices to the outside.

[0012] The "external server means" is a central management system that has the function of storing and analyzing data received via the communication means.

[0013] A "navigation system" is an electronic system that provides route guidance and traffic congestion information to drivers.

[0014] "Insights" refers to useful information or insights derived from the results of data analysis.

[0015] "Insurance premium discount" means reducing the insurance amount based on the user's driving data. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The present invention is a system for preventing drowsy driving and ensuring driver safety. The system includes an electrooculography sensor and camera device that detects eyelid and eyeball movements, an artificial intelligence algorithm that analyzes data in real time, visual and audio warning devices, a communication module that transmits data, data storage and analysis on an external server, linkage with a navigation system, and an insurance premium discount function.

[0038] System configuration

[0039] 1. Detection equipment and artificial intelligence algorithms

[0040] The device is equipped with an electrooculography sensor and a camera device to detect the driver's eye and eyelid movements. This allows for real-time monitoring of the driver's eye and eye movements while driving, and eye movement data is acquired. This data is analyzed by an artificial intelligence algorithm to detect microsleep.

[0041] 2. Visual and audible warnings

[0042] When the device detects momentary drowsiness, an LED mounted on the frame lights up to provide a visual warning to the driver. In addition, an audible alert sounds from the speaker, providing an audible warning. This helps to draw the driver's attention and prevent drowsy driving.

[0043] 3. Data communication and analysis

[0044] The terminal includes a communication module for transmitting real-time acquired sensor data to an external server, which receives and stores the data. The stored data is used to analyze the driver's driving characteristics and provide feedback to the driver.

[0045] 4. Navigation linkage

[0046] The device is linked to the navigation system, allowing drivers to receive real-time navigation and traffic information through the speaker. For example, it will announce the optimal route to the destination or changes in road conditions.

[0047] 5. Premium discount function

[0048] The server provides drivers with discounts on insurance premiums through collaboration with insurance companies. Specifically, it evaluates safe driving based on driving data and discounts insurance premiums according to the evaluation results. This function provides drivers with an incentive to drive safely.

[0049] Program processing

[0050] Data collection and instantaneous sleep detection

[0051] The device uses sensors and a camera to collect eye movement data, which is then analyzed in real time by an artificial intelligence algorithm to detect momentary sleep.

[0052] Actions triggered by the warning

[0053] When the device detects a moment of sleep, it immediately issues a visual and audible alert: the LED on the frame lights up and an alert sound comes from the speaker.

[0054] Data transmission and analysis

[0055] The device sends the collected data to an external server, which stores the data and analyzes driving characteristics. The analysis results are provided to the driver as feedback.

[0056] Providing navigation information

[0057] The device works in conjunction with the navigation system to obtain the latest traffic information, and announces navigation and traffic congestion information to the driver through a speaker.

[0058] Insurance premium discount notice

[0059] The server evaluates the driver's safe driving based on the driving data and provides discounts on insurance premiums according to the evaluation results. Discount information is notified to the driver.

[0060] Specific usage example

[0061] Suppose a user is wearing AI-enabled glasses and driving a car. While driving, the device's electrooculography sensor and camera device collect eye movement data and detect momentary sleep. When momentary sleep is detected, the device's LED lights up and an alert sounds from the speaker to warn the driver. At the same time, the sensor data is sent from the device to a server, where it is accumulated and analyzed. Next, the device obtains the latest traffic information from the navigation system and announces it to the driver through the speaker. Finally, the server evaluates insurance premium discounts based on the driving data and notifies the driver of this information.

[0062] As described above, the system of the present invention ensures the safety of drivers and contributes to the prevention of traffic accidents.

[0063] The processing flow will be explained below.

[0064] Step 1:

[0065] The user turns on the AI-enabled glasses, which puts the system into initial setup mode.

[0066] Step 2:

[0067] The device automatically calibrates the electrooculography sensor in the nose pad and the camera in the frame, allowing the device to accurately detect the movement of the user's eyes and eyelids.

[0068] Step 3:

[0069] The device measures eye and eyelid movements in real time while driving, and electrooculography sensors and a camera device collect data that is sent to an artificial intelligence algorithm.

[0070] Step 4:

[0071] The device's AI algorithm analyzes the collected data to detect momentary sleep. Specifically, it analyzes patterns such as minute eye movements and eyelid closure, and if it determines that a momentary sleep has occurred, it moves on to the next action.

[0072] Step 5:

[0073] When the device detects momentary sleep, it immediately issues a visual and audible warning: an LED on the frame lights up to provide a visual warning, and an audible alert sounds from the speaker.

[0074] Step 6:

[0075] The device sends the collected sensor data, including eye movements, eyelid movements, and timing of momentary sleep, to an external server via a communication module.

[0076] Step 7:

[0077] The server stores and analyzes the received data, and the results of the analysis reveal the driver's driving characteristics and attentional patterns.

[0078] Step 8:

[0079] The server then provides feedback to the driver based on the analysis results, including advice on times when attention is waning and driving environments.

[0080] Step 9:

[0081] The device connects to the navigation system to obtain the latest traffic and congestion information, which is then announced to the driver through a speaker in the frame.

[0082] Step 10:

[0083] The server works with insurance companies to assess insurance premium discounts based on safe driving characteristics, and notifies the driver of the results, along with details of any discounts that may be applied.

[0084] Step 11:

[0085] After the user has worn the glasses for a day, the system automatically provides a summary of the analysis results, including the number of warnings encountered while driving and the total amount of driving data.

[0086] This not only helps the system prevent drowsy driving and support safe driving, but also provides feedback to drivers and insurance premium incentives.

[0087] Example 1

[0088] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0089] The purpose of this invention is to detect microsleep while driving and immediately warn the driver to prevent drowsiness at the wheel and ensure driver safety. Another objective is to improve driver safety by analyzing driving characteristics based on collected data and providing appropriate feedback. Furthermore, the invention aims to improve driving efficiency by providing route guidance and traffic congestion information in real time.

[0090] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0091] In this invention, the server includes a means including a photoelectric sensor and an image capture device that instantly detect eyelid and eye movement, a means including a machine learning algorithm that analyzes data captured by the sensor and the image capture device to detect momentary sleep, a light-emitting means that issues a visual warning in response to the detection of momentary sleep, a voice output means that issues an auditory warning in response to the detection of momentary sleep, a transmission means that communicates data from the sensor and the image capture device, and an external computing device that stores and analyzes the data transmitted by the transmission means. This enables momentary sleep while driving to be detected in real time and an immediate visual and auditory warning to be issued. Furthermore, the server analyzes driving characteristics and provides navigation information in cooperation with a route guidance system, thereby improving driver safety and efficiency.

[0092] A "photoelectric sensor" is a sensor that converts light into an electrical signal and detects the movement of the eyelids and eyeballs.

[0093] An "image capture device" is a device, such as a camera or image sensor, that is used to capture visual data.

[0094] A "machine learning algorithm" is an algorithm that recognizes patterns based on large amounts of data and makes predictions and classifications.

[0095] "Light-emitting means" means a means for issuing a visual warning to the driver using a light source such as an LED.

[0096] The "audio output means" refers to a means for issuing an audible warning to the driver using a speaker or a sound generating device.

[0097] The "transmission means" is a communication module for transmitting data to other devices or servers.

[0098] An "external computing device" is a server or cloud computer installed in a remote location, and is a device for storing and analyzing data.

[0099] "Driving characteristics" refers to a series of driving-related characteristics, such as the driver's driving behavior and patterns, for example, the frequency and time of day of momentary sleep.

[0100] A "route guidance system" is a system that provides navigation and provides optimal routes to a destination and traffic information.

[0101] This invention is a system that detects microsleep while driving and issues an immediate warning to ensure driver safety. This system consists of three main components: a terminal, a server, and a user.

[0102] Device details

[0103] The device is equipped with a photoelectric sensor and an image capture device to detect the movement of the driver's eyes and eyelids. This allows the device to monitor the driver's eyelid and eye movements in real time, and a machine learning algorithm is used to detect momentary sleep. Specifically, the photoelectric sensor captures minute eye movements, and the image capture device collects video data. The device processes this data in real time to determine whether momentary sleep is likely.

[0104] Example: While a user is driving, the device uses a photoelectric sensor and an image capture device to capture eye movement data. A machine learning algorithm analyzes this data and, if it detects prolonged blinking, determines that the user is fast asleep.

[0105] If a momentary sleep is detected, the device will immediately issue a warning using the "light emitting means" and "audio output means." Specifically, the LED light will turn on and a warning sound will be emitted from the speaker.

[0106] Server Details

[0107] The server is an "external computing device" that receives data sent from the terminal, stores it, and analyzes it. The terminal sends data to the server in real time using a "transmission means." The server stores this data in a database and analyzes the driver's "driving characteristics."

[0108] The analyzed data is then fed back to the driver as feedback to help them drive safely.

[0109] Example: The server analyzes a week's worth of driving data and finds a tendency for frequent short-term sleep to occur during certain times of the day. The server notifies the driver of this and warns them to be careful.

[0110] Furthermore, the server will be linked to the "route guidance system" to provide real-time route guidance and traffic congestion information. Navigation information will be announced to the user through the device's speaker.

[0111] How users use it

[0112] Users can use this system to ensure safety while driving. The device uses a photoelectric sensor and an image capture device to collect eye movement data and detects momentary sleep. If detected, users can receive an immediate warning.

[0113] In addition, safe and efficient driving is possible by receiving feedback based on driving behavior analysis provided by the server and navigation information from the route guidance system.

[0114] Example prompt sentence:

[0115] "Please explain in detail about the system that uses sensors and cameras to detect eyelid and eye movement, analyzes the data in real time, and issues a warning to prevent drowsy driving."

[0116] In this way, the system of the present invention ensures the safety of the driver and contributes greatly to preventing traffic accidents and improving driving efficiency.

[0117] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0118] Step 1:

[0119] As soon as the device starts driving, it activates a photoelectric sensor and image capture device to capture the driver's eye movement data in real time. This device samples data several dozen times per second. Specifically, image data that continuously captures the driver's eye movements and electrooculography data are input. This makes it possible to accurately detect the opening and closing of the eyelids and minute eye movements.

[0120] Step 2:

[0121] The device inputs the acquired eye movement data into a machine learning algorithm. This algorithm uses a pre-trained generative AI model to analyze instantaneous sleep patterns (e.g., prolonged blinking or gaze fixation). Specifically, it detects abnormal blink patterns and changes in gaze based on the input data to determine whether instantaneous sleep has occurred. The device then outputs whether instantaneous sleep has been detected.

[0122] Step 3:

[0123] When the machine learning algorithm detects a momentary sleep, the device immediately activates the light-emitting means (LED light) and the audio output means (speaker). This causes the LED light to light up and an alarm to sound from the speaker to notify the user of the momentary sleep. The output of this step is a visual and audio warning.

[0124] Step 4:

[0125] The device transmits all acquired eye movement data to an external computing device (server) via a transmission means. This data includes time, eyelid movement, detailed information on eye movement, etc. Secure HTTPS communication is used as the transmission protocol. This allows the server to receive detailed data in real time while driving.

[0126] Step 5:

[0127] The server stores the received data in a database. The stored data is then organized for analysis. For example, data cleaning is performed to remove incomplete or noisy data. This results in cleaned driving data.

[0128] Step 6:

[0129] The server analyzes driving characteristics using a generative AI model based on the accumulated data. This analysis determines patterns of instantaneous sleep during specific times and driving conditions, and generates useful feedback for the driver. For example, insights such as "instantaneous sleep occurs more frequently when driving at night" can be obtained. This feedback is notified to the user via email or a smartphone app.

[0130] Step 7:

[0131] The device is linked to a route guidance system. This link allows the device to obtain real-time traffic information and navigation data and announce it to the user through the device's speaker. For example, the device provides information on traffic congestion while driving and guidance on the optimal route to the destination. This allows the user to reach their destination efficiently and safely.

[0132] Step 8:

[0133] The server evaluates the driver's safe driving based on the driving data. Depending on the evaluation results, the server offers the user a discount on insurance premiums. The evaluation results are converted into a numerical value for the driver's safe driving level, and the data is sent to the insurance company based on that. Details of the discount are provided to the user via email or app notification.

[0134] (Application example 1)

[0135] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0136] Conventional driver monitoring systems have limitations in not only detecting momentary drowsiness but also in providing immediate warnings to drivers. There are also issues with how to provide feedback to drivers based on the analysis results of acquired driving data, and how to provide incentives such as insurance discounts. Furthermore, there is a lack of means to provide real-time traffic information obtained while driving. Under these circumstances, there are problems with ensuring driver safety and providing a comfortable driving environment.

[0137] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0138] In this invention, the server includes means including an electrooculography sensor and a camera device that instantly detects eyelid and eyeball movements, means including an artificial intelligence algorithm that analyzes data acquired by the sensor and camera device to detect momentary sleep, light emitting means that issues a visual warning in response to the detection of momentary sleep, audio output means that issues an audio warning in response to the detection of momentary sleep, communication means that communicates data from the sensor and camera device, external server means that accumulates and analyzes the data transmitted by the communication means, navigation linkage means that provides the latest traffic information acquired from the external server means via the audio output means, and means that provides insurance premium discount information based on driving data evaluated by the external server means. This makes it possible to instantly detect drowsy driving while driving and issue visual and audio warnings, and also to provide feedback and insurance premium discount information based on the results of analysis of driving characteristic data, and the latest navigation information in real time.

[0139] Definitions of important words

[0140] The eyelid is a thin layer of skin that protects the eye and allows you to blink.

[0141] The "eyeball" is a spherical organ that senses visual information and is the main part of the human eye.

[0142] An "electrooculography sensor" is a device that detects minute electrical activity in the eyelids and eyeballs and converts it into digital data.

[0143] A "camera device" is a device that captures images and videos and stores and analyzes them as digital data.

[0144] An "artificial intelligence algorithm" is a computational processing method that analyzes large amounts of data, learns patterns, and makes appropriate decisions and predictions.

[0145] A "light emitting means" is a device or system that emits light to visually convey information.

[0146] "Audio output means" refers to a device or system for conveying information using audio.

[0147] "Communication means" refers to the technology or equipment used to transmit data to other devices or servers.

[0148] The "external server means" is a remote server system for storing and analyzing data via a network.

[0149] The "navigation-linked means" is a system that links with the navigation system and provides traffic information and route guidance in real time.

[0150] "Driving characteristics" refers to a driver's behavioral characteristics such as driving habits, patterns, and reaction speed.

[0151] "Insurance premium discount information" is information about insurance premium discounts that are offered when criteria such as safe driving are met.

[0152] MODE FOR CARRYING OUT THE INVENTION

[0153] System configuration

[0154] This invention is a multi-function system for preventing drowsy driving and ensuring the safety of the driver. The elements that specifically constitute the system will be described below.

[0155] 1. Eyelid and eyeball movement detection device and artificial intelligence algorithm

[0156] The device is equipped with an electrooculography sensor and a camera device to instantly detect eyelid and eyeball movements. This allows for real-time monitoring of the driver's eyelid and eyeball movements while driving. The acquired eyeball movement data is analyzed by an artificial intelligence algorithm to detect microsleep.

[0157] 2. Visual and audible warning devices

[0158] When the device detects momentary drowsiness, an LED mounted on the frame lights up to provide a visual warning to the driver. In addition, an audible alert sounds from the speaker, providing an audible warning. This helps to draw the driver's attention and prevent drowsy driving.

[0159] 3. Data communication and external servers

[0160] The terminal includes a communication module for transmitting real-time acquired sensor data to an external server, which receives and stores the data. The stored data is used to analyze the driver's driving characteristics and provide feedback to the driver.

[0161] 4. Navigation linkage

[0162] The device is linked to the navigation system, allowing drivers to receive real-time navigation and traffic information through the speaker. For example, it will announce the optimal route to the destination or changes in road conditions.

[0163] 5. Premium discount function

[0164] The server provides drivers with discounts on insurance premiums through collaboration with insurance companies. Specifically, it evaluates safe driving based on driving data and discounts insurance premiums according to the evaluation results. This function provides drivers with an incentive to drive safely.

[0165] Program processing explanation

[0166] The device uses sensors and a camera device to collect eye movement data. An artificial intelligence algorithm then analyzes this data in real time to detect momentary sleep. When a momentary sleep is detected, the device will illuminate LEDs on the frame and play an audible alert through the speaker to provide visual and audio warnings.

[0167] The collected data is sent to an external server via the communication module. The server stores this data and analyzes driving characteristics. The analysis results are provided to the driver as feedback.

[0168] Furthermore, the terminal is linked to the navigation system, has the function of obtaining the latest traffic information and announcing navigation and congestion information to the driver through a speaker. The server also evaluates the driver's safe driving based on the driving data and notifies the driver of insurance premium discount information according to the evaluation results.

[0169] Specific usage example

[0170] For example, consider a case where a user drives a car wearing smart glasses equipped with this system. While driving, the device's electrooculography sensor and camera device collect eye movement data and detect momentary sleep. When momentary sleep is detected, the device's LED lights up and an alert sounds from the speaker to warn the driver. At the same time, the sensor data is sent from the device to a server, where it is stored and analyzed. The device then obtains the latest traffic information from the navigation system and announces it to the driver through the speaker. Finally, the server evaluates insurance premium discounts based on the driving data and notifies the driver of this information.

[0171] A specific example of a prompt sentence is "Analyze my current driving data and tell me the insurance premium discount rate."

[0172] As described above, the system of the present invention ensures the safety of drivers and makes it possible to prevent traffic accidents.

[0173] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0174] Program processing flow

[0175] Step 1:

[0176] Input: Data from electrooculography sensors and camera devices for real-time detection of eyelid and eye movements.

[0177] How it works: The device activates the electrooculography sensor and camera device to collect data on the driver's eyelid and eyeball movements in milliseconds, making it possible to understand the driver's eye movement, blink frequency, and eye fatigue.

[0178] Output: Collected eye movement data.

[0179] Step 2:

[0180] Input: Collected eye movement data.

[0181] How it works: The device's AI algorithm analyzes collected data in real time to detect microsleep. It processes the data by analyzing eye movement speed and eyelid opening / closing patterns, and determines that a microsleep has occurred if it exceeds a certain threshold.

[0182] Output: Instantaneous sleep detection result.

[0183] Step 3:

[0184] Input: Instantaneous sleep detection result.

[0185] Specific operation: When the device detects momentary sleep, it immediately activates the warning system. A visual warning is provided by the LED on the frame, and an audible warning is provided by the speaker. This immediately alerts the driver.

[0186] Output: Warning signals to the driver (visual and audible).

[0187] Step 4:

[0188] Input: Collected eye movement data.

[0189] Specific operation: The terminal uses the communication module to send the collected data to an external server. Specifically, the data is compressed and encrypted before being sent to the server via a communication protocol.

[0190] Output: Sending encrypted and compressed data to an external server.

[0191] Step 5:

[0192] Input: Data sent from the terminal.

[0193] How it works: The server accumulates the received data and analyzes the driver's driving characteristics. The analysis uses a generative AI model to identify multiple driving patterns and identify the driver's habits and characteristics.

[0194] Output: Insights into driving characteristics.

[0195] Step 6:

[0196] Input: Latest traffic and navigation data.

[0197] Specific operation: The terminal works in conjunction with the navigation system to announce real-time traffic information and the optimal route to the driver via voice output. In this process, the terminal analyzes the traffic information obtained via the communication module and selects the content that is most appropriate for the driver.

[0198] Output: Navigation information and traffic announcements to the driver.

[0199] Step 7:

[0200] Input: Accumulated driving data and driving characteristics analysis results.

[0201] Specific operation: The server evaluates safe driving based on driving data and generates insurance premium discount information based on the evaluation results. The generated insurance premium discount information is notified to the driver via the terminal.

[0202] Output: Notification of insurance premium discount information to the driver.

[0203] Through the above processing steps, this system ensures driver safety and makes it possible to prevent traffic accidents.

[0204] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0205] This invention is a system that includes an electrooculography sensor, a camera device, an artificial intelligence algorithm, an emotion engine, and a communication means and external server that link these together to prevent drowsy driving and ensure driver safety. The system aims to reduce the risk of accidents while driving by monitoring and evaluating the user's condition from multiple angles and issuing warnings at appropriate times.

[0206] System configuration

[0207] 1. Detection equipment and artificial intelligence algorithms

[0208] The device is equipped with an electrooculography sensor and a camera device to detect the driver's eye and eyelid movements. This allows real-time collection of eye movement data and eyelid movements while driving. This data is analyzed by an artificial intelligence algorithm to detect microsleep.

[0209] 2. Emotion Engine

[0210] The device also features an emotion engine that can recognize emotions by analyzing the user's facial expressions and voice. The emotion engine detects the user's stress level, fatigue level, and new emotional changes, and combines this data with other sensor information to provide a comprehensive condition assessment.

[0211] 3. Warning measures

[0212] When the device detects high stress or fatigue as assessed by the instantaneous sleep and emotion engine, it immediately issues visual and audible warnings. Specifically, the LED on the frame lights up to provide a visual warning, and an audible alert sounds from the speaker. Furthermore, if the emotion engine detects high stress or fatigue, the intensity of the warnings is automatically adjusted.

[0213] 4. Data communication and analysis

[0214] The device transmits the collected sensor data and emotional data via a communication module to an external server, which receives, stores, and analyzes the data. The analysis reveals the driver's driving characteristics, attentional tendencies, emotional state, and other information.

[0215] 5. Navigation system linkage

[0216] The device is equipped with a function that links with the navigation system, obtaining the latest traffic and congestion information and announcing it to the driver in real time through a speaker, allowing the driver to obtain the appropriate information to reach their destination safely.

[0217] 6. Premium discount function

[0218] The server works with insurance companies to offer drivers discounts on their insurance premiums. In particular, if the emotion engine records positive emotions, it will reflect that in the evaluation of insurance premiums. When a discount is applied, the details are notified to the driver.

[0219] Program processing

[0220] Data collection and instantaneous sleep detection

[0221] The device uses an electrooculography sensor and a camera to collect eye movement data. This data is analyzed in real time by an artificial intelligence algorithm to detect momentary sleep. The device's emotion engine also analyzes the user's facial expressions and voice data to recognize emotions. Emotional data is also collected in real time and used to comprehensively evaluate the user's state.

[0222] Actions triggered by the warning

[0223] If the device detects momentary sleep, it will simultaneously issue a visual and audible alert. The LED on the frame will light up and an alert sound will be emitted from the speaker. The intensity of the alert will also automatically adjust if the emotion engine detects high stress or fatigue.

[0224] Data transmission and analysis

[0225] The device sends the collected sensor data and emotional data to an external server, which then stores the received data and analyzes the driver's driving characteristics and emotional state. As a result, it becomes possible to understand the driver's behavioral patterns, changes in attention, and emotional trends.

[0226] Providing navigation information

[0227] The device connects to the navigation system to obtain the latest traffic information and announces traffic conditions and route guidance to the driver through a speaker. This information is provided in real time to assist the driver in making decisions.

[0228] Insurance premium discount notice

[0229] The server evaluates safe driving based on driving data and emotional data and offers discounts on insurance premiums. If the emotion engine detects positive emotions, it will also be reflected in the evaluation, and the driver will be notified when a discount is applied.

[0230] Specific usage example

[0231] The user drives a car using AI-enabled glasses. While driving, the device's electrooculography sensor and camera device collect eye and eyelid movements in real time, and an artificial intelligence algorithm detects momentary sleep. At the same time, the device's emotion engine analyzes the user's facial expressions and voice data to assess their stress and fatigue level. If momentary sleep or high stress is detected, an LED on the device's frame lights up and an alert sounds from the speaker to warn the user. All collected data is sent to a server in real time, where it is stored and analyzed. The device works in conjunction with the navigation system to provide the latest traffic information to support the driver. The server also evaluates the driver's safe driving, assesses insurance premium discounts, and notifies the driver.

[0232] In this way, the system is a powerful tool for monitoring the user's multifaceted condition and reducing risks while driving.

[0233] The processing flow will be explained below.

[0234] Step 1:

[0235] The user turns on the AI-enabled glasses, which puts the system into initial setup mode.

[0236] Step 2:

[0237] The device automatically calibrates the electrooculography sensor in the nose pad and the camera in the frame, a process that allows the device to accurately detect the movement of the user's eyes and eyelids.

[0238] Step 3:

[0239] The device measures eye and eyelid movements in real time while driving. Electrooculography sensors and a camera device collect data, which is then sent to an artificial intelligence algorithm for analysis.

[0240] Step 4:

[0241] The device's AI algorithm analyzes the collected data to detect microsleep. Specifically, it analyzes patterns such as minute eye movements and eyelid closure, and if it determines that a microsleep has occurred, it moves on to the next step.

[0242] Step 5:

[0243] The device's emotion engine analyzes the user's facial expressions and voice to assess their stress level and fatigue. This emotion data is then combined with instantaneous sleep detection data to provide a comprehensive assessment of their condition.

[0244] Step 6:

[0245] If the device detects a moment of deep sleep or high stress, it will immediately issue a visual and audible alert: an LED on the frame will light up and an alert sound will sound from the speaker.

[0246] Step 7:

[0247] The device transmits collected sensor data and emotional data, including eye movements, eyelid movements, timing of momentary sleep, and emotional state, to an external server via a communication module.

[0248] Step 8:

[0249] The server stores the received data and analyzes the driver's driving characteristics and emotional state, specifically analyzing trends in the driver's attention and changes in stress level.

[0250] Step 9:

[0251] The server then provides the driver with feedback based on the analysis results, including advice on times when attention may be waning due to long driving periods and advice on avoiding driving under high stress conditions.

[0252] Step 10:

[0253] The device connects to the navigation system to obtain the latest traffic and congestion information, which is then provided to the driver in real time through a speaker in the frame.

[0254] Step 11:

[0255] The server connects with insurance companies and evaluates insurance discounts based on safe driving and emotional state data. If the emotion engine records positive emotions, they are also reflected in the evaluation.

[0256] Step 12:

[0257] The server notifies the driver of details when the discount is applied, including the discount rate and the period of application.

[0258] Step 13:

[0259] After the user has worn the glasses for a day, the system automatically provides a summary of the analysis results, including the number of warnings encountered while driving, fluctuations in emotional state, and the total amount of driving data.

[0260] Through these steps, the system not only prevents drowsy driving, but also contributes to comprehensive driver condition management, supports safe driving, and provides incentives through insurance premium discounts.

[0261] Example 2

[0262] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0263] Drowsiness and high stress while driving are cited as causes of traffic accidents. Conventional driving monitoring systems detect momentary drowsiness by monitoring only eye and eyelid movements, but this alone makes it difficult to fully grasp the driver's overall condition. Furthermore, they lack the functionality to evaluate the driver's emotional state and fatigue level in real time and adjust the intensity of warnings according to the situation. Therefore, the objective of this invention is to reduce the risk of traffic accidents by detecting momentary drowsiness, evaluating the driver's emotional state and fatigue level, and issuing prompt and appropriate warnings to the driver.

[0264] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means including a sensor and a video capture device that instantly detect eyelid and eyeball movements, means including a machine learning algorithm that analyzes data acquired by the sensor and the video capture device to detect subtle sleep states, and means for recognizing emotions by analyzing the user's facial expressions and voice. This makes it possible to evaluate the driver's momentary sleep and emotional state from multiple angles and issue a warning at an appropriate time.

[0265] A "sensor" is a device that detects physical quantities such as light, sound, and electric current and converts them into electrical signals.

[0266] A "video capture device" is a device that records images and videos using optical means.

[0267] A "machine learning algorithm" is a computational method for learning patterns from data and using the results to analyze new data.

[0268] "Light emitting means" means a device or method for emitting light, primarily used to provide visual notification.

[0269] "Audio output means" refers to a device or method for generating and outputting audio.

[0270] "Information transmission means" means a means for transmitting data from one point to another, and includes wired or wireless communication means.

[0271] An "external computer system" is an external computing device or server accessed over a network and used for data storage and analysis.

[0272] An "emotion recognition means" is a device or system that analyzes a user's facial expressions and voice and identifies their emotional state.

[0273] A "means for adjusting the intensity of the warning" is a device or function for varying the intensity of the warning depending on the detected condition.

[0274] "Traffic information" refers to the latest travel information necessary for drivers, such as road conditions, traffic congestion, and accident information.

[0275] This invention is a system that detects drowsiness and high stress while driving in real time and issues a warning. To ensure safe driving, this system works by combining sensors, video capture devices, machine learning algorithms, emotion recognition means, information transmission means, and external computer systems.

[0276] System configuration and functions

[0277] Data collection

[0278] The device is equipped with high-precision sensors and video capture devices to detect eyelid and eyeball movements in real time. These devices collect the user's eye movement data and eyelid movements in real time. For example, when a user moves their eyes left and right while driving, the data is detected immediately.

[0279] Detecting subtle sleep states

[0280] The collected data is analyzed by a machine learning algorithm installed on the device, which detects microsleep based on the user's eye and eyelid movement patterns. For example, if the user's eyelids remain closed for a few seconds, the system will identify this as a microsleep.

[0281] emotion recognition

[0282] The device is equipped with a means of recognizing emotions by analyzing the user's facial expressions and voice. This data is also collected in real time to evaluate the user's stress level and fatigue. For example, if a frowning expression or a tired voice is detected, it is judged to be in a state of high stress or fatigue.

[0283] Warning

[0284] If a state of microsleep or high stress is detected, the device will issue a visual and audible warning. Specifically, an LED on the frame will light up and an alert sound will be emitted from the speaker. The intensity of the warning will also be automatically adjusted depending on the situation evaluated by the emotion recognition system. As a result, the user will receive a warning at the appropriate time and be able to respond quickly.

[0285] Data communication and the role of external servers

[0286] The sensor data and emotion data collected by the device are transmitted to an external computer system via an information transmission medium. The server receives, stores, and analyzes this data. The analysis results provide insights into the user's driving characteristics and emotional state.

[0287] Providing navigation information

[0288] The device works in conjunction with the navigation system to obtain the latest traffic and congestion information and announce it to the user in real time. For example, it obtains information about highway congestion and provides voice guidance such as, "You can avoid the congestion by getting off at the next exit."

[0289] Insurance premium discount notification

[0290] The server evaluates safe driving based on the collected and analyzed driving data and emotional data. For example, if the emotion recognition means frequently detects positive emotions, that evaluation will be reflected in the insurance premium discount. The user will be notified when the discount is applied.

[0291] Specific usage example

[0292] «Example of prompt sentences to input to a generative AI model»

[0293] "Please explain how the system works to detect drowsiness or high stress while driving and issue a warning."

[0294] "Please tell me the specific processing method of the driver monitoring system using electrooculography sensors and an emotion engine."

[0295] "Please explain with examples how this system collects data, issues alerts, and analyzes the data."

[0296] This system enables multifaceted condition monitoring while the user is driving, and serves as a powerful tool to support safe driving.

[0297] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0298] Step 1: Data collection

[0299] The device uses sensors and video capture devices to detect eyelid and eye movements in real time. These devices collect the user's eye movement data and eyelid movements in real time. Specifically, they detect the user's gaze movements and blinking movements and generate electrical signals corresponding to those movements. These electrical signals are stored as data. The input is the user's eye movement and eyelid movement, which are detected by the sensors. The output is eye movement data and eyelid movement data.

[0300] Step 2: Detecting Minor Sleep States

[0301] The device inputs the collected eye movement data and eyelid movements into a machine learning algorithm to detect microsleep. For example, if the data shows that the eyes do not blink or the eyelids remain closed for a long period of time, the algorithm determines this to be microsleep. The input is the eye movement data and eyelid movement data obtained in the data collection step. The output is the detection result of the microsleep state, specifically a judgment result such as "microsleep has been detected."

[0302] Step 3: Emotion Recognition

[0303] The device collects the user's facial expressions and voice using emotion recognition means. Based on this, an emotion recognition algorithm analyzes the user's emotional state. Specifically, it detects facial muscle movements and tone of voice to evaluate the user's stress level and fatigue level. The input is the user's facial expression data and voice data. The output is the evaluation result of the emotional state, such as "a high stress level has been detected."

[0304] Step 4: Send an alert

[0305] If the device detects momentary sleep or a high-stress state, it issues a visual and audible warning. Specifically, an LED on the frame lights up and an alert sound is emitted from the speaker. The strength of the warning is also automatically adjusted according to the state evaluated by the emotion recognition means. The input is the detection result of the minute sleep state and the evaluation result of the emotional state. The output is a visual and audible warning.

[0306] Step 5: Send data

[0307] The terminal transmits the collected sensor data and emotion data to an external computer system via an information transmission means. Specifically, the data is collectively organized into packets and transmitted to a server using wireless communication technology. The input is the sensor data and emotion data. The output is the transmitted data packets.

[0308] Step 6: Data analysis

[0309] The server stores the received data and analyzes the driving characteristics and emotional state. This allows the user's behavioral patterns, changes in attention, and emotional trends to be understood. The input is the transmitted sensor data and emotional data. The output is the analysis results of the driving characteristics and emotional state.

[0310] Step 7: Providing navigation information

[0311] The terminal works in conjunction with the navigation system to obtain the latest traffic information and announces it to the driver through a speaker. For example, it obtains information about highway congestion and provides guidance such as, "You can avoid the congestion by getting off at the next exit." The input is the latest traffic and congestion information. The output is a real-time voice announcement.

[0312] Step 8: Premium Discount Notification

[0313] The server evaluates safe driving based on the collected and analyzed driving data and emotional data. If the evaluation result of safe driving is positive, it applies an insurance premium discount and notifies the user. The input is the safe driving evaluation result. The output is a notification of the insurance premium discount.

[0314] (Application example 2)

[0315] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0316] As autonomous vehicles become more widespread, safety when drivers switch to manual driving has become an important issue. In particular, there is a risk of accidents occurring due to drivers falling asleep at the wheel after long hours of driving or due to overwork, or due to distraction caused by stress. For this reason, there is a demand for systems that can monitor the driver's condition from multiple angles and issue immediate warnings.

[0317] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means including a sensor and a camera that instantly detects eyelid and eyeball movements, means including an artificial intelligence algorithm that analyzes data acquired by the sensor and the camera to detect momentary sleep, means including an emotion engine that analyzes the user's facial expressions and voice to recognize emotions, means for issuing visual and audio warnings in response to detection of a high-stress state, communication means for communicating data from the sensor and the camera, and external information processing device means for storing and analyzing data transmitted by the communication means. This makes it possible to reduce the risk of an accident by evaluating the driver's condition in real time and issuing warnings when necessary.

[0318] The "sensor" is a device for detecting eyelid and eyeball movements in real time.

[0319] The "photography device" refers to a camera device that is combined with a sensor to visually capture the movement of the eyelids and eyeballs.

[0320] The "artificial intelligence algorithm" is an advanced computational method for analyzing acquired data and detecting instantaneous sleep.

[0321] The "emotion engine" is a system that analyzes the user's facial expressions and voice to recognize their emotions.

[0322] "Visual warning" means a warning that is visually conveyed to the user by means of light.

[0323] "Light emitting means" means a device that emits light to a user to effect a visual warning.

[0324] An "auditory warning" is a warning that is given to the user through the ears of the user by a voice output means.

[0325] "Audio output means" refers to a device that outputs a sound to the user to implement an audible warning.

[0326] The "communication means" is a device for transmitting data from the sensor and the image capturing device to an external information processing device.

[0327] The "external information processing device" is a server for storing and analyzing data transmitted by communication means.

[0328] "Driving characteristics" refers to the behavioral patterns and tendencies of a user when driving a vehicle.

[0329] "Attention trends" refers to information obtained by analyzing changes in a user's concentration and attentiveness.

[0330] "Emotional state" refers to the user's emotional state or state of mind.

[0331] A "navigation system" is a system that provides guidance on the current location of a vehicle and the route to a destination.

[0332] This invention is a system that supports safe driving by combining a sensor and a camera that detects eyelid and eyeball movements in real time, an artificial intelligence algorithm that analyzes the acquired data to detect momentary sleep, and an emotion engine. The system is configured as follows:

[0333] 1. Data Collection

[0334] The sensors and imaging devices detect the driver's eye and eyelid movements in real time using standard camera devices and application-specific electrooculography sensors. These devices constantly monitor and collect data on eye and eyelid movements while the driver is operating the vehicle.

[0335] 2. Instantaneous sleep and emotion detection

[0336] The collected data is analyzed in real time by an artificial intelligence algorithm installed on the device. If momentary sleep is detected, the system immediately issues a warning. In addition, an emotion engine analyzes the driver's facial expressions and voice data to assess their level of stress and fatigue. This emotion data is also collected in real time and used to comprehensively evaluate the driver's condition.

[0337] 3. Issuance of warnings

[0338] If momentary sleep or a high stress state is detected, the device will issue visual and audio warnings. Visual warnings are issued by lighting up the device's LED using the light-emitting means to alert the driver. Audio warnings are issued by emitting an alert sound from the speaker via the audio output means to alert the driver.

[0339] 4. Data communication and transmission to external servers

[0340] The collected sensor data and emotional data are transmitted to an external information processing device via communication means. The external information processing device stores this data and analyzes the driver's driving characteristics, attentional tendencies, and emotional state. This clarifies the driver's driving behavior patterns and provides insights for safe driving.

[0341] 5. Linkage with navigation systems

[0342] The device works in conjunction with the navigation system to provide up-to-date traffic information and route guidance, which is announced to the driver via voice output means and provided in real time, helping drivers reach their destination safely and efficiently.

[0343] 6. Insurance premium discounts

[0344] The external information processing device evaluates the driver's safe driving based on driving data and emotional data, and offers discounts on insurance premiums. If particularly positive emotions are recorded, this will be reflected in the insurance premium evaluation, and the driver will be notified when a discount is applied.

[0345] Specific examples

[0346] The user drives an autonomous vehicle using AI-enabled glasses. While driving, sensors and a camera collect eye and eyelid movements in real time, and an artificial intelligence algorithm detects momentary sleep. At the same time, an emotion engine analyzes the user's facial expressions and voice data to assess their stress and fatigue levels. If momentary sleep or high stress is detected, an LED on the device's frame lights up and an alert sounds from the speaker to warn the user. All collected data is sent in real time to an external information processing device for storage and analysis. The device works in conjunction with the navigation system to provide the latest traffic information and support the user. The external information processing device also evaluates the user's safe driving, assesses insurance premium discounts, and notifies the user.

[0347] Prompt Sentence Examples

[0348] "I want to develop an application that can detect drowsiness and fatigue while driving and issue a warning. I need a mechanism to analyze frames from the camera in real time using dlib for face detection, and a function to evaluate emotions and detect high stress and fatigue."

[0349] "This program uses an AI model to analyze eye movement data to detect momentary sleep and prevent drowsiness at the wheel. It also uses an emotion engine to assess the driver's stress and fatigue level and issue a warning if necessary. It also sends the collected data to a server and links it with the navigation system."

[0350] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0351] Step 1:

[0352] Data collection (terminal):

[0353] The device collects data using sensors and imaging devices that detect eyelid and eye movements in real time. Specifically, the imaging device (camera) captures video and the sensor records electrical potential changes. As input, real-time video and electrical potential data are used, collected every millisecond. As output, a collected dataset is generated and sent to the next analysis step.

[0354] Step 2:

[0355] Instantaneous sleep detection (device):

[0356] The device inputs the collected eye movement data and eyelid movement data into an artificial intelligence algorithm to detect signs of instantaneous sleep. Specifically, the data is first preprocessed and converted into an appropriate format. The artificial intelligence algorithm then analyzes this data and calculates the probability of instantaneous sleep. The input is the preprocessed eye movement and eyelid movement data, and the output is the instantaneous sleep detection result (e.g., a probability value or a warning flag).

[0357] Step 3:

[0358] Emotional rating (device):

[0359] The device collects the user's facial expression and voice data and inputs it into the emotion engine. The emotion engine analyzes this data and evaluates the user's stress and fatigue levels. Specifically, the emotion engine analyzes the video and voice data to identify the user's emotional state. The input is facial expression and voice data, and the output is the emotion evaluation result (e.g., stress level, fatigue level).

[0360] Step 4:

[0361] Issue a warning (terminal):

[0362] If a momentary sleep or high stress state is detected, the device issues a visual and audible warning. Specifically, the light-emitting means turns on an LED light and the audio output means plays an alert sound. The inputs are the momentary sleep detection result and the emotion evaluation result, and the outputs are visual and audible warnings (e.g., lighting up an LED, playing an alert sound).

[0363] Step 5:

[0364] Data communication (terminal):

[0365] The collected and analyzed sensor data and emotion data are transmitted to an external information processing device via a communication means. Specifically, the data is packetized using an appropriate protocol and transmitted over a network. The input is the sensor data and emotion data, and the output is the transmission of data to the external information processing device.

[0366] Step 6:

[0367] Data storage and analysis (server):

[0368] The external information processing device accumulates and analyzes the transmitted data. Specifically, it stores the data in a database and uses data analysis algorithms to analyze driving characteristics, attention trends, and emotional states. The input is the transmitted sensor data and emotional data, and the output is the analysis results (e.g., driving characteristics report, attention trends).

[0369] Step 7:

[0370] Providing navigation information (device):

[0371] The terminal works in conjunction with the navigation system to obtain the latest traffic information and provide it to the user in real time. Specifically, it obtains traffic information and route guidance information from the navigation system and announces it to the user through voice output means. The input is traffic information from the navigation system, and the output is voice traffic information and route guidance.

[0372] Step 8:

[0373] Insurance premium discount notification (server):

[0374] The external information processing device evaluates the driver's safe driving based on the analyzed data and notifies the user if an insurance premium discount is to be offered. Specifically, the device applies a safe driving evaluation algorithm, generates an evaluation result, and notifies the user of insurance premium discount information as needed. The input is the analysis result, and the output is an insurance premium discount notification.

[0375] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0376] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0377] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0378] [Second embodiment]

[0379] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0380] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0381] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0382] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0383] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0384] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0385] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0386] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0387] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0388] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0389] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0390] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0391] The present invention is a system for preventing drowsy driving and ensuring driver safety. The system includes an electrooculography sensor and camera device that detects eyelid and eyeball movements, an artificial intelligence algorithm that analyzes data in real time, visual and audio warning devices, a communication module that transmits data, data storage and analysis on an external server, linkage with a navigation system, and an insurance premium discount function.

[0392] System configuration

[0393] 1. Detection equipment and artificial intelligence algorithms

[0394] The device is equipped with an electrooculography sensor and a camera device to detect the driver's eye and eyelid movements. This allows for real-time monitoring of the driver's eye and eye movements while driving, and eye movement data is acquired. This data is analyzed by an artificial intelligence algorithm to detect microsleep.

[0395] 2. Visual and audible warnings

[0396] When the device detects momentary drowsiness, an LED mounted on the frame lights up to provide a visual warning to the driver. In addition, an audible alert sounds from the speaker, providing an audible warning. This helps to draw the driver's attention and prevent drowsy driving.

[0397] 3. Data communication and analysis

[0398] The terminal includes a communication module for transmitting real-time acquired sensor data to an external server, which receives and stores the data. The stored data is used to analyze the driver's driving characteristics and provide feedback to the driver.

[0399] 4. Navigation linkage

[0400] The device is linked to the navigation system, allowing drivers to receive real-time navigation and traffic information through the speaker. For example, it will announce the optimal route to the destination or changes in road conditions.

[0401] 5. Premium discount function

[0402] The server provides drivers with discounts on insurance premiums through collaboration with insurance companies. Specifically, it evaluates safe driving based on driving data and discounts insurance premiums according to the evaluation results. This function provides drivers with an incentive to drive safely.

[0403] Program processing

[0404] Data collection and instantaneous sleep detection

[0405] The device uses sensors and a camera to collect eye movement data, which is then analyzed in real time by an artificial intelligence algorithm to detect momentary sleep.

[0406] Actions triggered by the warning

[0407] When the device detects a moment of sleep, it immediately issues a visual and audible alert: the LED on the frame lights up and an alert sound comes from the speaker.

[0408] Data transmission and analysis

[0409] The device sends the collected data to an external server, which stores the data and analyzes driving characteristics. The analysis results are provided to the driver as feedback.

[0410] Providing navigation information

[0411] The device works in conjunction with the navigation system to obtain the latest traffic information, and announces navigation and traffic congestion information to the driver through a speaker.

[0412] Insurance premium discount notice

[0413] The server evaluates the driver's safe driving based on the driving data and provides discounts on insurance premiums according to the evaluation results. Discount information is notified to the driver.

[0414] Specific usage example

[0415] Suppose a user is wearing AI-enabled glasses and driving a car. While driving, the device's electrooculography sensor and camera device collect eye movement data and detect momentary sleep. When momentary sleep is detected, the device's LED lights up and an alert sounds from the speaker to warn the driver. At the same time, the sensor data is sent from the device to a server, where it is accumulated and analyzed. Next, the device obtains the latest traffic information from the navigation system and announces it to the driver through the speaker. Finally, the server evaluates insurance premium discounts based on the driving data and notifies the driver of this information.

[0416] As described above, the system of the present invention ensures the safety of drivers and contributes to the prevention of traffic accidents.

[0417] The processing flow will be explained below.

[0418] Step 1:

[0419] The user turns on the AI-enabled glasses, which puts the system into initial setup mode.

[0420] Step 2:

[0421] The device automatically calibrates the electrooculography sensor in the nose pad and the camera in the frame, allowing the device to accurately detect the movement of the user's eyes and eyelids.

[0422] Step 3:

[0423] The device measures eye and eyelid movements in real time while driving, and electrooculography sensors and a camera device collect data that is sent to an artificial intelligence algorithm.

[0424] Step 4:

[0425] The device's AI algorithm analyzes the collected data to detect momentary sleep. Specifically, it analyzes patterns such as minute eye movements and eyelid closure, and if it determines that a momentary sleep has occurred, it moves on to the next action.

[0426] Step 5:

[0427] When the device detects momentary sleep, it immediately issues a visual and audible warning: an LED on the frame lights up to provide a visual warning, and an audible alert sounds from the speaker.

[0428] Step 6:

[0429] The device sends the collected sensor data, including eye movements, eyelid movements, and timing of momentary sleep, to an external server via a communication module.

[0430] Step 7:

[0431] The server stores and analyzes the received data, and the results of the analysis reveal the driver's driving characteristics and attentional patterns.

[0432] Step 8:

[0433] The server then provides feedback to the driver based on the analysis results, including advice on times when attention is waning and driving environments.

[0434] Step 9:

[0435] The device connects to the navigation system to obtain the latest traffic and congestion information, which is then announced to the driver through a speaker in the frame.

[0436] Step 10:

[0437] The server works with insurance companies to assess insurance premium discounts based on safe driving characteristics, and notifies the driver of the results, along with details of any discounts that may be applied.

[0438] Step 11:

[0439] After the user has worn the glasses for a day, the system automatically provides a summary of the analysis results, including the number of warnings encountered while driving and the total amount of driving data.

[0440] This not only helps the system prevent drowsy driving and support safe driving, but also provides feedback to drivers and insurance premium incentives.

[0441] Example 1

[0442] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0443] The purpose of this invention is to detect microsleep while driving and immediately warn the driver to prevent drowsiness at the wheel and ensure driver safety. Another objective is to improve driver safety by analyzing driving characteristics based on collected data and providing appropriate feedback. Furthermore, the invention aims to improve driving efficiency by providing route guidance and traffic congestion information in real time.

[0444] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0445] In this invention, the server includes a means including a photoelectric sensor and an image capture device that instantly detect eyelid and eye movement, a means including a machine learning algorithm that analyzes data captured by the sensor and the image capture device to detect momentary sleep, a light-emitting means that issues a visual warning in response to the detection of momentary sleep, a voice output means that issues an auditory warning in response to the detection of momentary sleep, a transmission means that communicates data from the sensor and the image capture device, and an external computing device that stores and analyzes the data transmitted by the transmission means. This enables momentary sleep while driving to be detected in real time and an immediate visual and auditory warning to be issued. Furthermore, the server analyzes driving characteristics and provides navigation information in cooperation with a route guidance system, thereby improving driver safety and efficiency.

[0446] A "photoelectric sensor" is a sensor that converts light into an electrical signal and detects the movement of the eyelids and eyeballs.

[0447] An "image capture device" is a device, such as a camera or image sensor, that is used to capture visual data.

[0448] A "machine learning algorithm" is an algorithm that recognizes patterns based on large amounts of data and makes predictions and classifications.

[0449] "Light-emitting means" means a means for issuing a visual warning to the driver using a light source such as an LED.

[0450] The "audio output means" refers to a means for issuing an audible warning to the driver using a speaker or a sound generating device.

[0451] The "transmission means" is a communication module for transmitting data to other devices or servers.

[0452] An "external computing device" is a server or cloud computer installed in a remote location, and is a device for storing and analyzing data.

[0453] "Driving characteristics" refers to a series of driving-related characteristics, such as the driver's driving behavior and patterns, for example, the frequency and time of day of momentary sleep.

[0454] A "route guidance system" is a system that provides navigation and provides optimal routes to a destination and traffic information.

[0455] This invention is a system that detects microsleep while driving and issues an immediate warning to ensure driver safety. This system consists of three main components: a terminal, a server, and a user.

[0456] Device details

[0457] The device is equipped with a photoelectric sensor and an image capture device to detect the movement of the driver's eyes and eyelids. This allows the device to monitor the driver's eyelid and eye movements in real time, and a machine learning algorithm is used to detect momentary sleep. Specifically, the photoelectric sensor captures minute eye movements, and the image capture device collects video data. The device processes this data in real time to determine whether momentary sleep is likely.

[0458] Example: While a user is driving, the device uses a photoelectric sensor and an image capture device to capture eye movement data. A machine learning algorithm analyzes this data and, if it detects prolonged blinking, determines that the user is fast asleep.

[0459] If a momentary sleep is detected, the device will immediately issue a warning using the "light emitting means" and "audio output means." Specifically, the LED light will turn on and a warning sound will be emitted from the speaker.

[0460] Server Details

[0461] The server is an "external computing device" that receives data sent from the terminal, stores it, and analyzes it. The terminal sends data to the server in real time using a "transmission means." The server stores this data in a database and analyzes the driver's "driving characteristics."

[0462] The analyzed data is then fed back to the driver as feedback to help them drive safely.

[0463] Example: The server analyzes a week's worth of driving data and finds a tendency for frequent short-term sleep to occur during certain times of the day. The server notifies the driver of this and warns them to be careful.

[0464] Furthermore, the server will be linked to the "route guidance system" to provide real-time route guidance and traffic congestion information. Navigation information will be announced to the user through the device's speaker.

[0465] How users use it

[0466] Users can use this system to ensure safety while driving. The device uses a photoelectric sensor and an image capture device to collect eye movement data and detects momentary sleep. If detected, users can receive an immediate warning.

[0467] In addition, safe and efficient driving is possible by receiving feedback based on driving behavior analysis provided by the server and navigation information from the route guidance system.

[0468] Example prompt sentence:

[0469] "Please explain in detail about the system that uses sensors and cameras to detect eyelid and eye movement, analyzes the data in real time, and issues a warning to prevent drowsy driving."

[0470] In this way, the system of the present invention ensures the safety of the driver and contributes greatly to preventing traffic accidents and improving driving efficiency.

[0471] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0472] Step 1:

[0473] As soon as the device starts driving, it activates a photoelectric sensor and image capture device to capture the driver's eye movement data in real time. This device samples data several dozen times per second. Specifically, image data that continuously captures the driver's eye movements and electrooculography data are input. This makes it possible to accurately detect the opening and closing of the eyelids and minute eye movements.

[0474] Step 2:

[0475] The device inputs the acquired eye movement data into a machine learning algorithm. This algorithm uses a pre-trained generative AI model to analyze instantaneous sleep patterns (e.g., prolonged blinking or gaze fixation). Specifically, it detects abnormal blink patterns and changes in gaze based on the input data to determine whether instantaneous sleep has occurred. The device then outputs whether instantaneous sleep has been detected.

[0476] Step 3:

[0477] When the machine learning algorithm detects a momentary sleep, the device immediately activates the light-emitting means (LED light) and the audio output means (speaker). This causes the LED light to light up and an alarm to sound from the speaker to notify the user of the momentary sleep. The output of this step is a visual and audio warning.

[0478] Step 4:

[0479] The device transmits all acquired eye movement data to an external computing device (server) via a transmission means. This data includes time, eyelid movement, detailed information on eye movement, etc. Secure HTTPS communication is used as the transmission protocol. This allows the server to receive detailed data in real time while driving.

[0480] Step 5:

[0481] The server stores the received data in a database. The stored data is then organized for analysis. For example, data cleaning is performed to remove incomplete or noisy data. This results in cleaned driving data.

[0482] Step 6:

[0483] The server analyzes driving characteristics using a generative AI model based on the accumulated data. This analysis determines patterns of instantaneous sleep during specific times and driving conditions, and generates useful feedback for the driver. For example, insights such as "instantaneous sleep occurs more frequently when driving at night" can be obtained. This feedback is notified to the user via email or a smartphone app.

[0484] Step 7:

[0485] The device is linked to a route guidance system. This link allows the device to obtain real-time traffic information and navigation data and announce it to the user through the device's speaker. For example, the device provides information on traffic congestion while driving and guidance on the optimal route to the destination. This allows the user to reach their destination efficiently and safely.

[0486] Step 8:

[0487] The server evaluates the driver's safe driving based on the driving data. Depending on the evaluation results, the server offers the user a discount on insurance premiums. The evaluation results are converted into a numerical value for the driver's safe driving level, and the data is sent to the insurance company based on that. Details of the discount are provided to the user via email or app notification.

[0488] (Application example 1)

[0489] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0490] Conventional driver monitoring systems have limitations in not only detecting momentary drowsiness but also in providing immediate warnings to drivers. There are also issues with how to provide feedback to drivers based on the analysis results of acquired driving data, and how to provide incentives such as insurance discounts. Furthermore, there is a lack of means to provide real-time traffic information obtained while driving. Under these circumstances, there are problems with ensuring driver safety and providing a comfortable driving environment.

[0491] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0492] In this invention, the server includes means including an electrooculography sensor and a camera device that instantly detects eyelid and eyeball movements, means including an artificial intelligence algorithm that analyzes data acquired by the sensor and camera device to detect momentary sleep, light emitting means that issues a visual warning in response to the detection of momentary sleep, audio output means that issues an audio warning in response to the detection of momentary sleep, communication means that communicates data from the sensor and camera device, external server means that accumulates and analyzes the data transmitted by the communication means, navigation linkage means that provides the latest traffic information acquired from the external server means via the audio output means, and means that provides insurance premium discount information based on driving data evaluated by the external server means. This makes it possible to instantly detect drowsy driving while driving and issue visual and audio warnings, and also to provide feedback and insurance premium discount information based on the results of analysis of driving characteristic data, and the latest navigation information in real time.

[0493] Definitions of important words

[0494] The eyelid is a thin layer of skin that protects the eye and allows you to blink.

[0495] The "eyeball" is a spherical organ that senses visual information and is the main part of the human eye.

[0496] An "electrooculography sensor" is a device that detects minute electrical activity in the eyelids and eyeballs and converts it into digital data.

[0497] A "camera device" is a device that captures images and videos and stores and analyzes them as digital data.

[0498] An "artificial intelligence algorithm" is a computational processing method that analyzes large amounts of data, learns patterns, and makes appropriate decisions and predictions.

[0499] A "light emitting means" is a device or system that emits light to visually convey information.

[0500] "Audio output means" refers to a device or system for conveying information using audio.

[0501] "Communication means" refers to the technology or equipment used to transmit data to other devices or servers.

[0502] The "external server means" is a remote server system for storing and analyzing data via a network.

[0503] The "navigation-linked means" is a system that links with the navigation system and provides traffic information and route guidance in real time.

[0504] "Driving characteristics" refers to a driver's behavioral characteristics such as driving habits, patterns, and reaction speed.

[0505] "Insurance premium discount information" is information about insurance premium discounts that are offered when criteria such as safe driving are met.

[0506] MODE FOR CARRYING OUT THE INVENTION

[0507] System configuration

[0508] This invention is a multi-function system for preventing drowsy driving and ensuring the safety of the driver. The elements that specifically constitute the system will be described below.

[0509] 1. Eyelid and eyeball movement detection device and artificial intelligence algorithm

[0510] The device is equipped with an electrooculography sensor and a camera device to instantly detect eyelid and eyeball movements. This allows for real-time monitoring of the driver's eyelid and eyeball movements while driving. The acquired eyeball movement data is analyzed by an artificial intelligence algorithm to detect microsleep.

[0511] 2. Visual and audible warning devices

[0512] When the device detects momentary drowsiness, an LED mounted on the frame lights up to provide a visual warning to the driver. In addition, an audible alert sounds from the speaker, providing an audible warning. This helps to draw the driver's attention and prevent drowsy driving.

[0513] 3. Data communication and external servers

[0514] The terminal includes a communication module for transmitting real-time acquired sensor data to an external server, which receives and stores the data. The stored data is used to analyze the driver's driving characteristics and provide feedback to the driver.

[0515] 4. Navigation linkage

[0516] The device is linked to the navigation system, allowing drivers to receive real-time navigation and traffic information through the speaker. For example, it will announce the optimal route to the destination or changes in road conditions.

[0517] 5. Premium discount function

[0518] The server provides drivers with discounts on insurance premiums through collaboration with insurance companies. Specifically, it evaluates safe driving based on driving data and discounts insurance premiums according to the evaluation results. This function provides drivers with an incentive to drive safely.

[0519] Program processing explanation

[0520] The device uses sensors and a camera device to collect eye movement data. An artificial intelligence algorithm then analyzes this data in real time to detect momentary sleep. When a momentary sleep is detected, the device will illuminate LEDs on the frame and play an audible alert through the speaker to provide visual and audio warnings.

[0521] The collected data is sent to an external server via the communication module. The server stores this data and analyzes driving characteristics. The analysis results are provided to the driver as feedback.

[0522] Furthermore, the terminal is linked to the navigation system, has the function of obtaining the latest traffic information and announcing navigation and congestion information to the driver through a speaker. The server also evaluates the driver's safe driving based on the driving data and notifies the driver of insurance premium discount information according to the evaluation results.

[0523] Specific usage example

[0524] For example, consider a case where a user drives a car wearing smart glasses equipped with this system. While driving, the device's electrooculography sensor and camera device collect eye movement data and detect momentary sleep. When momentary sleep is detected, the device's LED lights up and an alert sounds from the speaker to warn the driver. At the same time, the sensor data is sent from the device to a server, where it is stored and analyzed. The device then obtains the latest traffic information from the navigation system and announces it to the driver through the speaker. Finally, the server evaluates insurance premium discounts based on the driving data and notifies the driver of this information.

[0525] A specific example of a prompt sentence is "Analyze my current driving data and tell me the insurance premium discount rate."

[0526] As described above, the system of the present invention ensures the safety of drivers and makes it possible to prevent traffic accidents.

[0527] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0528] Program processing flow

[0529] Step 1:

[0530] Input: Data from electrooculography sensors and camera devices for real-time detection of eyelid and eye movements.

[0531] How it works: The device activates the electrooculography sensor and camera device to collect data on the driver's eyelid and eyeball movements in milliseconds, making it possible to understand the driver's eye movement, blink frequency, and eye fatigue.

[0532] Output: Collected eye movement data.

[0533] Step 2:

[0534] Input: Collected eye movement data.

[0535] How it works: The device's AI algorithm analyzes collected data in real time to detect microsleep. It processes the data by analyzing eye movement speed and eyelid opening / closing patterns, and determines that a microsleep has occurred if it exceeds a certain threshold.

[0536] Output: Instantaneous sleep detection result.

[0537] Step 3:

[0538] Input: Instantaneous sleep detection result.

[0539] Specific operation: When the device detects momentary sleep, it immediately activates the warning system. A visual warning is provided by the LED on the frame, and an audible warning is provided by the speaker. This immediately alerts the driver.

[0540] Output: Warning signals to the driver (visual and audible).

[0541] Step 4:

[0542] Input: Collected eye movement data.

[0543] Specific operation: The terminal uses the communication module to send the collected data to an external server. Specifically, the data is compressed and encrypted before being sent to the server via a communication protocol.

[0544] Output: Sending encrypted and compressed data to an external server.

[0545] Step 5:

[0546] Input: Data sent from the terminal.

[0547] How it works: The server accumulates the received data and analyzes the driver's driving characteristics. The analysis uses a generative AI model to identify multiple driving patterns and identify the driver's habits and characteristics.

[0548] Output: Insights into driving characteristics.

[0549] Step 6:

[0550] Input: Latest traffic and navigation data.

[0551] Specific operation: The terminal works in conjunction with the navigation system to announce real-time traffic information and the optimal route to the driver via voice output. In this process, the terminal analyzes the traffic information obtained via the communication module and selects the content that is most appropriate for the driver.

[0552] Output: Navigation information and traffic announcements to the driver.

[0553] Step 7:

[0554] Input: Accumulated driving data and driving characteristics analysis results.

[0555] Specific operation: The server evaluates safe driving based on driving data and generates insurance premium discount information based on the evaluation results. The generated insurance premium discount information is notified to the driver via the terminal.

[0556] Output: Notification of insurance premium discount information to the driver.

[0557] Through the above processing steps, this system ensures driver safety and makes it possible to prevent traffic accidents.

[0558] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0559] This invention is a system that includes an electrooculography sensor, a camera device, an artificial intelligence algorithm, an emotion engine, and a communication means and external server that link these together to prevent drowsy driving and ensure driver safety. The system aims to reduce the risk of accidents while driving by monitoring and evaluating the user's condition from multiple angles and issuing warnings at appropriate times.

[0560] System configuration

[0561] 1. Detection equipment and artificial intelligence algorithms

[0562] The device is equipped with an electrooculography sensor and a camera device to detect the driver's eye and eyelid movements. This allows real-time collection of eye movement data and eyelid movements while driving. This data is analyzed by an artificial intelligence algorithm to detect microsleep.

[0563] 2. Emotion Engine

[0564] The device also features an emotion engine that can recognize emotions by analyzing the user's facial expressions and voice. The emotion engine detects the user's stress level, fatigue level, and new emotional changes, and combines this data with other sensor information to provide a comprehensive condition assessment.

[0565] 3. Warning measures

[0566] When the device detects high stress or fatigue as assessed by the instantaneous sleep and emotion engine, it immediately issues visual and audible warnings. Specifically, the LED on the frame lights up to provide a visual warning, and an audible alert sounds from the speaker. Furthermore, if the emotion engine detects high stress or fatigue, the intensity of the warnings is automatically adjusted.

[0567] 4. Data communication and analysis

[0568] The device transmits the collected sensor data and emotional data via a communication module to an external server, which receives, stores, and analyzes the data. The analysis reveals the driver's driving characteristics, attentional tendencies, emotional state, and other information.

[0569] 5. Navigation system linkage

[0570] The device is equipped with a function that links with the navigation system, obtaining the latest traffic and congestion information and announcing it to the driver in real time through a speaker, allowing the driver to obtain the appropriate information to reach their destination safely.

[0571] 6. Premium discount function

[0572] The server works with insurance companies to offer drivers discounts on their insurance premiums. In particular, if the emotion engine records positive emotions, it will reflect that in the evaluation of insurance premiums. When a discount is applied, the details are notified to the driver.

[0573] Program processing

[0574] Data collection and instantaneous sleep detection

[0575] The device uses an electrooculography sensor and a camera to collect eye movement data. This data is analyzed in real time by an artificial intelligence algorithm to detect momentary sleep. The device's emotion engine also analyzes the user's facial expressions and voice data to recognize emotions. Emotional data is also collected in real time and used to comprehensively evaluate the user's state.

[0576] Actions triggered by the warning

[0577] If the device detects momentary sleep, it will simultaneously issue a visual and audible alert. The LED on the frame will light up and an alert sound will be emitted from the speaker. The intensity of the alert will also automatically adjust if the emotion engine detects high stress or fatigue.

[0578] Data transmission and analysis

[0579] The device sends the collected sensor data and emotional data to an external server, which then stores the received data and analyzes the driver's driving characteristics and emotional state. As a result, it becomes possible to understand the driver's behavioral patterns, changes in attention, and emotional trends.

[0580] Providing navigation information

[0581] The device connects to the navigation system to obtain the latest traffic information and announces traffic conditions and route guidance to the driver through a speaker. This information is provided in real time to assist the driver in making decisions.

[0582] Insurance premium discount notice

[0583] The server evaluates safe driving based on driving data and emotional data and offers discounts on insurance premiums. If the emotion engine detects positive emotions, it will also be reflected in the evaluation, and the driver will be notified when a discount is applied.

[0584] Specific usage example

[0585] The user drives a car using AI-enabled glasses. While driving, the device's electrooculography sensor and camera device collect eye and eyelid movements in real time, and an artificial intelligence algorithm detects momentary sleep. At the same time, the device's emotion engine analyzes the user's facial expressions and voice data to assess their stress and fatigue level. If momentary sleep or high stress is detected, an LED on the device's frame lights up and an alert sounds from the speaker to warn the user. All collected data is sent to a server in real time, where it is stored and analyzed. The device works in conjunction with the navigation system to provide the latest traffic information to support the driver. The server also evaluates the driver's safe driving, assesses insurance premium discounts, and notifies the driver.

[0586] In this way, the system is a powerful tool for monitoring the user's multifaceted condition and reducing risks while driving.

[0587] The processing flow will be explained below.

[0588] Step 1:

[0589] The user turns on the AI-enabled glasses, which puts the system into initial setup mode.

[0590] Step 2:

[0591] The device automatically calibrates the electrooculography sensor in the nose pad and the camera in the frame, a process that allows the device to accurately detect the movement of the user's eyes and eyelids.

[0592] Step 3:

[0593] The device measures eye and eyelid movements in real time while driving. Electrooculography sensors and a camera device collect data, which is then sent to an artificial intelligence algorithm for analysis.

[0594] Step 4:

[0595] The device's AI algorithm analyzes the collected data to detect microsleep. Specifically, it analyzes patterns such as minute eye movements and eyelid closure, and if it determines that a microsleep has occurred, it moves on to the next step.

[0596] Step 5:

[0597] The device's emotion engine analyzes the user's facial expressions and voice to assess their stress level and fatigue. This emotion data is then combined with instantaneous sleep detection data to provide a comprehensive assessment of their condition.

[0598] Step 6:

[0599] If the device detects a moment of deep sleep or high stress, it will immediately issue a visual and audible alert: an LED on the frame will light up and an alert sound will sound from the speaker.

[0600] Step 7:

[0601] The device transmits collected sensor data and emotional data, including eye movements, eyelid movements, timing of momentary sleep, and emotional state, to an external server via a communication module.

[0602] Step 8:

[0603] The server stores the received data and analyzes the driver's driving characteristics and emotional state, specifically analyzing trends in the driver's attention and changes in stress level.

[0604] Step 9:

[0605] The server then provides the driver with feedback based on the analysis results, including advice on times when attention may be waning due to long driving periods and advice on avoiding driving under high stress conditions.

[0606] Step 10:

[0607] The device connects to the navigation system to obtain the latest traffic and congestion information, which is then provided to the driver in real time through a speaker in the frame.

[0608] Step 11:

[0609] The server connects with insurance companies and evaluates insurance discounts based on safe driving and emotional state data. If the emotion engine records positive emotions, they are also reflected in the evaluation.

[0610] Step 12:

[0611] The server notifies the driver of details when the discount is applied, including the discount rate and the period of application.

[0612] Step 13:

[0613] After the user has worn the glasses for a day, the system automatically provides a summary of the analysis results, including the number of warnings encountered while driving, fluctuations in emotional state, and the total amount of driving data.

[0614] Through these steps, the system not only prevents drowsy driving, but also contributes to comprehensive driver condition management, supports safe driving, and provides incentives through insurance premium discounts.

[0615] Example 2

[0616] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0617] Drowsiness and high stress while driving are cited as causes of traffic accidents. Conventional driving monitoring systems detect momentary drowsiness by monitoring only eye and eyelid movements, but this alone makes it difficult to fully grasp the driver's overall condition. Furthermore, they lack the functionality to evaluate the driver's emotional state and fatigue level in real time and adjust the intensity of warnings according to the situation. Therefore, the objective of this invention is to reduce the risk of traffic accidents by detecting momentary drowsiness, evaluating the driver's emotional state and fatigue level, and issuing prompt and appropriate warnings to the driver.

[0618] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means including a sensor and a video capture device that instantly detect eyelid and eyeball movements, means including a machine learning algorithm that analyzes data acquired by the sensor and the video capture device to detect subtle sleep states, and means for recognizing emotions by analyzing the user's facial expressions and voice. This makes it possible to evaluate the driver's momentary sleep and emotional state from multiple angles and issue a warning at an appropriate time.

[0619] A "sensor" is a device that detects physical quantities such as light, sound, and electric current and converts them into electrical signals.

[0620] A "video capture device" is a device that records images and videos using optical means.

[0621] A "machine learning algorithm" is a computational method for learning patterns from data and using the results to analyze new data.

[0622] "Light emitting means" means a device or method for emitting light, primarily used to provide visual notification.

[0623] "Audio output means" refers to a device or method for generating and outputting audio.

[0624] "Information transmission means" means a means for transmitting data from one point to another, and includes wired or wireless communication means.

[0625] An "external computer system" is an external computing device or server accessed over a network and used for data storage and analysis.

[0626] An "emotion recognition means" is a device or system that analyzes a user's facial expressions and voice and identifies their emotional state.

[0627] A "means for adjusting the intensity of the warning" is a device or function for varying the intensity of the warning depending on the detected condition.

[0628] "Traffic information" refers to the latest travel information necessary for drivers, such as road conditions, traffic congestion, and accident information.

[0629] This invention is a system that detects drowsiness and high stress while driving in real time and issues a warning. To ensure safe driving, this system works by combining sensors, video capture devices, machine learning algorithms, emotion recognition means, information transmission means, and external computer systems.

[0630] System configuration and functions

[0631] Data collection

[0632] The device is equipped with high-precision sensors and video capture devices to detect eyelid and eyeball movements in real time. These devices collect the user's eye movement data and eyelid movements in real time. For example, when a user moves their eyes left and right while driving, the data is detected immediately.

[0633] Detecting subtle sleep states

[0634] The collected data is analyzed by a machine learning algorithm installed on the device, which detects microsleep based on the user's eye and eyelid movement patterns. For example, if the user's eyelids remain closed for a few seconds, the system will identify this as a microsleep.

[0635] emotion recognition

[0636] The device is equipped with a means of recognizing emotions by analyzing the user's facial expressions and voice. This data is also collected in real time to evaluate the user's stress level and fatigue. For example, if a frowning expression or a tired voice is detected, it is judged to be in a state of high stress or fatigue.

[0637] Warning

[0638] If a state of microsleep or high stress is detected, the device will issue a visual and audible warning. Specifically, an LED on the frame will light up and an alert sound will be emitted from the speaker. The intensity of the warning will also be automatically adjusted depending on the situation evaluated by the emotion recognition system. As a result, the user will receive a warning at the appropriate time and be able to respond quickly.

[0639] Data communication and the role of external servers

[0640] The sensor data and emotion data collected by the device are transmitted to an external computer system via an information transmission medium. The server receives, stores, and analyzes this data. The analysis results provide insights into the user's driving characteristics and emotional state.

[0641] Providing navigation information

[0642] The device works in conjunction with the navigation system to obtain the latest traffic and congestion information and announce it to the user in real time. For example, it obtains information about highway congestion and provides voice guidance such as, "You can avoid the congestion by getting off at the next exit."

[0643] Insurance premium discount notification

[0644] The server evaluates safe driving based on the collected and analyzed driving data and emotional data. For example, if the emotion recognition means frequently detects positive emotions, that evaluation will be reflected in the insurance premium discount. The user will be notified when the discount is applied.

[0645] Specific usage example

[0646] «Example of prompt sentences to input to a generative AI model»

[0647] "Please explain how the system works to detect drowsiness or high stress while driving and issue a warning."

[0648] "Please tell me the specific processing method of the driver monitoring system using electrooculography sensors and an emotion engine."

[0649] "Please explain with examples how this system collects data, issues alerts, and analyzes the data."

[0650] This system enables multifaceted condition monitoring while the user is driving, and serves as a powerful tool to support safe driving.

[0651] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0652] Step 1: Data collection

[0653] The device uses sensors and video capture devices to detect eyelid and eye movements in real time. These devices collect the user's eye movement data and eyelid movements in real time. Specifically, they detect the user's gaze movements and blinking movements and generate electrical signals corresponding to those movements. These electrical signals are stored as data. The input is the user's eye movement and eyelid movement, which are detected by the sensors. The output is eye movement data and eyelid movement data.

[0654] Step 2: Detecting Minor Sleep States

[0655] The device inputs the collected eye movement data and eyelid movements into a machine learning algorithm to detect microsleep. For example, if the data shows that the eyes do not blink or the eyelids remain closed for a long period of time, the algorithm determines this to be microsleep. The input is the eye movement data and eyelid movement data obtained in the data collection step. The output is the detection result of the microsleep state, specifically a judgment result such as "microsleep has been detected."

[0656] Step 3: Emotion Recognition

[0657] The device collects the user's facial expressions and voice using emotion recognition means. Based on this, an emotion recognition algorithm analyzes the user's emotional state. Specifically, it detects facial muscle movements and tone of voice to evaluate the user's stress level and fatigue level. The input is the user's facial expression data and voice data. The output is the evaluation result of the emotional state, such as "a high stress level has been detected."

[0658] Step 4: Send an alert

[0659] If the device detects momentary sleep or a high-stress state, it issues a visual and audible warning. Specifically, an LED on the frame lights up and an alert sound is emitted from the speaker. The strength of the warning is also automatically adjusted according to the state evaluated by the emotion recognition means. The input is the detection result of the minute sleep state and the evaluation result of the emotional state. The output is a visual and audible warning.

[0660] Step 5: Send data

[0661] The terminal transmits the collected sensor data and emotion data to an external computer system via an information transmission means. Specifically, the data is collectively organized into packets and transmitted to a server using wireless communication technology. The input is the sensor data and emotion data. The output is the transmitted data packets.

[0662] Step 6: Data analysis

[0663] The server stores the received data and analyzes the driving characteristics and emotional state. This allows the user's behavioral patterns, changes in attention, and emotional trends to be understood. The input is the transmitted sensor data and emotional data. The output is the analysis results of the driving characteristics and emotional state.

[0664] Step 7: Providing navigation information

[0665] The terminal works in conjunction with the navigation system to obtain the latest traffic information and announces it to the driver through a speaker. For example, it obtains information about highway congestion and provides guidance such as, "You can avoid the congestion by getting off at the next exit." The input is the latest traffic and congestion information. The output is a real-time voice announcement.

[0666] Step 8: Premium Discount Notification

[0667] The server evaluates safe driving based on the collected and analyzed driving data and emotional data. If the evaluation result of safe driving is positive, it applies an insurance premium discount and notifies the user. The input is the safe driving evaluation result. The output is a notification of the insurance premium discount.

[0668] (Application example 2)

[0669] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0670] As autonomous vehicles become more widespread, safety when drivers switch to manual driving has become an important issue. In particular, there is a risk of accidents occurring due to drivers falling asleep at the wheel after long hours of driving or due to overwork, or due to distraction caused by stress. For this reason, there is a demand for systems that can monitor the driver's condition from multiple angles and issue immediate warnings.

[0671] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means including a sensor and a camera that instantly detects eyelid and eyeball movements, means including an artificial intelligence algorithm that analyzes data acquired by the sensor and the camera to detect momentary sleep, means including an emotion engine that analyzes the user's facial expressions and voice to recognize emotions, means for issuing visual and audio warnings in response to detection of a high-stress state, communication means for communicating data from the sensor and the camera, and external information processing device means for storing and analyzing data transmitted by the communication means. This makes it possible to reduce the risk of an accident by evaluating the driver's condition in real time and issuing warnings when necessary.

[0672] The "sensor" is a device for detecting eyelid and eyeball movements in real time.

[0673] The "photography device" refers to a camera device that is combined with a sensor to visually capture the movement of the eyelids and eyeballs.

[0674] The "artificial intelligence algorithm" is an advanced computational method for analyzing acquired data and detecting instantaneous sleep.

[0675] The "emotion engine" is a system that analyzes the user's facial expressions and voice to recognize their emotions.

[0676] "Visual warning" means a warning that is visually conveyed to the user by means of light.

[0677] "Light emitting means" means a device that emits light to a user to effect a visual warning.

[0678] An "auditory warning" is a warning that is given to the user through the ears of the user by a voice output means.

[0679] "Audio output means" refers to a device that outputs a sound to the user to implement an audible warning.

[0680] The "communication means" is a device for transmitting data from the sensor and the image capturing device to an external information processing device.

[0681] The "external information processing device" is a server for storing and analyzing data transmitted by communication means.

[0682] "Driving characteristics" refers to the behavioral patterns and tendencies of a user when driving a vehicle.

[0683] "Attention trends" refers to information obtained by analyzing changes in a user's concentration and attentiveness.

[0684] "Emotional state" refers to the user's emotional state or state of mind.

[0685] A "navigation system" is a system that provides guidance on the current location of a vehicle and the route to a destination.

[0686] This invention is a system that supports safe driving by combining a sensor and a camera that detects eyelid and eyeball movements in real time, an artificial intelligence algorithm that analyzes the acquired data to detect momentary sleep, and an emotion engine. The system is configured as follows:

[0687] 1. Data Collection

[0688] The sensors and imaging devices detect the driver's eye and eyelid movements in real time using standard camera devices and application-specific electrooculography sensors. These devices constantly monitor and collect data on eye and eyelid movements while the driver is operating the vehicle.

[0689] 2. Instantaneous sleep and emotion detection

[0690] The collected data is analyzed in real time by an artificial intelligence algorithm installed on the device. If momentary sleep is detected, the system immediately issues a warning. In addition, an emotion engine analyzes the driver's facial expressions and voice data to assess their level of stress and fatigue. This emotion data is also collected in real time and used to comprehensively evaluate the driver's condition.

[0691] 3. Issuance of warnings

[0692] If momentary sleep or a high stress state is detected, the device will issue visual and audio warnings. Visual warnings are issued by lighting up the device's LED using the light-emitting means to alert the driver. Audio warnings are issued by emitting an alert sound from the speaker via the audio output means to alert the driver.

[0693] 4. Data communication and transmission to external servers

[0694] The collected sensor data and emotional data are transmitted to an external information processing device via communication means. The external information processing device stores this data and analyzes the driver's driving characteristics, attentional tendencies, and emotional state. This clarifies the driver's driving behavior patterns and provides insights for safe driving.

[0695] 5. Linkage with navigation systems

[0696] The device works in conjunction with the navigation system to provide up-to-date traffic information and route guidance, which is announced to the driver via voice output means and provided in real time, helping drivers reach their destination safely and efficiently.

[0697] 6. Insurance premium discounts

[0698] The external information processing device evaluates the driver's safe driving based on driving data and emotional data, and offers discounts on insurance premiums. If particularly positive emotions are recorded, this will be reflected in the insurance premium evaluation, and the driver will be notified when a discount is applied.

[0699] Specific examples

[0700] The user drives an autonomous vehicle using AI-enabled glasses. While driving, sensors and a camera collect eye and eyelid movements in real time, and an artificial intelligence algorithm detects momentary sleep. At the same time, an emotion engine analyzes the user's facial expressions and voice data to assess their stress and fatigue levels. If momentary sleep or high stress is detected, an LED on the device's frame lights up and an alert sounds from the speaker to warn the user. All collected data is sent in real time to an external information processing device for storage and analysis. The device works in conjunction with the navigation system to provide the latest traffic information and support the user. The external information processing device also evaluates the user's safe driving, assesses insurance premium discounts, and notifies the user.

[0701] Prompt Sentence Examples

[0702] "I want to develop an application that can detect drowsiness and fatigue while driving and issue a warning. I need a mechanism to analyze frames from the camera in real time using dlib for face detection, and a function to evaluate emotions and detect high stress and fatigue."

[0703] "This program uses an AI model to analyze eye movement data to detect momentary sleep and prevent drowsiness at the wheel. It also uses an emotion engine to assess the driver's stress and fatigue level and issue a warning if necessary. It also sends the collected data to a server and links it with the navigation system."

[0704] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0705] Step 1:

[0706] Data collection (terminal):

[0707] The device collects data using sensors and imaging devices that detect eyelid and eye movements in real time. Specifically, the imaging device (camera) captures video and the sensor records electrical potential changes. As input, real-time video and electrical potential data are used, collected every millisecond. As output, a collected dataset is generated and sent to the next analysis step.

[0708] Step 2:

[0709] Instantaneous sleep detection (device):

[0710] The device inputs the collected eye movement data and eyelid movement data into an artificial intelligence algorithm to detect signs of instantaneous sleep. Specifically, the data is first preprocessed and converted into an appropriate format. The artificial intelligence algorithm then analyzes this data and calculates the probability of instantaneous sleep. The input is the preprocessed eye movement and eyelid movement data, and the output is the instantaneous sleep detection result (e.g., a probability value or a warning flag).

[0711] Step 3:

[0712] Emotional rating (device):

[0713] The device collects the user's facial expression and voice data and inputs it into the emotion engine. The emotion engine analyzes this data and evaluates the user's stress and fatigue levels. Specifically, the emotion engine analyzes the video and voice data to identify the user's emotional state. The input is facial expression and voice data, and the output is the emotion evaluation result (e.g., stress level, fatigue level).

[0714] Step 4:

[0715] Issue a warning (terminal):

[0716] If a momentary sleep or high stress state is detected, the device issues a visual and audible warning. Specifically, the light-emitting means turns on an LED light and the audio output means plays an alert sound. The inputs are the momentary sleep detection result and the emotion evaluation result, and the outputs are visual and audible warnings (e.g., lighting up an LED, playing an alert sound).

[0717] Step 5:

[0718] Data communication (terminal):

[0719] The collected and analyzed sensor data and emotion data are transmitted to an external information processing device via a communication means. Specifically, the data is packetized using an appropriate protocol and transmitted over a network. The input is the sensor data and emotion data, and the output is the transmission of data to the external information processing device.

[0720] Step 6:

[0721] Data storage and analysis (server):

[0722] The external information processing device accumulates and analyzes the transmitted data. Specifically, it stores the data in a database and uses data analysis algorithms to analyze driving characteristics, attention trends, and emotional states. The input is the transmitted sensor data and emotional data, and the output is the analysis results (e.g., driving characteristics report, attention trends).

[0723] Step 7:

[0724] Providing navigation information (device):

[0725] The terminal works in conjunction with the navigation system to obtain the latest traffic information and provide it to the user in real time. Specifically, it obtains traffic information and route guidance information from the navigation system and announces it to the user through voice output means. The input is traffic information from the navigation system, and the output is voice traffic information and route guidance.

[0726] Step 8:

[0727] Insurance premium discount notification (server):

[0728] The external information processing device evaluates the driver's safe driving based on the analyzed data and notifies the user if an insurance premium discount is to be offered. Specifically, the device applies a safe driving evaluation algorithm, generates an evaluation result, and notifies the user of insurance premium discount information as needed. The input is the analysis result, and the output is an insurance premium discount notification.

[0729] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0730] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0731] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0732] [Third embodiment]

[0733] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0734] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0735] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0736] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0737] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0738] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0739] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0740] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0741] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0742] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0743] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0744] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0745] The present invention is a system for preventing drowsy driving and ensuring driver safety. The system includes an electrooculography sensor and camera device that detects eyelid and eyeball movements, an artificial intelligence algorithm that analyzes data in real time, visual and audio warning devices, a communication module that transmits data, data storage and analysis on an external server, linkage with a navigation system, and an insurance premium discount function.

[0746] System configuration

[0747] 1. Detection equipment and artificial intelligence algorithms

[0748] The device is equipped with an electrooculography sensor and a camera device to detect the driver's eye and eyelid movements. This allows for real-time monitoring of the driver's eye and eye movements while driving, and eye movement data is acquired. This data is analyzed by an artificial intelligence algorithm to detect microsleep.

[0749] 2. Visual and audible warnings

[0750] When the device detects momentary drowsiness, an LED mounted on the frame lights up to provide a visual warning to the driver. In addition, an audible alert sounds from the speaker, providing an audible warning. This helps to draw the driver's attention and prevent drowsy driving.

[0751] 3. Data communication and analysis

[0752] The terminal includes a communication module for transmitting real-time acquired sensor data to an external server, which receives and stores the data. The stored data is used to analyze the driver's driving characteristics and provide feedback to the driver.

[0753] 4. Navigation linkage

[0754] The device is linked to the navigation system, allowing drivers to receive real-time navigation and traffic information through the speaker. For example, it will announce the optimal route to the destination or changes in road conditions.

[0755] 5. Premium discount function

[0756] The server provides drivers with discounts on insurance premiums through collaboration with insurance companies. Specifically, it evaluates safe driving based on driving data and discounts insurance premiums according to the evaluation results. This function provides drivers with an incentive to drive safely.

[0757] Program processing

[0758] Data collection and instantaneous sleep detection

[0759] The device uses sensors and a camera to collect eye movement data, which is then analyzed in real time by an artificial intelligence algorithm to detect momentary sleep.

[0760] Actions triggered by the warning

[0761] When the device detects a moment of sleep, it immediately issues a visual and audible alert: the LED on the frame lights up and an alert sound comes from the speaker.

[0762] Data transmission and analysis

[0763] The device sends the collected data to an external server, which stores the data and analyzes driving characteristics. The analysis results are provided to the driver as feedback.

[0764] Providing navigation information

[0765] The device works in conjunction with the navigation system to obtain the latest traffic information, and announces navigation and traffic congestion information to the driver through a speaker.

[0766] Insurance premium discount notice

[0767] The server evaluates the driver's safe driving based on the driving data and provides discounts on insurance premiums according to the evaluation results. Discount information is notified to the driver.

[0768] Specific usage example

[0769] Suppose a user is wearing AI-enabled glasses and driving a car. While driving, the device's electrooculography sensor and camera device collect eye movement data and detect momentary sleep. When momentary sleep is detected, the device's LED lights up and an alert sounds from the speaker to warn the driver. At the same time, the sensor data is sent from the device to a server, where it is accumulated and analyzed. Next, the device obtains the latest traffic information from the navigation system and announces it to the driver through the speaker. Finally, the server evaluates insurance premium discounts based on the driving data and notifies the driver of this information.

[0770] As described above, the system of the present invention ensures the safety of drivers and contributes to the prevention of traffic accidents.

[0771] The processing flow will be explained below.

[0772] Step 1:

[0773] The user turns on the AI-enabled glasses, which puts the system into initial setup mode.

[0774] Step 2:

[0775] The device automatically calibrates the electrooculography sensor in the nose pad and the camera in the frame, allowing the device to accurately detect the movement of the user's eyes and eyelids.

[0776] Step 3:

[0777] The device measures eye and eyelid movements in real time while driving, and electrooculography sensors and a camera device collect data that is sent to an artificial intelligence algorithm.

[0778] Step 4:

[0779] The device's AI algorithm analyzes the collected data to detect momentary sleep. Specifically, it analyzes patterns such as minute eye movements and eyelid closure, and if it determines that a momentary sleep has occurred, it moves on to the next action.

[0780] Step 5:

[0781] When the device detects momentary sleep, it immediately issues a visual and audible warning: an LED on the frame lights up to provide a visual warning, and an audible alert sounds from the speaker.

[0782] Step 6:

[0783] The device sends the collected sensor data, including eye movements, eyelid movements, and timing of momentary sleep, to an external server via a communication module.

[0784] Step 7:

[0785] The server stores and analyzes the received data, and the results of the analysis reveal the driver's driving characteristics and attentional patterns.

[0786] Step 8:

[0787] The server then provides feedback to the driver based on the analysis results, including advice on times when attention is waning and driving environments.

[0788] Step 9:

[0789] The device connects to the navigation system to obtain the latest traffic and congestion information, which is then announced to the driver through a speaker in the frame.

[0790] Step 10:

[0791] The server works with insurance companies to assess insurance premium discounts based on safe driving characteristics, and notifies the driver of the results, along with details of any discounts that may be applied.

[0792] Step 11:

[0793] After the user has worn the glasses for a day, the system automatically provides a summary of the analysis results, including the number of warnings encountered while driving and the total amount of driving data.

[0794] This not only helps the system prevent drowsy driving and support safe driving, but also provides feedback to drivers and insurance premium incentives.

[0795] Example 1

[0796] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0797] The purpose of this invention is to detect microsleep while driving and immediately warn the driver to prevent drowsiness at the wheel and ensure driver safety. Another objective is to improve driver safety by analyzing driving characteristics based on collected data and providing appropriate feedback. Furthermore, the invention aims to improve driving efficiency by providing route guidance and traffic congestion information in real time.

[0798] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0799] In this invention, the server includes a means including a photoelectric sensor and an image capture device that instantly detect eyelid and eye movement, a means including a machine learning algorithm that analyzes data captured by the sensor and the image capture device to detect momentary sleep, a light-emitting means that issues a visual warning in response to the detection of momentary sleep, a voice output means that issues an auditory warning in response to the detection of momentary sleep, a transmission means that communicates data from the sensor and the image capture device, and an external computing device that stores and analyzes the data transmitted by the transmission means. This enables momentary sleep while driving to be detected in real time and an immediate visual and auditory warning to be issued. Furthermore, the server analyzes driving characteristics and provides navigation information in cooperation with a route guidance system, thereby improving driver safety and efficiency.

[0800] A "photoelectric sensor" is a sensor that converts light into an electrical signal and detects the movement of the eyelids and eyeballs.

[0801] An "image capture device" is a device, such as a camera or image sensor, that is used to capture visual data.

[0802] A "machine learning algorithm" is an algorithm that recognizes patterns based on large amounts of data and makes predictions and classifications.

[0803] "Light-emitting means" means a means for issuing a visual warning to the driver using a light source such as an LED.

[0804] The "audio output means" refers to a means for issuing an audible warning to the driver using a speaker or a sound generating device.

[0805] The "transmission means" is a communication module for transmitting data to other devices or servers.

[0806] An "external computing device" is a server or cloud computer installed in a remote location, and is a device for storing and analyzing data.

[0807] "Driving characteristics" refers to a series of driving-related characteristics, such as the driver's driving behavior and patterns, for example, the frequency and time of day of momentary sleep.

[0808] A "route guidance system" is a system that provides navigation and provides optimal routes to a destination and traffic information.

[0809] This invention is a system that detects microsleep while driving and issues an immediate warning to ensure driver safety. This system consists of three main components: a terminal, a server, and a user.

[0810] Device details

[0811] The device is equipped with a photoelectric sensor and an image capture device to detect the movement of the driver's eyes and eyelids. This allows the device to monitor the driver's eyelid and eye movements in real time, and a machine learning algorithm is used to detect momentary sleep. Specifically, the photoelectric sensor captures minute eye movements, and the image capture device collects video data. The device processes this data in real time to determine whether momentary sleep is likely.

[0812] Example: While a user is driving, the device uses a photoelectric sensor and an image capture device to capture eye movement data. A machine learning algorithm analyzes this data and, if it detects prolonged blinking, determines that the user is fast asleep.

[0813] If a momentary sleep is detected, the device will immediately issue a warning using the "light emitting means" and "audio output means." Specifically, the LED light will turn on and a warning sound will be emitted from the speaker.

[0814] Server Details

[0815] The server is an "external computing device" that receives data sent from the terminal, stores it, and analyzes it. The terminal sends data to the server in real time using a "transmission means." The server stores this data in a database and analyzes the driver's "driving characteristics."

[0816] The analyzed data is then fed back to the driver as feedback to help them drive safely.

[0817] Example: The server analyzes a week's worth of driving data and finds a tendency for frequent short-term sleep to occur during certain times of the day. The server notifies the driver of this and warns them to be careful.

[0818] Furthermore, the server will be linked to the "route guidance system" to provide real-time route guidance and traffic congestion information. Navigation information will be announced to the user through the device's speaker.

[0819] How users use it

[0820] Users can use this system to ensure safety while driving. The device uses a photoelectric sensor and an image capture device to collect eye movement data and detects momentary sleep. If detected, users can receive an immediate warning.

[0821] In addition, safe and efficient driving is possible by receiving feedback based on driving behavior analysis provided by the server and navigation information from the route guidance system.

[0822] Example prompt sentence:

[0823] "Please explain in detail about the system that uses sensors and cameras to detect eyelid and eye movement, analyzes the data in real time, and issues a warning to prevent drowsy driving."

[0824] In this way, the system of the present invention ensures the safety of the driver and contributes greatly to preventing traffic accidents and improving driving efficiency.

[0825] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0826] Step 1:

[0827] As soon as the device starts driving, it activates a photoelectric sensor and image capture device to capture the driver's eye movement data in real time. This device samples data several dozen times per second. Specifically, image data that continuously captures the driver's eye movements and electrooculography data are input. This makes it possible to accurately detect the opening and closing of the eyelids and minute eye movements.

[0828] Step 2:

[0829] The device inputs the acquired eye movement data into a machine learning algorithm. This algorithm uses a pre-trained generative AI model to analyze instantaneous sleep patterns (e.g., prolonged blinking or gaze fixation). Specifically, it detects abnormal blink patterns and changes in gaze based on the input data to determine whether instantaneous sleep has occurred. The device then outputs whether instantaneous sleep has been detected.

[0830] Step 3:

[0831] When the machine learning algorithm detects a momentary sleep, the device immediately activates the light-emitting means (LED light) and the audio output means (speaker). This causes the LED light to light up and an alarm to sound from the speaker to notify the user of the momentary sleep. The output of this step is a visual and audio warning.

[0832] Step 4:

[0833] The device transmits all acquired eye movement data to an external computing device (server) via a transmission means. This data includes time, eyelid movement, detailed information on eye movement, etc. Secure HTTPS communication is used as the transmission protocol. This allows the server to receive detailed data in real time while driving.

[0834] Step 5:

[0835] The server stores the received data in a database. The stored data is then organized for analysis. For example, data cleaning is performed to remove incomplete or noisy data. This results in cleaned driving data.

[0836] Step 6:

[0837] The server analyzes driving characteristics using a generative AI model based on the accumulated data. This analysis determines patterns of instantaneous sleep during specific times and driving conditions, and generates useful feedback for the driver. For example, insights such as "instantaneous sleep occurs more frequently when driving at night" can be obtained. This feedback is notified to the user via email or a smartphone app.

[0838] Step 7:

[0839] The device is linked to a route guidance system. This link allows the device to obtain real-time traffic information and navigation data and announce it to the user through the device's speaker. For example, the device provides information on traffic congestion while driving and guidance on the optimal route to the destination. This allows the user to reach their destination efficiently and safely.

[0840] Step 8:

[0841] The server evaluates the driver's safe driving based on the driving data. Depending on the evaluation results, the server offers the user a discount on insurance premiums. The evaluation results are converted into a numerical value for the driver's safe driving level, and the data is sent to the insurance company based on that. Details of the discount are provided to the user via email or app notification.

[0842] (Application example 1)

[0843] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0844] Conventional driver monitoring systems have limitations in not only detecting momentary drowsiness but also in providing immediate warnings to drivers. There are also issues with how to provide feedback to drivers based on the analysis results of acquired driving data, and how to provide incentives such as insurance discounts. Furthermore, there is a lack of means to provide real-time traffic information obtained while driving. Under these circumstances, there are problems with ensuring driver safety and providing a comfortable driving environment.

[0845] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0846] In this invention, the server includes means including an electrooculography sensor and a camera device that instantly detects eyelid and eyeball movements, means including an artificial intelligence algorithm that analyzes data acquired by the sensor and camera device to detect momentary sleep, light emitting means that issues a visual warning in response to the detection of momentary sleep, audio output means that issues an audio warning in response to the detection of momentary sleep, communication means that communicates data from the sensor and camera device, external server means that accumulates and analyzes the data transmitted by the communication means, navigation linkage means that provides the latest traffic information acquired from the external server means via the audio output means, and means that provides insurance premium discount information based on driving data evaluated by the external server means. This makes it possible to instantly detect drowsy driving while driving and issue visual and audio warnings, and also to provide feedback and insurance premium discount information based on the results of analysis of driving characteristic data, and the latest navigation information in real time.

[0847] Definitions of important words

[0848] The eyelid is a thin layer of skin that protects the eye and allows you to blink.

[0849] The "eyeball" is a spherical organ that senses visual information and is the main part of the human eye.

[0850] An "electrooculography sensor" is a device that detects minute electrical activity in the eyelids and eyeballs and converts it into digital data.

[0851] A "camera device" is a device that captures images and videos and stores and analyzes them as digital data.

[0852] An "artificial intelligence algorithm" is a computational processing method that analyzes large amounts of data, learns patterns, and makes appropriate decisions and predictions.

[0853] A "light emitting means" is a device or system that emits light to visually convey information.

[0854] "Audio output means" refers to a device or system for conveying information using audio.

[0855] "Communication means" refers to the technology or equipment used to transmit data to other devices or servers.

[0856] The "external server means" is a remote server system for storing and analyzing data via a network.

[0857] The "navigation-linked means" is a system that links with the navigation system and provides traffic information and route guidance in real time.

[0858] "Driving characteristics" refers to a driver's behavioral characteristics such as driving habits, patterns, and reaction speed.

[0859] "Insurance premium discount information" is information about insurance premium discounts that are offered when criteria such as safe driving are met.

[0860] MODE FOR CARRYING OUT THE INVENTION

[0861] System configuration

[0862] This invention is a multi-function system for preventing drowsy driving and ensuring the safety of the driver. The elements that specifically constitute the system will be described below.

[0863] 1. Eyelid and eyeball movement detection device and artificial intelligence algorithm

[0864] The device is equipped with an electrooculography sensor and a camera device to instantly detect eyelid and eyeball movements. This allows for real-time monitoring of the driver's eyelid and eyeball movements while driving. The acquired eyeball movement data is analyzed by an artificial intelligence algorithm to detect microsleep.

[0865] 2. Visual and audible warning devices

[0866] When the device detects momentary drowsiness, an LED mounted on the frame lights up to provide a visual warning to the driver. In addition, an audible alert sounds from the speaker, providing an audible warning. This helps to draw the driver's attention and prevent drowsy driving.

[0867] 3. Data communication and external servers

[0868] The terminal includes a communication module for transmitting real-time acquired sensor data to an external server, which receives and stores the data. The stored data is used to analyze the driver's driving characteristics and provide feedback to the driver.

[0869] 4. Navigation linkage

[0870] The device is linked to the navigation system, allowing drivers to receive real-time navigation and traffic information through the speaker. For example, it will announce the optimal route to the destination or changes in road conditions.

[0871] 5. Premium discount function

[0872] The server provides drivers with discounts on insurance premiums through collaboration with insurance companies. Specifically, it evaluates safe driving based on driving data and discounts insurance premiums according to the evaluation results. This function provides drivers with an incentive to drive safely.

[0873] Program processing explanation

[0874] The device uses sensors and a camera device to collect eye movement data. An artificial intelligence algorithm then analyzes this data in real time to detect momentary sleep. When a momentary sleep is detected, the device will illuminate LEDs on the frame and play an audible alert through the speaker to provide visual and audio warnings.

[0875] The collected data is sent to an external server via the communication module. The server stores this data and analyzes driving characteristics. The analysis results are provided to the driver as feedback.

[0876] Furthermore, the terminal is linked to the navigation system, has the function of obtaining the latest traffic information and announcing navigation and congestion information to the driver through a speaker. The server also evaluates the driver's safe driving based on the driving data and notifies the driver of insurance premium discount information according to the evaluation results.

[0877] Specific usage example

[0878] For example, consider a case where a user drives a car wearing smart glasses equipped with this system. While driving, the device's electrooculography sensor and camera device collect eye movement data and detect momentary sleep. When momentary sleep is detected, the device's LED lights up and an alert sounds from the speaker to warn the driver. At the same time, the sensor data is sent from the device to a server, where it is stored and analyzed. The device then obtains the latest traffic information from the navigation system and announces it to the driver through the speaker. Finally, the server evaluates insurance premium discounts based on the driving data and notifies the driver of this information.

[0879] A specific example of a prompt sentence is "Analyze my current driving data and tell me the insurance premium discount rate."

[0880] As described above, the system of the present invention ensures the safety of drivers and makes it possible to prevent traffic accidents.

[0881] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0882] Program processing flow

[0883] Step 1:

[0884] Input: Data from electrooculography sensors and camera devices for real-time detection of eyelid and eye movements.

[0885] How it works: The device activates the electrooculography sensor and camera device to collect data on the driver's eyelid and eyeball movements in milliseconds, making it possible to understand the driver's eye movement, blink frequency, and eye fatigue.

[0886] Output: Collected eye movement data.

[0887] Step 2:

[0888] Input: Collected eye movement data.

[0889] How it works: The device's AI algorithm analyzes collected data in real time to detect microsleep. It processes the data by analyzing eye movement speed and eyelid opening / closing patterns, and determines that a microsleep has occurred if it exceeds a certain threshold.

[0890] Output: Instantaneous sleep detection result.

[0891] Step 3:

[0892] Input: Instantaneous sleep detection result.

[0893] Specific operation: When the device detects momentary sleep, it immediately activates the warning system. A visual warning is provided by the LED on the frame, and an audible warning is provided by the speaker. This immediately alerts the driver.

[0894] Output: Warning signals to the driver (visual and audible).

[0895] Step 4:

[0896] Input: Collected eye movement data.

[0897] Specific operation: The terminal uses the communication module to send the collected data to an external server. Specifically, the data is compressed and encrypted before being sent to the server via a communication protocol.

[0898] Output: Sending encrypted and compressed data to an external server.

[0899] Step 5:

[0900] Input: Data sent from the terminal.

[0901] How it works: The server accumulates the received data and analyzes the driver's driving characteristics. The analysis uses a generative AI model to identify multiple driving patterns and identify the driver's habits and characteristics.

[0902] Output: Insights into driving characteristics.

[0903] Step 6:

[0904] Input: Latest traffic and navigation data.

[0905] Specific operation: The terminal works in conjunction with the navigation system to announce real-time traffic information and the optimal route to the driver via voice output. In this process, the terminal analyzes the traffic information obtained via the communication module and selects the content that is most appropriate for the driver.

[0906] Output: Navigation information and traffic announcements to the driver.

[0907] Step 7:

[0908] Input: Accumulated driving data and driving characteristics analysis results.

[0909] Specific operation: The server evaluates safe driving based on driving data and generates insurance premium discount information based on the evaluation results. The generated insurance premium discount information is notified to the driver via the terminal.

[0910] Output: Notification of insurance premium discount information to the driver.

[0911] Through the above processing steps, this system ensures driver safety and makes it possible to prevent traffic accidents.

[0912] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0913] This invention is a system that includes an electrooculography sensor, a camera device, an artificial intelligence algorithm, an emotion engine, and a communication means and external server that link these together to prevent drowsy driving and ensure driver safety. The system aims to reduce the risk of accidents while driving by monitoring and evaluating the user's condition from multiple angles and issuing warnings at appropriate times.

[0914] System configuration

[0915] 1. Detection equipment and artificial intelligence algorithms

[0916] The device is equipped with an electrooculography sensor and a camera device to detect the driver's eye and eyelid movements. This allows real-time collection of eye movement data and eyelid movements while driving. This data is analyzed by an artificial intelligence algorithm to detect microsleep.

[0917] 2. Emotion Engine

[0918] The device also features an emotion engine that can recognize emotions by analyzing the user's facial expressions and voice. The emotion engine detects the user's stress level, fatigue level, and new emotional changes, and combines this data with other sensor information to provide a comprehensive condition assessment.

[0919] 3. Warning measures

[0920] When the device detects high stress or fatigue as assessed by the instantaneous sleep and emotion engine, it immediately issues visual and audible warnings. Specifically, the LED on the frame lights up to provide a visual warning, and an audible alert sounds from the speaker. Furthermore, if the emotion engine detects high stress or fatigue, the intensity of the warnings is automatically adjusted.

[0921] 4. Data communication and analysis

[0922] The device transmits the collected sensor data and emotional data via a communication module to an external server, which receives, stores, and analyzes the data. The analysis reveals the driver's driving characteristics, attentional tendencies, emotional state, and other information.

[0923] 5. Navigation system linkage

[0924] The device is equipped with a function that links with the navigation system, obtaining the latest traffic and congestion information and announcing it to the driver in real time through a speaker, allowing the driver to obtain the appropriate information to reach their destination safely.

[0925] 6. Premium discount function

[0926] The server works with insurance companies to offer drivers discounts on their insurance premiums. In particular, if the emotion engine records positive emotions, it will reflect that in the evaluation of insurance premiums. When a discount is applied, the details are notified to the driver.

[0927] Program processing

[0928] Data collection and instantaneous sleep detection

[0929] The device uses an electrooculography sensor and a camera to collect eye movement data. This data is analyzed in real time by an artificial intelligence algorithm to detect momentary sleep. The device's emotion engine also analyzes the user's facial expressions and voice data to recognize emotions. Emotional data is also collected in real time and used to comprehensively evaluate the user's state.

[0930] Actions triggered by the warning

[0931] If the device detects momentary sleep, it will simultaneously issue a visual and audible alert. The LED on the frame will light up and an alert sound will be emitted from the speaker. The intensity of the alert will also automatically adjust if the emotion engine detects high stress or fatigue.

[0932] Data transmission and analysis

[0933] The device sends the collected sensor data and emotional data to an external server, which then stores the received data and analyzes the driver's driving characteristics and emotional state. As a result, it becomes possible to understand the driver's behavioral patterns, changes in attention, and emotional trends.

[0934] Providing navigation information

[0935] The device connects to the navigation system to obtain the latest traffic information and announces traffic conditions and route guidance to the driver through a speaker. This information is provided in real time to assist the driver in making decisions.

[0936] Insurance premium discount notice

[0937] The server evaluates safe driving based on driving data and emotional data and offers discounts on insurance premiums. If the emotion engine detects positive emotions, it will also be reflected in the evaluation, and the driver will be notified when a discount is applied.

[0938] Specific usage example

[0939] The user drives a car using AI-enabled glasses. While driving, the device's electrooculography sensor and camera device collect eye and eyelid movements in real time, and an artificial intelligence algorithm detects momentary sleep. At the same time, the device's emotion engine analyzes the user's facial expressions and voice data to assess their stress and fatigue level. If momentary sleep or high stress is detected, an LED on the device's frame lights up and an alert sounds from the speaker to warn the user. All collected data is sent to a server in real time, where it is stored and analyzed. The device works in conjunction with the navigation system to provide the latest traffic information to support the driver. The server also evaluates the driver's safe driving, assesses insurance premium discounts, and notifies the driver.

[0940] In this way, the system is a powerful tool for monitoring the user's multifaceted condition and reducing risks while driving.

[0941] The processing flow will be explained below.

[0942] Step 1:

[0943] The user turns on the AI-enabled glasses, which puts the system into initial setup mode.

[0944] Step 2:

[0945] The device automatically calibrates the electrooculography sensor in the nose pad and the camera in the frame, a process that allows the device to accurately detect the movement of the user's eyes and eyelids.

[0946] Step 3:

[0947] The device measures eye and eyelid movements in real time while driving. Electrooculography sensors and a camera device collect data, which is then sent to an artificial intelligence algorithm for analysis.

[0948] Step 4:

[0949] The device's AI algorithm analyzes the collected data to detect microsleep. Specifically, it analyzes patterns such as minute eye movements and eyelid closure, and if it determines that a microsleep has occurred, it moves on to the next step.

[0950] Step 5:

[0951] The device's emotion engine analyzes the user's facial expressions and voice to assess their stress level and fatigue. This emotion data is then combined with instantaneous sleep detection data to provide a comprehensive assessment of their condition.

[0952] Step 6:

[0953] If the device detects a moment of deep sleep or high stress, it will immediately issue a visual and audible alert: an LED on the frame will light up and an alert sound will sound from the speaker.

[0954] Step 7:

[0955] The device transmits collected sensor data and emotional data, including eye movements, eyelid movements, timing of momentary sleep, and emotional state, to an external server via a communication module.

[0956] Step 8:

[0957] The server stores the received data and analyzes the driver's driving characteristics and emotional state, specifically analyzing trends in the driver's attention and changes in stress level.

[0958] Step 9:

[0959] The server then provides the driver with feedback based on the analysis results, including advice on times when attention may be waning due to long driving periods and advice on avoiding driving under high stress conditions.

[0960] Step 10:

[0961] The device connects to the navigation system to obtain the latest traffic and congestion information, which is then provided to the driver in real time through a speaker in the frame.

[0962] Step 11:

[0963] The server connects with insurance companies and evaluates insurance discounts based on safe driving and emotional state data. If the emotion engine records positive emotions, they are also reflected in the evaluation.

[0964] Step 12:

[0965] The server notifies the driver of details when the discount is applied, including the discount rate and the period of application.

[0966] Step 13:

[0967] After the user has worn the glasses for a day, the system automatically provides a summary of the analysis results, including the number of warnings encountered while driving, fluctuations in emotional state, and the total amount of driving data.

[0968] Through these steps, the system not only prevents drowsy driving, but also contributes to comprehensive driver condition management, supports safe driving, and provides incentives through insurance premium discounts.

[0969] Example 2

[0970] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0971] Drowsiness and high stress while driving are cited as causes of traffic accidents. Conventional driving monitoring systems detect momentary drowsiness by monitoring only eye and eyelid movements, but this alone makes it difficult to fully grasp the driver's overall condition. Furthermore, they lack the functionality to evaluate the driver's emotional state and fatigue level in real time and adjust the intensity of warnings according to the situation. Therefore, the objective of this invention is to reduce the risk of traffic accidents by detecting momentary drowsiness, evaluating the driver's emotional state and fatigue level, and issuing prompt and appropriate warnings to the driver.

[0972] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means including a sensor and a video capture device that instantly detect eyelid and eyeball movements, means including a machine learning algorithm that analyzes data acquired by the sensor and the video capture device to detect subtle sleep states, and means for recognizing emotions by analyzing the user's facial expressions and voice. This makes it possible to evaluate the driver's momentary sleep and emotional state from multiple angles and issue a warning at an appropriate time.

[0973] A "sensor" is a device that detects physical quantities such as light, sound, and electric current and converts them into electrical signals.

[0974] A "video capture device" is a device that records images and videos using optical means.

[0975] A "machine learning algorithm" is a computational method for learning patterns from data and using the results to analyze new data.

[0976] "Light emitting means" means a device or method for emitting light, primarily used to provide visual notification.

[0977] "Audio output means" refers to a device or method for generating and outputting audio.

[0978] "Information transmission means" means a means for transmitting data from one point to another, and includes wired or wireless communication means.

[0979] An "external computer system" is an external computing device or server accessed over a network and used for data storage and analysis.

[0980] An "emotion recognition means" is a device or system that analyzes a user's facial expressions and voice and identifies their emotional state.

[0981] A "means for adjusting the intensity of the warning" is a device or function for varying the intensity of the warning depending on the detected condition.

[0982] "Traffic information" refers to the latest travel information necessary for drivers, such as road conditions, traffic congestion, and accident information.

[0983] This invention is a system that detects drowsiness and high stress while driving in real time and issues a warning. To ensure safe driving, this system works by combining sensors, video capture devices, machine learning algorithms, emotion recognition means, information transmission means, and external computer systems.

[0984] System configuration and functions

[0985] Data collection

[0986] The device is equipped with high-precision sensors and video capture devices to detect eyelid and eyeball movements in real time. These devices collect the user's eye movement data and eyelid movements in real time. For example, when a user moves their eyes left and right while driving, the data is detected immediately.

[0987] Detecting subtle sleep states

[0988] The collected data is analyzed by a machine learning algorithm installed on the device, which detects microsleep based on the user's eye and eyelid movement patterns. For example, if the user's eyelids remain closed for a few seconds, the system will identify this as a microsleep.

[0989] emotion recognition

[0990] The device is equipped with a means of recognizing emotions by analyzing the user's facial expressions and voice. This data is also collected in real time to evaluate the user's stress level and fatigue. For example, if a frowning expression or a tired voice is detected, it is judged to be in a state of high stress or fatigue.

[0991] Warning

[0992] If a state of microsleep or high stress is detected, the device will issue a visual and audible warning. Specifically, an LED on the frame will light up and an alert sound will be emitted from the speaker. The intensity of the warning will also be automatically adjusted depending on the situation evaluated by the emotion recognition system. As a result, the user will receive a warning at the appropriate time and be able to respond quickly.

[0993] Data communication and the role of external servers

[0994] The sensor data and emotion data collected by the device are transmitted to an external computer system via an information transmission medium. The server receives, stores, and analyzes this data. The analysis results provide insights into the user's driving characteristics and emotional state.

[0995] Providing navigation information

[0996] The device works in conjunction with the navigation system to obtain the latest traffic and congestion information and announce it to the user in real time. For example, it obtains information about highway congestion and provides voice guidance such as, "You can avoid the congestion by getting off at the next exit."

[0997] Insurance premium discount notification

[0998] The server evaluates safe driving based on the collected and analyzed driving data and emotional data. For example, if the emotion recognition means frequently detects positive emotions, that evaluation will be reflected in the insurance premium discount. The user will be notified when the discount is applied.

[0999] Specific usage example

[1000] «Example of prompt sentences to input to a generative AI model»

[1001] "Please explain how the system works to detect drowsiness or high stress while driving and issue a warning."

[1002] "Please tell me the specific processing method of the driver monitoring system using electrooculography sensors and an emotion engine."

[1003] "Please explain with examples how this system collects data, issues alerts, and analyzes the data."

[1004] This system enables multifaceted condition monitoring while the user is driving, and serves as a powerful tool to support safe driving.

[1005] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1006] Step 1: Data collection

[1007] The device uses sensors and video capture devices to detect eyelid and eye movements in real time. These devices collect the user's eye movement data and eyelid movements in real time. Specifically, they detect the user's gaze movements and blinking movements and generate electrical signals corresponding to those movements. These electrical signals are stored as data. The input is the user's eye movement and eyelid movement, which are detected by the sensors. The output is eye movement data and eyelid movement data.

[1008] Step 2: Detecting Minor Sleep States

[1009] The device inputs the collected eye movement data and eyelid movements into a machine learning algorithm to detect microsleep. For example, if the data shows that the eyes do not blink or the eyelids remain closed for a long period of time, the algorithm determines this to be microsleep. The input is the eye movement data and eyelid movement data obtained in the data collection step. The output is the detection result of the microsleep state, specifically a judgment result such as "microsleep has been detected."

[1010] Step 3: Emotion Recognition

[1011] The device collects the user's facial expressions and voice using emotion recognition means. Based on this, an emotion recognition algorithm analyzes the user's emotional state. Specifically, it detects facial muscle movements and tone of voice to evaluate the user's stress level and fatigue level. The input is the user's facial expression data and voice data. The output is the evaluation result of the emotional state, such as "a high stress level has been detected."

[1012] Step 4: Send an alert

[1013] If the device detects momentary sleep or a high-stress state, it issues a visual and audible warning. Specifically, an LED on the frame lights up and an alert sound is emitted from the speaker. The strength of the warning is also automatically adjusted according to the state evaluated by the emotion recognition means. The input is the detection result of the minute sleep state and the evaluation result of the emotional state. The output is a visual and audible warning.

[1014] Step 5: Send data

[1015] The terminal transmits the collected sensor data and emotion data to an external computer system via an information transmission means. Specifically, the data is collectively organized into packets and transmitted to a server using wireless communication technology. The input is the sensor data and emotion data. The output is the transmitted data packets.

[1016] Step 6: Data analysis

[1017] The server stores the received data and analyzes the driving characteristics and emotional state. This allows the user's behavioral patterns, changes in attention, and emotional trends to be understood. The input is the transmitted sensor data and emotional data. The output is the analysis results of the driving characteristics and emotional state.

[1018] Step 7: Providing navigation information

[1019] The terminal works in conjunction with the navigation system to obtain the latest traffic information and announces it to the driver through a speaker. For example, it obtains information about highway congestion and provides guidance such as, "You can avoid the congestion by getting off at the next exit." The input is the latest traffic and congestion information. The output is a real-time voice announcement.

[1020] Step 8: Premium Discount Notification

[1021] The server evaluates safe driving based on the collected and analyzed driving data and emotional data. If the evaluation result of safe driving is positive, it applies an insurance premium discount and notifies the user. The input is the safe driving evaluation result. The output is a notification of the insurance premium discount.

[1022] (Application example 2)

[1023] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1024] As autonomous vehicles become more widespread, safety when drivers switch to manual driving has become an important issue. In particular, there is a risk of accidents occurring due to drivers falling asleep at the wheel after long hours of driving or due to overwork, or due to distraction caused by stress. For this reason, there is a demand for systems that can monitor the driver's condition from multiple angles and issue immediate warnings.

[1025] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means including a sensor and a camera that instantly detects eyelid and eyeball movements, means including an artificial intelligence algorithm that analyzes data acquired by the sensor and the camera to detect momentary sleep, means including an emotion engine that analyzes the user's facial expressions and voice to recognize emotions, means for issuing visual and audio warnings in response to detection of a high-stress state, communication means for communicating data from the sensor and the camera, and external information processing device means for storing and analyzing data transmitted by the communication means. This makes it possible to reduce the risk of an accident by evaluating the driver's condition in real time and issuing warnings when necessary.

[1026] The "sensor" is a device for detecting eyelid and eyeball movements in real time.

[1027] The "photography device" refers to a camera device that is combined with a sensor to visually capture the movement of the eyelids and eyeballs.

[1028] The "artificial intelligence algorithm" is an advanced computational method for analyzing acquired data and detecting instantaneous sleep.

[1029] The "emotion engine" is a system that analyzes the user's facial expressions and voice to recognize their emotions.

[1030] "Visual warning" means a warning that is visually conveyed to the user by means of light.

[1031] "Light emitting means" means a device that emits light to a user to effect a visual warning.

[1032] An "auditory warning" is a warning that is given to the user through the ears of the user by a voice output means.

[1033] "Audio output means" refers to a device that outputs a sound to the user to implement an audible warning.

[1034] The "communication means" is a device for transmitting data from the sensor and the image capturing device to an external information processing device.

[1035] The "external information processing device" is a server for storing and analyzing data transmitted by communication means.

[1036] "Driving characteristics" refers to the behavioral patterns and tendencies of a user when driving a vehicle.

[1037] "Attention trends" refers to information obtained by analyzing changes in a user's concentration and attentiveness.

[1038] "Emotional state" refers to the user's emotional state or state of mind.

[1039] A "navigation system" is a system that provides guidance on the current location of a vehicle and the route to a destination.

[1040] This invention is a system that supports safe driving by combining a sensor and a camera that detects eyelid and eyeball movements in real time, an artificial intelligence algorithm that analyzes the acquired data to detect momentary sleep, and an emotion engine. The system is configured as follows:

[1041] 1. Data Collection

[1042] The sensors and imaging devices detect the driver's eye and eyelid movements in real time using standard camera devices and application-specific electrooculography sensors. These devices constantly monitor and collect data on eye and eyelid movements while the driver is operating the vehicle.

[1043] 2. Instantaneous sleep and emotion detection

[1044] The collected data is analyzed in real time by an artificial intelligence algorithm installed on the device. If momentary sleep is detected, the system immediately issues a warning. In addition, an emotion engine analyzes the driver's facial expressions and voice data to assess their level of stress and fatigue. This emotion data is also collected in real time and used to comprehensively evaluate the driver's condition.

[1045] 3. Issuance of warnings

[1046] If momentary sleep or a high stress state is detected, the device will issue visual and audio warnings. Visual warnings are issued by lighting up the device's LED using the light-emitting means to alert the driver. Audio warnings are issued by emitting an alert sound from the speaker via the audio output means to alert the driver.

[1047] 4. Data communication and transmission to external servers

[1048] The collected sensor data and emotional data are transmitted to an external information processing device via communication means. The external information processing device stores this data and analyzes the driver's driving characteristics, attentional tendencies, and emotional state. This clarifies the driver's driving behavior patterns and provides insights for safe driving.

[1049] 5. Linkage with navigation systems

[1050] The device works in conjunction with the navigation system to provide up-to-date traffic information and route guidance, which is announced to the driver via voice output means and provided in real time, helping drivers reach their destination safely and efficiently.

[1051] 6. Insurance premium discounts

[1052] The external information processing device evaluates the driver's safe driving based on driving data and emotional data, and offers discounts on insurance premiums. If particularly positive emotions are recorded, this will be reflected in the insurance premium evaluation, and the driver will be notified when a discount is applied.

[1053] Specific examples

[1054] The user drives an autonomous vehicle using AI-enabled glasses. While driving, sensors and a camera collect eye and eyelid movements in real time, and an artificial intelligence algorithm detects momentary sleep. At the same time, an emotion engine analyzes the user's facial expressions and voice data to assess their stress and fatigue levels. If momentary sleep or high stress is detected, an LED on the device's frame lights up and an alert sounds from the speaker to warn the user. All collected data is sent in real time to an external information processing device for storage and analysis. The device works in conjunction with the navigation system to provide the latest traffic information and support the user. The external information processing device also evaluates the user's safe driving, assesses insurance premium discounts, and notifies the user.

[1055] Prompt Sentence Examples

[1056] "I want to develop an application that can detect drowsiness and fatigue while driving and issue a warning. I need a mechanism to analyze frames from the camera in real time using dlib for face detection, and a function to evaluate emotions and detect high stress and fatigue."

[1057] "This program uses an AI model to analyze eye movement data to detect momentary sleep and prevent drowsiness at the wheel. It also uses an emotion engine to assess the driver's stress and fatigue level and issue a warning if necessary. It also sends the collected data to a server and links it with the navigation system."

[1058] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1059] Step 1:

[1060] Data collection (terminal):

[1061] The device collects data using sensors and imaging devices that detect eyelid and eye movements in real time. Specifically, the imaging device (camera) captures video and the sensor records electrical potential changes. As input, real-time video and electrical potential data are used, collected every millisecond. As output, a collected dataset is generated and sent to the next analysis step.

[1062] Step 2:

[1063] Instantaneous sleep detection (device):

[1064] The device inputs the collected eye movement data and eyelid movement data into an artificial intelligence algorithm to detect signs of instantaneous sleep. Specifically, the data is first preprocessed and converted into an appropriate format. The artificial intelligence algorithm then analyzes this data and calculates the probability of instantaneous sleep. The input is the preprocessed eye movement and eyelid movement data, and the output is the instantaneous sleep detection result (e.g., a probability value or a warning flag).

[1065] Step 3:

[1066] Emotional rating (device):

[1067] The device collects the user's facial expression and voice data and inputs it into the emotion engine. The emotion engine analyzes this data and evaluates the user's stress and fatigue levels. Specifically, the emotion engine analyzes the video and voice data to identify the user's emotional state. The input is facial expression and voice data, and the output is the emotion evaluation result (e.g., stress level, fatigue level).

[1068] Step 4:

[1069] Issue a warning (terminal):

[1070] If a momentary sleep or high stress state is detected, the device issues a visual and audible warning. Specifically, the light-emitting means turns on an LED light and the audio output means plays an alert sound. The inputs are the momentary sleep detection result and the emotion evaluation result, and the outputs are visual and audible warnings (e.g., lighting up an LED, playing an alert sound).

[1071] Step 5:

[1072] Data communication (terminal):

[1073] The collected and analyzed sensor data and emotion data are transmitted to an external information processing device via a communication means. Specifically, the data is packetized using an appropriate protocol and transmitted over a network. The input is the sensor data and emotion data, and the output is the transmission of data to the external information processing device.

[1074] Step 6:

[1075] Data storage and analysis (server):

[1076] The external information processing device accumulates and analyzes the transmitted data. Specifically, it stores the data in a database and uses data analysis algorithms to analyze driving characteristics, attention trends, and emotional states. The input is the transmitted sensor data and emotional data, and the output is the analysis results (e.g., driving characteristics report, attention trends).

[1077] Step 7:

[1078] Providing navigation information (device):

[1079] The terminal works in conjunction with the navigation system to obtain the latest traffic information and provide it to the user in real time. Specifically, it obtains traffic information and route guidance information from the navigation system and announces it to the user through voice output means. The input is traffic information from the navigation system, and the output is voice traffic information and route guidance.

[1080] Step 8:

[1081] Insurance premium discount notification (server):

[1082] The external information processing device evaluates the driver's safe driving based on the analyzed data and notifies the user if an insurance premium discount is to be offered. Specifically, the device applies a safe driving evaluation algorithm, generates an evaluation result, and notifies the user of insurance premium discount information as needed. The input is the analysis result, and the output is an insurance premium discount notification.

[1083] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1084] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1085] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1086] [Fourth embodiment]

[1087] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1088] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1089] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1090] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1091] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1092] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1093] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1094] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1095] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1096] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1097] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1098] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1099] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1100] The present invention is a system for preventing drowsy driving and ensuring driver safety. The system includes an electrooculography sensor and camera device that detects eyelid and eyeball movements, an artificial intelligence algorithm that analyzes data in real time, visual and audio warning devices, a communication module that transmits data, data storage and analysis on an external server, linkage with a navigation system, and an insurance premium discount function.

[1101] System configuration

[1102] 1. Detection equipment and artificial intelligence algorithms

[1103] The device is equipped with an electrooculography sensor and a camera device to detect the driver's eye and eyelid movements. This allows for real-time monitoring of the driver's eye and eye movements while driving, and eye movement data is acquired. This data is analyzed by an artificial intelligence algorithm to detect microsleep.

[1104] 2. Visual and audible warnings

[1105] When the device detects momentary drowsiness, an LED mounted on the frame lights up to provide a visual warning to the driver. In addition, an audible alert sounds from the speaker, providing an audible warning. This helps to draw the driver's attention and prevent drowsy driving.

[1106] 3. Data communication and analysis

[1107] The terminal includes a communication module for transmitting real-time acquired sensor data to an external server, which receives and stores the data. The stored data is used to analyze the driver's driving characteristics and provide feedback to the driver.

[1108] 4. Navigation linkage

[1109] The device is linked to the navigation system, allowing drivers to receive real-time navigation and traffic information through the speaker. For example, it will announce the optimal route to the destination or changes in road conditions.

[1110] 5. Premium discount function

[1111] The server provides drivers with discounts on insurance premiums through collaboration with insurance companies. Specifically, it evaluates safe driving based on driving data and discounts insurance premiums according to the evaluation results. This function provides drivers with an incentive to drive safely.

[1112] Program processing

[1113] Data collection and instantaneous sleep detection

[1114] The device uses sensors and a camera to collect eye movement data, which is then analyzed in real time by an artificial intelligence algorithm to detect momentary sleep.

[1115] Actions triggered by the warning

[1116] When the device detects a moment of sleep, it immediately issues a visual and audible alert: the LED on the frame lights up and an alert sound comes from the speaker.

[1117] Data transmission and analysis

[1118] The device sends the collected data to an external server, which stores the data and analyzes driving characteristics. The analysis results are provided to the driver as feedback.

[1119] Providing navigation information

[1120] The device works in conjunction with the navigation system to obtain the latest traffic information, and announces navigation and traffic congestion information to the driver through a speaker.

[1121] Insurance premium discount notice

[1122] The server evaluates the driver's safe driving based on the driving data and provides discounts on insurance premiums according to the evaluation results. Discount information is notified to the driver.

[1123] Specific usage example

[1124] Suppose a user is wearing AI-enabled glasses and driving a car. While driving, the device's electrooculography sensor and camera device collect eye movement data and detect momentary sleep. When momentary sleep is detected, the device's LED lights up and an alert sounds from the speaker to warn the driver. At the same time, the sensor data is sent from the device to a server, where it is accumulated and analyzed. Next, the device obtains the latest traffic information from the navigation system and announces it to the driver through the speaker. Finally, the server evaluates insurance premium discounts based on the driving data and notifies the driver of this information.

[1125] As described above, the system of the present invention ensures the safety of drivers and contributes to the prevention of traffic accidents.

[1126] The processing flow will be explained below.

[1127] Step 1:

[1128] The user turns on the AI-enabled glasses, which puts the system into initial setup mode.

[1129] Step 2:

[1130] The device automatically calibrates the electrooculography sensor in the nose pad and the camera in the frame, allowing the device to accurately detect the movement of the user's eyes and eyelids.

[1131] Step 3:

[1132] The device measures eye and eyelid movements in real time while driving, and electrooculography sensors and a camera device collect data that is sent to an artificial intelligence algorithm.

[1133] Step 4:

[1134] The device's AI algorithm analyzes the collected data to detect momentary sleep. Specifically, it analyzes patterns such as minute eye movements and eyelid closure, and if it determines that a momentary sleep has occurred, it moves on to the next action.

[1135] Step 5:

[1136] When the device detects momentary sleep, it immediately issues a visual and audible warning: an LED on the frame lights up to provide a visual warning, and an audible alert sounds from the speaker.

[1137] Step 6:

[1138] The device sends the collected sensor data, including eye movements, eyelid movements, and timing of momentary sleep, to an external server via a communication module.

[1139] Step 7:

[1140] The server stores and analyzes the received data, and the results of the analysis reveal the driver's driving characteristics and attentional patterns.

[1141] Step 8:

[1142] The server then provides feedback to the driver based on the analysis results, including advice on times when attention is waning and driving environments.

[1143] Step 9:

[1144] The device connects to the navigation system to obtain the latest traffic and congestion information, which is then announced to the driver through a speaker in the frame.

[1145] Step 10:

[1146] The server works with insurance companies to assess insurance premium discounts based on safe driving characteristics, and notifies the driver of the results, along with details of any discounts that may be applied.

[1147] Step 11:

[1148] After the user has worn the glasses for a day, the system automatically provides a summary of the analysis results, including the number of warnings encountered while driving and the total amount of driving data.

[1149] This not only helps the system prevent drowsy driving and support safe driving, but also provides feedback to drivers and insurance premium incentives.

[1150] Example 1

[1151] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1152] The purpose of this invention is to detect microsleep while driving and immediately warn the driver to prevent drowsiness at the wheel and ensure driver safety. Another objective is to improve driver safety by analyzing driving characteristics based on collected data and providing appropriate feedback. Furthermore, the invention aims to improve driving efficiency by providing route guidance and traffic congestion information in real time.

[1153] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1154] In this invention, the server includes a means including a photoelectric sensor and an image capture device that instantly detect eyelid and eye movement, a means including a machine learning algorithm that analyzes data captured by the sensor and the image capture device to detect momentary sleep, a light-emitting means that issues a visual warning in response to the detection of momentary sleep, a voice output means that issues an auditory warning in response to the detection of momentary sleep, a transmission means that communicates data from the sensor and the image capture device, and an external computing device that stores and analyzes the data transmitted by the transmission means. This enables momentary sleep while driving to be detected in real time and an immediate visual and auditory warning to be issued. Furthermore, the server analyzes driving characteristics and provides navigation information in cooperation with a route guidance system, thereby improving driver safety and efficiency.

[1155] A "photoelectric sensor" is a sensor that converts light into an electrical signal and detects the movement of the eyelids and eyeballs.

[1156] An "image capture device" is a device, such as a camera or image sensor, that is used to capture visual data.

[1157] A "machine learning algorithm" is an algorithm that recognizes patterns based on large amounts of data and makes predictions and classifications.

[1158] "Light-emitting means" means a means for issuing a visual warning to the driver using a light source such as an LED.

[1159] The "audio output means" refers to a means for issuing an audible warning to the driver using a speaker or a sound generating device.

[1160] The "transmission means" is a communication module for transmitting data to other devices or servers.

[1161] An "external computing device" is a server or cloud computer installed in a remote location, and is a device for storing and analyzing data.

[1162] "Driving characteristics" refers to a series of driving-related characteristics, such as the driver's driving behavior and patterns, for example, the frequency and time of day of momentary sleep.

[1163] A "route guidance system" is a system that provides navigation and provides optimal routes to a destination and traffic information.

[1164] This invention is a system that detects microsleep while driving and issues an immediate warning to ensure driver safety. This system consists of three main components: a terminal, a server, and a user.

[1165] Device details

[1166] The device is equipped with a photoelectric sensor and an image capture device to detect the movement of the driver's eyes and eyelids. This allows the device to monitor the driver's eyelid and eye movements in real time, and a machine learning algorithm is used to detect momentary sleep. Specifically, the photoelectric sensor captures minute eye movements, and the image capture device collects video data. The device processes this data in real time to determine whether momentary sleep is likely.

[1167] Example: While a user is driving, the device uses a photoelectric sensor and an image capture device to capture eye movement data. A machine learning algorithm analyzes this data and, if it detects prolonged blinking, determines that the user is fast asleep.

[1168] If a momentary sleep is detected, the device will immediately issue a warning using the "light emitting means" and "audio output means." Specifically, the LED light will turn on and a warning sound will be emitted from the speaker.

[1169] Server Details

[1170] The server is an "external computing device" that receives data sent from the terminal, stores it, and analyzes it. The terminal sends data to the server in real time using a "transmission means." The server stores this data in a database and analyzes the driver's "driving characteristics."

[1171] The analyzed data is then fed back to the driver as feedback to help them drive safely.

[1172] Example: The server analyzes a week's worth of driving data and finds a tendency for frequent short-term sleep to occur during certain times of the day. The server notifies the driver of this and warns them to be careful.

[1173] Furthermore, the server will be linked to the "route guidance system" to provide real-time route guidance and traffic congestion information. Navigation information will be announced to the user through the device's speaker.

[1174] How users use it

[1175] Users can use this system to ensure safety while driving. The device uses a photoelectric sensor and an image capture device to collect eye movement data and detects momentary sleep. If detected, users can receive an immediate warning.

[1176] In addition, safe and efficient driving is possible by receiving feedback based on driving behavior analysis provided by the server and navigation information from the route guidance system.

[1177] Example prompt sentence:

[1178] "Please explain in detail about the system that uses sensors and cameras to detect eyelid and eye movement, analyzes the data in real time, and issues a warning to prevent drowsy driving."

[1179] In this way, the system of the present invention ensures the safety of the driver and contributes greatly to preventing traffic accidents and improving driving efficiency.

[1180] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1181] Step 1:

[1182] As soon as the device starts driving, it activates a photoelectric sensor and image capture device to capture the driver's eye movement data in real time. This device samples data several dozen times per second. Specifically, image data that continuously captures the driver's eye movements and electrooculography data are input. This makes it possible to accurately detect the opening and closing of the eyelids and minute eye movements.

[1183] Step 2:

[1184] The device inputs the acquired eye movement data into a machine learning algorithm. This algorithm uses a pre-trained generative AI model to analyze instantaneous sleep patterns (e.g., prolonged blinking or gaze fixation). Specifically, it detects abnormal blink patterns and changes in gaze based on the input data to determine whether instantaneous sleep has occurred. The device then outputs whether instantaneous sleep has been detected.

[1185] Step 3:

[1186] When the machine learning algorithm detects a momentary sleep, the device immediately activates the light-emitting means (LED light) and the audio output means (speaker). This causes the LED light to light up and an alarm to sound from the speaker to notify the user of the momentary sleep. The output of this step is a visual and audio warning.

[1187] Step 4:

[1188] The device transmits all acquired eye movement data to an external computing device (server) via a transmission means. This data includes time, eyelid movement, detailed information on eye movement, etc. Secure HTTPS communication is used as the transmission protocol. This allows the server to receive detailed data in real time while driving.

[1189] Step 5:

[1190] The server stores the received data in a database. The stored data is then organized for analysis. For example, data cleaning is performed to remove incomplete or noisy data. This results in cleaned driving data.

[1191] Step 6:

[1192] The server analyzes driving characteristics using a generative AI model based on the accumulated data. This analysis determines patterns of instantaneous sleep during specific times and driving conditions, and generates useful feedback for the driver. For example, insights such as "instantaneous sleep occurs more frequently when driving at night" can be obtained. This feedback is notified to the user via email or a smartphone app.

[1193] Step 7:

[1194] The device is linked to a route guidance system. This link allows the device to obtain real-time traffic information and navigation data and announce it to the user through the device's speaker. For example, the device provides information on traffic congestion while driving and guidance on the optimal route to the destination. This allows the user to reach their destination efficiently and safely.

[1195] Step 8:

[1196] The server evaluates the driver's safe driving based on the driving data. Depending on the evaluation results, the server offers the user a discount on insurance premiums. The evaluation results are converted into a numerical value for the driver's safe driving level, and the data is sent to the insurance company based on that. Details of the discount are provided to the user via email or app notification.

[1197] (Application example 1)

[1198] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1199] Conventional driver monitoring systems have limitations in not only detecting momentary drowsiness but also in providing immediate warnings to drivers. There are also issues with how to provide feedback to drivers based on the analysis results of acquired driving data, and how to provide incentives such as insurance discounts. Furthermore, there is a lack of means to provide real-time traffic information obtained while driving. Under these circumstances, there are problems with ensuring driver safety and providing a comfortable driving environment.

[1200] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1201] In this invention, the server includes means including an electrooculography sensor and a camera device that instantly detects eyelid and eyeball movements, means including an artificial intelligence algorithm that analyzes data acquired by the sensor and camera device to detect momentary sleep, light emitting means that issues a visual warning in response to the detection of momentary sleep, audio output means that issues an audio warning in response to the detection of momentary sleep, communication means that communicates data from the sensor and camera device, external server means that accumulates and analyzes the data transmitted by the communication means, navigation linkage means that provides the latest traffic information acquired from the external server means via the audio output means, and means that provides insurance premium discount information based on driving data evaluated by the external server means. This makes it possible to instantly detect drowsy driving while driving and issue visual and audio warnings, and also to provide feedback and insurance premium discount information based on the results of analysis of driving characteristic data, and the latest navigation information in real time.

[1202] Definitions of important words

[1203] The eyelid is a thin layer of skin that protects the eye and allows you to blink.

[1204] The "eyeball" is a spherical organ that senses visual information and is the main part of the human eye.

[1205] An "electrooculography sensor" is a device that detects minute electrical activity in the eyelids and eyeballs and converts it into digital data.

[1206] A "camera device" is a device that captures images and videos and stores and analyzes them as digital data.

[1207] An "artificial intelligence algorithm" is a computational processing method that analyzes large amounts of data, learns patterns, and makes appropriate decisions and predictions.

[1208] A "light emitting means" is a device or system that emits light to visually convey information.

[1209] "Audio output means" refers to a device or system for conveying information using audio.

[1210] "Communication means" refers to the technology or equipment used to transmit data to other devices or servers.

[1211] The "external server means" is a remote server system for storing and analyzing data via a network.

[1212] The "navigation-linked means" is a system that links with the navigation system and provides traffic information and route guidance in real time.

[1213] "Driving characteristics" refers to a driver's behavioral characteristics such as driving habits, patterns, and reaction speed.

[1214] "Insurance premium discount information" is information about insurance premium discounts that are offered when criteria such as safe driving are met.

[1215] MODE FOR CARRYING OUT THE INVENTION

[1216] System configuration

[1217] This invention is a multi-function system for preventing drowsy driving and ensuring the safety of the driver. The elements that specifically constitute the system will be described below.

[1218] 1. Eyelid and eyeball movement detection device and artificial intelligence algorithm

[1219] The device is equipped with an electrooculography sensor and a camera device to instantly detect eyelid and eyeball movements. This allows for real-time monitoring of the driver's eyelid and eyeball movements while driving. The acquired eyeball movement data is analyzed by an artificial intelligence algorithm to detect microsleep.

[1220] 2. Visual and audible warning devices

[1221] When the device detects momentary drowsiness, an LED mounted on the frame lights up to provide a visual warning to the driver. In addition, an audible alert sounds from the speaker, providing an audible warning. This helps to draw the driver's attention and prevent drowsy driving.

[1222] 3. Data communication and external servers

[1223] The terminal includes a communication module for transmitting real-time acquired sensor data to an external server, which receives and stores the data. The stored data is used to analyze the driver's driving characteristics and provide feedback to the driver.

[1224] 4. Navigation linkage

[1225] The device is linked to the navigation system, allowing drivers to receive real-time navigation and traffic information through the speaker. For example, it will announce the optimal route to the destination or changes in road conditions.

[1226] 5. Premium discount function

[1227] The server provides drivers with discounts on insurance premiums through collaboration with insurance companies. Specifically, it evaluates safe driving based on driving data and discounts insurance premiums according to the evaluation results. This function provides drivers with an incentive to drive safely.

[1228] Program processing explanation

[1229] The device uses sensors and a camera device to collect eye movement data. An artificial intelligence algorithm then analyzes this data in real time to detect momentary sleep. When a momentary sleep is detected, the device will illuminate LEDs on the frame and play an audible alert through the speaker to provide visual and audio warnings.

[1230] The collected data is sent to an external server via the communication module. The server stores this data and analyzes driving characteristics. The analysis results are provided to the driver as feedback.

[1231] Furthermore, the terminal is linked to the navigation system, has the function of obtaining the latest traffic information and announcing navigation and congestion information to the driver through a speaker. The server also evaluates the driver's safe driving based on the driving data and notifies the driver of insurance premium discount information according to the evaluation results.

[1232] Specific usage example

[1233] For example, consider a case where a user drives a car wearing smart glasses equipped with this system. While driving, the device's electrooculography sensor and camera device collect eye movement data and detect momentary sleep. When momentary sleep is detected, the device's LED lights up and an alert sounds from the speaker to warn the driver. At the same time, the sensor data is sent from the device to a server, where it is stored and analyzed. The device then obtains the latest traffic information from the navigation system and announces it to the driver through the speaker. Finally, the server evaluates insurance premium discounts based on the driving data and notifies the driver of this information.

[1234] A specific example of a prompt sentence is "Analyze my current driving data and tell me the insurance premium discount rate."

[1235] As described above, the system of the present invention ensures the safety of drivers and makes it possible to prevent traffic accidents.

[1236] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1237] Program processing flow

[1238] Step 1:

[1239] Input: Data from electrooculography sensors and camera devices for real-time detection of eyelid and eye movements.

[1240] How it works: The device activates the electrooculography sensor and camera device to collect data on the driver's eyelid and eyeball movements in milliseconds, making it possible to understand the driver's eye movement, blink frequency, and eye fatigue.

[1241] Output: Collected eye movement data.

[1242] Step 2:

[1243] Input: Collected eye movement data.

[1244] How it works: The device's AI algorithm analyzes collected data in real time to detect microsleep. It processes the data by analyzing eye movement speed and eyelid opening / closing patterns, and determines that a microsleep has occurred if it exceeds a certain threshold.

[1245] Output: Instantaneous sleep detection result.

[1246] Step 3:

[1247] Input: Instantaneous sleep detection result.

[1248] Specific operation: When the device detects momentary sleep, it immediately activates the warning system. A visual warning is provided by the LED on the frame, and an audible warning is provided by the speaker. This immediately alerts the driver.

[1249] Output: Warning signals to the driver (visual and audible).

[1250] Step 4:

[1251] Input: Collected eye movement data.

[1252] Specific operation: The terminal uses the communication module to send the collected data to an external server. Specifically, the data is compressed and encrypted before being sent to the server via a communication protocol.

[1253] Output: Sending encrypted and compressed data to an external server.

[1254] Step 5:

[1255] Input: Data sent from the terminal.

[1256] How it works: The server accumulates the received data and analyzes the driver's driving characteristics. The analysis uses a generative AI model to identify multiple driving patterns and identify the driver's habits and characteristics.

[1257] Output: Insights into driving characteristics.

[1258] Step 6:

[1259] Input: Latest traffic and navigation data.

[1260] Specific operation: The terminal works in conjunction with the navigation system to announce real-time traffic information and the optimal route to the driver via voice output. In this process, the terminal analyzes the traffic information obtained via the communication module and selects the content that is most appropriate for the driver.

[1261] Output: Navigation information and traffic announcements to the driver.

[1262] Step 7:

[1263] Input: Accumulated driving data and driving characteristics analysis results.

[1264] Specific operation: The server evaluates safe driving based on driving data and generates insurance premium discount information based on the evaluation results. The generated insurance premium discount information is notified to the driver via the terminal.

[1265] Output: Notification of insurance premium discount information to the driver.

[1266] Through the above processing steps, this system ensures driver safety and makes it possible to prevent traffic accidents.

[1267] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1268] This invention is a system that includes an electrooculography sensor, a camera device, an artificial intelligence algorithm, an emotion engine, and a communication means and external server that link these together to prevent drowsy driving and ensure driver safety. The system aims to reduce the risk of accidents while driving by monitoring and evaluating the user's condition from multiple angles and issuing warnings at appropriate times.

[1269] System configuration

[1270] 1. Detection equipment and artificial intelligence algorithms

[1271] The device is equipped with an electrooculography sensor and a camera device to detect the driver's eye and eyelid movements. This allows real-time collection of eye movement data and eyelid movements while driving. This data is analyzed by an artificial intelligence algorithm to detect microsleep.

[1272] 2. Emotion Engine

[1273] The device also features an emotion engine that can recognize emotions by analyzing the user's facial expressions and voice. The emotion engine detects the user's stress level, fatigue level, and new emotional changes, and combines this data with other sensor information to provide a comprehensive condition assessment.

[1274] 3. Warning measures

[1275] When the device detects high stress or fatigue as assessed by the instantaneous sleep and emotion engine, it immediately issues visual and audible warnings. Specifically, the LED on the frame lights up to provide a visual warning, and an audible alert sounds from the speaker. Furthermore, if the emotion engine detects high stress or fatigue, the intensity of the warnings is automatically adjusted.

[1276] 4. Data communication and analysis

[1277] The device transmits the collected sensor data and emotional data via a communication module to an external server, which receives, stores, and analyzes the data. The analysis reveals the driver's driving characteristics, attentional tendencies, emotional state, and other information.

[1278] 5. Navigation system linkage

[1279] The device is equipped with a function that links with the navigation system, obtaining the latest traffic and congestion information and announcing it to the driver in real time through a speaker, allowing the driver to obtain the appropriate information to reach their destination safely.

[1280] 6. Premium discount function

[1281] The server works with insurance companies to offer drivers discounts on their insurance premiums. In particular, if the emotion engine records positive emotions, it will reflect that in the evaluation of insurance premiums. When a discount is applied, the details are notified to the driver.

[1282] Program processing

[1283] Data collection and instantaneous sleep detection

[1284] The device uses an electrooculography sensor and a camera to collect eye movement data. This data is analyzed in real time by an artificial intelligence algorithm to detect momentary sleep. The device's emotion engine also analyzes the user's facial expressions and voice data to recognize emotions. Emotional data is also collected in real time and used to comprehensively evaluate the user's state.

[1285] Actions triggered by the warning

[1286] If the device detects momentary sleep, it will simultaneously issue a visual and audible alert. The LED on the frame will light up and an alert sound will be emitted from the speaker. The intensity of the alert will also automatically adjust if the emotion engine detects high stress or fatigue.

[1287] Data transmission and analysis

[1288] The device sends the collected sensor data and emotional data to an external server, which then stores the received data and analyzes the driver's driving characteristics and emotional state. As a result, it becomes possible to understand the driver's behavioral patterns, changes in attention, and emotional trends.

[1289] Providing navigation information

[1290] The device connects to the navigation system to obtain the latest traffic information and announces traffic conditions and route guidance to the driver through a speaker. This information is provided in real time to assist the driver in making decisions.

[1291] Insurance premium discount notice

[1292] The server evaluates safe driving based on driving data and emotional data and offers discounts on insurance premiums. If the emotion engine detects positive emotions, it will also be reflected in the evaluation, and the driver will be notified when a discount is applied.

[1293] Specific usage example

[1294] The user drives a car using AI-enabled glasses. While driving, the device's electrooculography sensor and camera device collect eye and eyelid movements in real time, and an artificial intelligence algorithm detects momentary sleep. At the same time, the device's emotion engine analyzes the user's facial expressions and voice data to assess their stress and fatigue level. If momentary sleep or high stress is detected, an LED on the device's frame lights up and an alert sounds from the speaker to warn the user. All collected data is sent to a server in real time, where it is stored and analyzed. The device works in conjunction with the navigation system to provide the latest traffic information to support the driver. The server also evaluates the driver's safe driving, assesses insurance premium discounts, and notifies the driver.

[1295] In this way, the system is a powerful tool for monitoring the user's multifaceted condition and reducing risks while driving.

[1296] The processing flow will be explained below.

[1297] Step 1:

[1298] The user turns on the AI-enabled glasses, which puts the system into initial setup mode.

[1299] Step 2:

[1300] The device automatically calibrates the electrooculography sensor in the nose pad and the camera in the frame, a process that allows the device to accurately detect the movement of the user's eyes and eyelids.

[1301] Step 3:

[1302] The device measures eye and eyelid movements in real time while driving. Electrooculography sensors and a camera device collect data, which is then sent to an artificial intelligence algorithm for analysis.

[1303] Step 4:

[1304] The device's AI algorithm analyzes the collected data to detect microsleep. Specifically, it analyzes patterns such as minute eye movements and eyelid closure, and if it determines that a microsleep has occurred, it moves on to the next step.

[1305] Step 5:

[1306] The device's emotion engine analyzes the user's facial expressions and voice to assess their stress level and fatigue. This emotion data is then combined with instantaneous sleep detection data to provide a comprehensive assessment of their condition.

[1307] Step 6:

[1308] If the device detects a moment of deep sleep or high stress, it will immediately issue a visual and audible alert: an LED on the frame will light up and an alert sound will sound from the speaker.

[1309] Step 7:

[1310] The device transmits collected sensor data and emotional data, including eye movements, eyelid movements, timing of momentary sleep, and emotional state, to an external server via a communication module.

[1311] Step 8:

[1312] The server stores the received data and analyzes the driver's driving characteristics and emotional state, specifically analyzing trends in the driver's attention and changes in stress level.

[1313] Step 9:

[1314] The server then provides the driver with feedback based on the analysis results, including advice on times when attention may be waning due to long driving periods and advice on avoiding driving under high stress conditions.

[1315] Step 10:

[1316] The device connects to the navigation system to obtain the latest traffic and congestion information, which is then provided to the driver in real time through a speaker in the frame.

[1317] Step 11:

[1318] The server connects with insurance companies and evaluates insurance discounts based on safe driving and emotional state data. If the emotion engine records positive emotions, they are also reflected in the evaluation.

[1319] Step 12:

[1320] The server notifies the driver of details when the discount is applied, including the discount rate and the period of application.

[1321] Step 13:

[1322] After the user has worn the glasses for a day, the system automatically provides a summary of the analysis results, including the number of warnings encountered while driving, fluctuations in emotional state, and the total amount of driving data.

[1323] Through these steps, the system not only prevents drowsy driving, but also contributes to comprehensive driver condition management, supports safe driving, and provides incentives through insurance premium discounts.

[1324] Example 2

[1325] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1326] Drowsiness and high stress while driving are cited as causes of traffic accidents. Conventional driving monitoring systems detect momentary drowsiness by monitoring only eye and eyelid movements, but this alone makes it difficult to fully grasp the driver's overall condition. Furthermore, they lack the functionality to evaluate the driver's emotional state and fatigue level in real time and adjust the intensity of warnings according to the situation. Therefore, the objective of this invention is to reduce the risk of traffic accidents by detecting momentary drowsiness, evaluating the driver's emotional state and fatigue level, and issuing prompt and appropriate warnings to the driver.

[1327] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means including a sensor and a video capture device that instantly detect eyelid and eyeball movements, means including a machine learning algorithm that analyzes data acquired by the sensor and the video capture device to detect subtle sleep states, and means for recognizing emotions by analyzing the user's facial expressions and voice. This makes it possible to evaluate the driver's momentary sleep and emotional state from multiple angles and issue a warning at an appropriate time.

[1328] A "sensor" is a device that detects physical quantities such as light, sound, and electric current and converts them into electrical signals.

[1329] A "video capture device" is a device that records images and videos using optical means.

[1330] A "machine learning algorithm" is a computational method for learning patterns from data and using the results to analyze new data.

[1331] "Light emitting means" means a device or method for emitting light, primarily used to provide visual notification.

[1332] "Audio output means" refers to a device or method for generating and outputting audio.

[1333] "Information transmission means" means a means for transmitting data from one point to another, and includes wired or wireless communication means.

[1334] An "external computer system" is an external computing device or server accessed over a network and used for data storage and analysis.

[1335] An "emotion recognition means" is a device or system that analyzes a user's facial expressions and voice and identifies their emotional state.

[1336] A "means for adjusting the intensity of the warning" is a device or function for varying the intensity of the warning depending on the detected condition.

[1337] "Traffic information" refers to the latest travel information necessary for drivers, such as road conditions, traffic congestion, and accident information.

[1338] This invention is a system that detects drowsiness and high stress while driving in real time and issues a warning. To ensure safe driving, this system works by combining sensors, video capture devices, machine learning algorithms, emotion recognition means, information transmission means, and external computer systems.

[1339] System configuration and functions

[1340] Data collection

[1341] The device is equipped with high-precision sensors and video capture devices to detect eyelid and eyeball movements in real time. These devices collect the user's eye movement data and eyelid movements in real time. For example, when a user moves their eyes left and right while driving, the data is detected immediately.

[1342] Detecting subtle sleep states

[1343] The collected data is analyzed by a machine learning algorithm installed on the device, which detects microsleep based on the user's eye and eyelid movement patterns. For example, if the user's eyelids remain closed for a few seconds, the system will identify this as a microsleep.

[1344] emotion recognition

[1345] The device is equipped with a means of recognizing emotions by analyzing the user's facial expressions and voice. This data is also collected in real time to evaluate the user's stress level and fatigue. For example, if a frowning expression or a tired voice is detected, it is judged to be in a state of high stress or fatigue.

[1346] Warning

[1347] If a state of microsleep or high stress is detected, the device will issue a visual and audible warning. Specifically, an LED on the frame will light up and an alert sound will be emitted from the speaker. The intensity of the warning will also be automatically adjusted depending on the situation evaluated by the emotion recognition system. As a result, the user will receive a warning at the appropriate time and be able to respond quickly.

[1348] Data communication and the role of external servers

[1349] The sensor data and emotion data collected by the device are transmitted to an external computer system via an information transmission medium. The server receives, stores, and analyzes this data. The analysis results provide insights into the user's driving characteristics and emotional state.

[1350] Providing navigation information

[1351] The device works in conjunction with the navigation system to obtain the latest traffic and congestion information and announce it to the user in real time. For example, it obtains information about highway congestion and provides voice guidance such as, "You can avoid the congestion by getting off at the next exit."

[1352] Insurance premium discount notification

[1353] The server evaluates safe driving based on the collected and analyzed driving data and emotional data. For example, if the emotion recognition means frequently detects positive emotions, that evaluation will be reflected in the insurance premium discount. The user will be notified when the discount is applied.

[1354] Specific usage example

[1355] «Example of prompt sentences to input to a generative AI model»

[1356] "Please explain how the system works to detect drowsiness or high stress while driving and issue a warning."

[1357] "Please tell me the specific processing method of the driver monitoring system using electrooculography sensors and an emotion engine."

[1358] "Please explain with examples how this system collects data, issues alerts, and analyzes the data."

[1359] This system enables multifaceted condition monitoring while the user is driving, and serves as a powerful tool to support safe driving.

[1360] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1361] Step 1: Data collection

[1362] The device uses sensors and video capture devices to detect eyelid and eye movements in real time. These devices collect the user's eye movement data and eyelid movements in real time. Specifically, they detect the user's gaze movements and blinking movements and generate electrical signals corresponding to those movements. These electrical signals are stored as data. The input is the user's eye movement and eyelid movement, which are detected by the sensors. The output is eye movement data and eyelid movement data.

[1363] Step 2: Detecting Minor Sleep States

[1364] The device inputs the collected eye movement data and eyelid movements into a machine learning algorithm to detect microsleep. For example, if the data shows that the eyes do not blink or the eyelids remain closed for a long period of time, the algorithm determines this to be microsleep. The input is the eye movement data and eyelid movement data obtained in the data collection step. The output is the detection result of the microsleep state, specifically a judgment result such as "microsleep has been detected."

[1365] Step 3: Emotion Recognition

[1366] The device collects the user's facial expressions and voice using emotion recognition means. Based on this, an emotion recognition algorithm analyzes the user's emotional state. Specifically, it detects facial muscle movements and tone of voice to evaluate the user's stress level and fatigue level. The input is the user's facial expression data and voice data. The output is the evaluation result of the emotional state, such as "a high stress level has been detected."

[1367] Step 4: Send an alert

[1368] If the device detects momentary sleep or a high-stress state, it issues a visual and audible warning. Specifically, an LED on the frame lights up and an alert sound is emitted from the speaker. The strength of the warning is also automatically adjusted according to the state evaluated by the emotion recognition means. The input is the detection result of the minute sleep state and the evaluation result of the emotional state. The output is a visual and audible warning.

[1369] Step 5: Send data

[1370] The terminal transmits the collected sensor data and emotion data to an external computer system via an information transmission means. Specifically, the data is collectively organized into packets and transmitted to a server using wireless communication technology. The input is the sensor data and emotion data. The output is the transmitted data packets.

[1371] Step 6: Data analysis

[1372] The server stores the received data and analyzes the driving characteristics and emotional state. This allows the user's behavioral patterns, changes in attention, and emotional trends to be understood. The input is the transmitted sensor data and emotional data. The output is the analysis results of the driving characteristics and emotional state.

[1373] Step 7: Providing navigation information

[1374] The terminal works in conjunction with the navigation system to obtain the latest traffic information and announces it to the driver through a speaker. For example, it obtains information about highway congestion and provides guidance such as, "You can avoid the congestion by getting off at the next exit." The input is the latest traffic and congestion information. The output is a real-time voice announcement.

[1375] Step 8: Premium Discount Notification

[1376] The server evaluates safe driving based on the collected and analyzed driving data and emotional data. If the evaluation result of safe driving is positive, it applies an insurance premium discount and notifies the user. The input is the safe driving evaluation result. The output is a notification of the insurance premium discount.

[1377] (Application example 2)

[1378] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1379] As autonomous vehicles become more widespread, safety when drivers switch to manual driving has become an important issue. In particular, there is a risk of accidents occurring due to drivers falling asleep at the wheel after long hours of driving or due to overwork, or due to distraction caused by stress. For this reason, there is a demand for systems that can monitor the driver's condition from multiple angles and issue immediate warnings.

[1380] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means including a sensor and a camera that instantly detects eyelid and eyeball movements, means including an artificial intelligence algorithm that analyzes data acquired by the sensor and the camera to detect momentary sleep, means including an emotion engine that analyzes the user's facial expressions and voice to recognize emotions, means for issuing visual and audio warnings in response to detection of a high-stress state, communication means for communicating data from the sensor and the camera, and external information processing device means for storing and analyzing data transmitted by the communication means. This makes it possible to reduce the risk of an accident by evaluating the driver's condition in real time and issuing warnings when necessary.

[1381] The "sensor" is a device for detecting eyelid and eyeball movements in real time.

[1382] The "photography device" refers to a camera device that is combined with a sensor to visually capture the movement of the eyelids and eyeballs.

[1383] The "artificial intelligence algorithm" is an advanced computational method for analyzing acquired data and detecting instantaneous sleep.

[1384] The "emotion engine" is a system that analyzes the user's facial expressions and voice to recognize their emotions.

[1385] "Visual warning" means a warning that is visually conveyed to the user by means of light.

[1386] "Light emitting means" means a device that emits light to a user to effect a visual warning.

[1387] An "auditory warning" is a warning that is given to the user through the ears of the user by a voice output means.

[1388] "Audio output means" refers to a device that outputs a sound to the user to implement an audible warning.

[1389] The "communication means" is a device for transmitting data from the sensor and the image capturing device to an external information processing device.

[1390] The "external information processing device" is a server for storing and analyzing data transmitted by communication means.

[1391] "Driving characteristics" refers to the behavioral patterns and tendencies of a user when driving a vehicle.

[1392] "Attention trends" refers to information obtained by analyzing changes in a user's concentration and attentiveness.

[1393] "Emotional state" refers to the user's emotional state or state of mind.

[1394] A "navigation system" is a system that provides guidance on the current location of a vehicle and the route to a destination.

[1395] This invention is a system that supports safe driving by combining a sensor and a camera that detects eyelid and eyeball movements in real time, an artificial intelligence algorithm that analyzes the acquired data to detect momentary sleep, and an emotion engine. The system is configured as follows:

[1396] 1. Data Collection

[1397] The sensors and imaging devices detect the driver's eye and eyelid movements in real time using standard camera devices and application-specific electrooculography sensors. These devices constantly monitor and collect data on eye and eyelid movements while the driver is operating the vehicle.

[1398] 2. Instantaneous sleep and emotion detection

[1399] The collected data is analyzed in real time by an artificial intelligence algorithm installed on the device. If momentary sleep is detected, the system immediately issues a warning. In addition, an emotion engine analyzes the driver's facial expressions and voice data to assess their level of stress and fatigue. This emotion data is also collected in real time and used to comprehensively evaluate the driver's condition.

[1400] 3. Issuance of warnings

[1401] If momentary sleep or a high stress state is detected, the device will issue visual and audio warnings. Visual warnings are issued by lighting up the device's LED using the light-emitting means to alert the driver. Audio warnings are issued by emitting an alert sound from the speaker via the audio output means to alert the driver.

[1402] 4. Data communication and transmission to external servers

[1403] The collected sensor data and emotional data are transmitted to an external information processing device via communication means. The external information processing device stores this data and analyzes the driver's driving characteristics, attentional tendencies, and emotional state. This clarifies the driver's driving behavior patterns and provides insights for safe driving.

[1404] 5. Linkage with navigation systems

[1405] The device works in conjunction with the navigation system to provide up-to-date traffic information and route guidance, which is announced to the driver via voice output means and provided in real time, helping drivers reach their destination safely and efficiently.

[1406] 6. Insurance premium discounts

[1407] The external information processing device evaluates the driver's safe driving based on driving data and emotional data, and offers discounts on insurance premiums. If particularly positive emotions are recorded, this will be reflected in the insurance premium evaluation, and the driver will be notified when a discount is applied.

[1408] Specific examples

[1409] The user drives an autonomous vehicle using AI-enabled glasses. While driving, sensors and a camera collect eye and eyelid movements in real time, and an artificial intelligence algorithm detects momentary sleep. At the same time, an emotion engine analyzes the user's facial expressions and voice data to assess their stress and fatigue levels. If momentary sleep or high stress is detected, an LED on the device's frame lights up and an alert sounds from the speaker to warn the user. All collected data is sent in real time to an external information processing device for storage and analysis. The device works in conjunction with the navigation system to provide the latest traffic information and support the user. The external information processing device also evaluates the user's safe driving, assesses insurance premium discounts, and notifies the user.

[1410] Prompt Sentence Examples

[1411] "I want to develop an application that can detect drowsiness and fatigue while driving and issue a warning. I need a mechanism to analyze frames from the camera in real time using dlib for face detection, and a function to evaluate emotions and detect high stress and fatigue."

[1412] "This program uses an AI model to analyze eye movement data to detect momentary sleep and prevent drowsiness at the wheel. It also uses an emotion engine to assess the driver's stress and fatigue level and issue a warning if necessary. It also sends the collected data to a server and links it with the navigation system."

[1413] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1414] Step 1:

[1415] Data collection (terminal):

[1416] The device collects data using sensors and imaging devices that detect eyelid and eye movements in real time. Specifically, the imaging device (camera) captures video and the sensor records electrical potential changes. As input, real-time video and electrical potential data are used, collected every millisecond. As output, a collected dataset is generated and sent to the next analysis step.

[1417] Step 2:

[1418] Instantaneous sleep detection (device):

[1419] The device inputs the collected eye movement data and eyelid movement data into an artificial intelligence algorithm to detect signs of instantaneous sleep. Specifically, the data is first preprocessed and converted into an appropriate format. The artificial intelligence algorithm then analyzes this data and calculates the probability of instantaneous sleep. The input is the preprocessed eye movement and eyelid movement data, and the output is the instantaneous sleep detection result (e.g., a probability value or a warning flag).

[1420] Step 3:

[1421] Emotional rating (device):

[1422] The device collects the user's facial expression and voice data and inputs it into the emotion engine. The emotion engine analyzes this data and evaluates the user's stress and fatigue levels. Specifically, the emotion engine analyzes the video and voice data to identify the user's emotional state. The input is facial expression and voice data, and the output is the emotion evaluation result (e.g., stress level, fatigue level).

[1423] Step 4:

[1424] Issue a warning (terminal):

[1425] If a momentary sleep or high stress state is detected, the device issues a visual and audible warning. Specifically, the light-emitting means turns on an LED light and the audio output means plays an alert sound. The inputs are the momentary sleep detection result and the emotion evaluation result, and the outputs are visual and audible warnings (e.g., lighting up an LED, playing an alert sound).

[1426] Step 5:

[1427] Data communication (terminal):

[1428] The collected and analyzed sensor data and emotion data are transmitted to an external information processing device via a communication means. Specifically, the data is packetized using an appropriate protocol and transmitted over a network. The input is the sensor data and emotion data, and the output is the transmission of data to the external information processing device.

[1429] Step 6:

[1430] Data storage and analysis (server):

[1431] The external information processing device accumulates and analyzes the transmitted data. Specifically, it stores the data in a database and uses data analysis algorithms to analyze driving characteristics, attention trends, and emotional states. The input is the transmitted sensor data and emotional data, and the output is the analysis results (e.g., driving characteristics report, attention trends).

[1432] Step 7:

[1433] Providing navigation information (device):

[1434] The terminal works in conjunction with the navigation system to obtain the latest traffic information and provide it to the user in real time. Specifically, it obtains traffic information and route guidance information from the navigation system and announces it to the user through voice output means. The input is traffic information from the navigation system, and the output is voice traffic information and route guidance.

[1435] Step 8:

[1436] Insurance premium discount notification (server):

[1437] The external information processing device evaluates the driver's safe driving based on the analyzed data and notifies the user if an insurance premium discount is to be offered. Specifically, the device applies a safe driving evaluation algorithm, generates an evaluation result, and notifies the user of insurance premium discount information as needed. The input is the analysis result, and the output is an insurance premium discount notification.

[1438] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1439] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1440] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1441] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1442] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1443] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1444] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1445] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1446] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1447] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1448] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1449] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1450] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1451] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1452] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1453] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1454] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1455] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1456] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1457] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1458] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1459] The following is further disclosed regarding the above embodiment.

[1460] (Claim 1)

[1461] means including an electrooculography sensor and a camera device for detecting eyelid and eyeball movements in real time;

[1462] means including an artificial intelligence algorithm for analyzing data acquired by the sensor and camera device to detect instantaneous sleep;

[1463] a light emitting means for issuing a visual warning in response to the detection of the instantaneous sleep;

[1464] a voice output means for issuing an auditory warning in response to the detection of the instantaneous sleep;

[1465] communication means for communicating data from the sensor and camera device;

[1466] an external server means for storing and analyzing data transmitted by the communication means;

[1467] A system including:

[1468] (Claim 2)

[1469] 2. The system of claim 1, wherein the external server means provides insight into the driver's driving characteristics based on the accumulated and analyzed data.

[1470] (Claim 3)

[1471] 2. The system according to claim 1, which cooperates with a navigation system and announces navigation and traffic congestion information through said voice output means.

[1472] "Example 1"

[1473] (Claim 1)

[1474] means for detecting eyelid and eyeball movements in real time, including a photoelectric sensor and an image capture device;

[1475] a machine learning algorithm that analyzes the data acquired by the sensor and the image acquisition device to detect instantaneous sleep;

[1476] a light emitting means for issuing a visual warning in response to the detection of the instantaneous sleep;

[1477] a voice output means for issuing an auditory warning in response to the detection of the instantaneous sleep;

[1478] transmitting means for communicating data from said sensors and image capture devices;

[1479] A system including an external computing device for storing and analyzing data transmitted by said transmitting means.

[1480] (Claim 2)

[1481] 10. The system of claim 1, wherein the external computing device provides insight into a driver's driving characteristics based on the accumulated and analyzed data.

[1482] (Claim 3)

[1483] 2. The system according to claim 1, which cooperates with a route guidance system and announces route guidance and traffic congestion information through said voice output means.

[1484] "Application Example 1"

[1485] New Claims

[1486] (Claim 1)

[1487] means including an electrooculography sensor and a camera device for detecting eyelid and eyeball movements in real time;

[1488] means including an artificial intelligence algorithm for analyzing data acquired by the sensor and camera device to detect instantaneous sleep;

[1489] a light emitting means for issuing a visual warning in response to the detection of the instantaneous sleep;

[1490] a voice output means for issuing an auditory warning in response to the detection of the instantaneous sleep;

[1491] communication means for communicating data from the sensor and camera device;

[1492] an external server means for storing and analyzing data transmitted by the communication means;

[1493] a navigation-linked means for providing the latest traffic information acquired from the external server means through a voice output means;

[1494] means for providing insurance premium discount information based on the driving data evaluated by the external server means;

[1495] A system including:

[1496] (Claim 2)

[1497] 2. The system of claim 1, wherein the external server means provides insight into the driver's driving characteristics based on the accumulated and analyzed data.

[1498] (Claim 3)

[1499] 2. The system according to claim 1, which cooperates with a navigation system and announces navigation and traffic congestion information through said voice output means.

[1500] ---

[1501] By rewriting the claims in this way, the features of the original invention as well as the new technical elements incorporated in the application examples were reflected in the claims.

[1502] "Example 2: Combining Emotion Engines"

[1503] (Claim 1)

[1504] a means including a sensor and a video capture device for detecting eyelid and eyeball movements in real time;

[1505] means for detecting minute sleep states by analyzing data acquired by the sensor and the video capture device, the means including a machine learning algorithm;

[1506] a light emitting means for providing a visual warning in response to detecting said slight sleep state;

[1507] an audio output means for issuing an audible warning in response to detecting said slight sleep state;

[1508] an information transmission means for communicating data from the sensor and the imaging device;

[1509] an external computer system that stores and analyzes the data transmitted by the information transmission means;

[1510] A means for recognizing emotions by analyzing the user's facial expressions and voice;

[1511] means for evaluating a state of a user based on emotion data acquired by the emotion recognition means;

[1512] a means for detecting a high stress state or fatigue evaluated by the emotion recognition means;

[1513] means for adjusting the intensity of a warning in response to the detection of high stress or fatigue;

[1514] A system including:

[1515] (Claim 2)

[1516] 10. The system of claim 1, wherein the external computer system provides insight into a driver's driving characteristics based on the accumulated and analyzed data.

[1517] (Claim 3)

[1518] 2. The system according to claim 1, which cooperates with a communication system and announces traffic information through said audio output means.

[1519] "Application example 2 when combining emotion engines"

[1520] (Claim 1)

[1521] A means including a sensor and a photographing device for detecting eyelid and eyeball movements in real time;

[1522] means including an artificial intelligence algorithm for detecting instantaneous sleep by analyzing data acquired by the sensor and the imaging device;

[1523] means including an emotion engine that analyzes a user's facial expressions and voice to recognize emotions;

[1524] a light emitting means for issuing a visual warning in response to the detection of the instantaneous sleep and the detection of the high stress state;

[1525] a voice output means for issuing an auditory warning in response to the detection of the instantaneous sleep and the detection of the high stress state;

[1526] communication means for communicating data from the sensors and the imaging device;

[1527] an external information processing device that stores and analyzes the data transmitted by the communication device;

[1528] A system including:

[1529] (Claim 2)

[1530] 10. The system of claim 1, wherein the external information processing means provides insights into a user's driving characteristics, attentional tendencies, and emotional state based on the accumulated and analyzed data.

[1531] (Claim 3)

[1532] 2. The system according to claim 1, which cooperates with a navigation system and announces navigation information and traffic information through said voice output means. [Explanation of symbols]

[1533] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means including an electrooculography sensor and a camera device for detecting eyelid and eyeball movements in real time; means including an artificial intelligence algorithm for analyzing data acquired by the sensor and camera device to detect instantaneous sleep; a light emitting means for issuing a visual warning in response to the detection of the instantaneous sleep; a voice output means for issuing an auditory warning in response to the detection of the instantaneous sleep; communication means for communicating data from the sensor and camera device; an external server means for storing and analyzing data transmitted by the communication means; A system including:

2. 10. The system of claim 1, wherein the external server means provides insight into a driver's driving characteristics based on the accumulated and analyzed data.

3. 2. The system according to claim 1, wherein the system cooperates with a navigation system and announces navigation and traffic congestion information through the voice output means.

Images