System

Abstract

A system is provided.SOLUTION: A system comprising: means for capturing a driver's face; means for analyzing the captured facial image and evaluating the driver's eye openness and facial expression; means for detecting signs of driver drowsiness or inattention; means for providing a warning if signs of drowsiness or inattention are detected; and means for storing all of this data so that an administrator can monitor in real time.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In the transportation industry, where long driving hours are the norm, accidents caused by drivers falling asleep or being careless at the wheel are frequent. Accidents caused by this can cause serious damage and risk human life, but large-scale technologies such as automatic braking systems are still in the early stages of widespread adoption, and urgent measures are needed. Therefore, a system is needed that can monitor driver drowsiness, carelessness, and health status in real time and respond quickly. [Means for solving the problem]

[0005] In order to solve the above problems, the present invention provides the following means: The present invention is composed of a system including means for capturing the driver's face, means for analyzing the captured facial image to evaluate the driver's eye opening / closing state and facial expression, means for detecting signs of drowsiness or inattention, means for issuing an alarm when signs of drowsiness or inattention are detected, and means for accumulating all of this data so that an administrator can monitor it in real time.

[0006] Specifically, a camera device installed in the vehicle captures the driver's face and sends the image data to a server. The server then uses an image analysis algorithm to evaluate the eye status and facial expressions to detect signs of drowsiness or inattention. If signs are detected, a warning instruction is sent to the driver via voice and text message. Through this process, the driver's safe driving is monitored in real time, enabling prompt action if necessary. In addition, a dashboard is provided that centrally manages all data and allows administrators to monitor and analyze it in real time via the Internet.

[0007] "Driver" means a person engaged in the transportation business who drives a vehicle.

[0008] "Capture" means obtaining an image of the driver's face using a camera device.

[0009] "Analysis" is the process of evaluating the eye status and facial expression based on captured facial image data.

[0010] "Drowsy driving" refers to a state in which a driver loses consciousness while driving and keeps their eyes closed for a certain period of time.

[0011] "Inattention" refers to a state in which a driver's attention is reduced and appropriate driving operations are not performed.

[0012] "Symptoms" are changes in the eye opening and closing state and facial expression that are observed as precursors to drowsiness or inattention.

[0013] "Warning" is a message issued to the driver to alert them when signs of drowsiness or inattention are detected.

[0014] A "means" is a method or device used to achieve a particular purpose.

[0015] A "system" is a structure that combines multiple interrelated elements to provide a set of functions.

[0016] A "camera device" is a device for capturing images and is installed in a vehicle.

[0017] A "server" is a computer system that processes and stores data.

[0018] An "image analysis algorithm" is a computational procedure for analyzing image data and detecting specific features or patterns.

[0019] A "voice message" is information that is communicated to the driver using voice.

[0020] A "text message" is information communicated to a driver using written text.

[0021] A "dashboard" is a user interface that visualizes data and allows administrators to grasp the situation in real time.

[0022] The "Internet" is a communications network that interconnects computers and networks around the world. [Brief explanation of the drawings]

[0023] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0024] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0025] First, the terms used in the following description will be explained.

[0026] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0027] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0028] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0029] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0030] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0031] [First embodiment]

[0032] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0033] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0034] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0035] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0036] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0037] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0038] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0039] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0040] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0041] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0042] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0043] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0044] This invention is a system for preventing drivers in the transportation industry from falling asleep or being distracted while driving, and is composed of the following elements:

[0045] 1. A way to capture the driver's face

[0046] The terminal includes a camera device mounted on the vehicle. This camera device periodically captures the driver's face and sends the captured images to a server, thereby obtaining images of the driver's eye status and facial expressions.

[0047] 2. A means of analyzing captured facial images to assess the driver's eye status and facial expression.

[0048] The server then inputs the captured facial image data into an image analysis algorithm, which evaluates the driver's condition by assessing eye status, facial color, and changes in facial expression.

[0049] 3. Measures to detect signs of drowsiness and inattention

[0050] The server detects signs of drowsiness or inattention based on the results of image analysis, such as when the eyes remain closed for a certain period of time or when the facial expression becomes neutral. These criteria are built into the image analysis algorithm.

[0051] 4. A means of issuing warnings if signs of drowsiness or inattention are detected

[0052] When the server detects signs of drowsiness or inattention, it sends a warning instruction to the device. Based on this instruction, the device issues a warning to the driver by voice or text message. An example of a voice warning is, "Caution: You may be drowsy at the wheel. Please take a break."

[0053] 5. A means to accumulate all this data and allow administrators to monitor it in real time

[0054] The server stores all warning history and analysis results in a database and provides an administrator dashboard connected via the internet, which allows administrators to monitor and manage the status of multiple drivers in real time.

[0055] Explaining program processing in natural language

[0056] Step 1: Initial Setup and Authentication

[0057] The user gets into the vehicle, starts up the terminal (camera device), and authenticates himself / herself.

[0058] The device sends authentication information to the server to verify that it is the driver.

[0059] Step 2: Data Acquisition

[0060] The device periodically (for example, every second) captures an image of the driver's face and sends it to the server.

[0061] Step 3: Data analysis

[0062] The server inputs the received facial images into an analysis algorithm to evaluate whether the eyes are open or closed and changes in facial expression.

[0063] The server assesses the driver's level of attention and detects signs of drowsiness or inattention.

[0064] Step 4: Warning occurs

[0065] If the server detects signs of drowsiness or inattention, it sends a warning instruction to the terminal.

[0066] Step 5: Warning Notification

[0067] The device will warn the driver via voice and text messages based on the received warning instructions.

[0068] Step 6: Data storage and management

[0069] The server stores all warning history and analysis results in a database.

[0070] Managers can monitor and manage the status of multiple drivers in real time using an internet-based dashboard.

[0071] Specific examples

[0072] For example, consider a situation where a driver keeps their eyes closed for three seconds while driving.

[0073] 1. The user starts driving, and the device (camera device) captures facial images every second and sends them to the server.

[0074] 2. The server analyzes the captured facial images and determines that the eyes are closed in the first image.

[0075] 3. Next, the eyes are recognized as closed in the next two images (total of 3 seconds).

[0076] 4. The server detects signs of drowsiness because the eyes remain closed for more than 3 seconds.

[0077] 5. The server sends a warning instruction to the terminal, and the terminal issues a voice warning saying, "Caution: You may be drowsy while driving. Please take a break."

[0078] 6. The server stores this warning event in a database and makes it available for administrators to view in real time on a dashboard.

[0079] This series of steps enables the system to detect drowsy or careless driving by the driver in real time and issue a prompt warning, contributing to maintaining a safe driving environment.

[0080] The processing flow will be explained below.

[0081] Step 1:

[0082] The user gets into the vehicle and turns on the terminal (camera device).

[0083] The device displays a screen for entering the driver's authentication information.

[0084] Step 2:

[0085] The user enters their authentication information (e.g., ID and password) into the terminal.

[0086] The terminal transmits this authentication information to the server.

[0087] Step 3:

[0088] The server compares the received authentication information with a database and authenticates the driver.

[0089] If the authentication is successful, the server returns a message to the terminal indicating that the driver authentication was successful.

[0090] Step 4:

[0091] The device will indicate that driver authentication was successful and prepare to enter capture mode.

[0092] Step 5:

[0093] The terminal uses a camera device to capture an image of the driver's face at regular intervals (e.g., every second).

[0094] The terminal transmits the captured images to the server in real time.

[0095] Step 6:

[0096] The server then feeds the received facial images into an image analysis algorithm.

[0097] The server analyzes the eye status and facial expressions to assess the driver's level of attention.

[0098] Step 7:

[0099] The server detects signs of drowsiness if the driver's eyes remain closed for a certain period of time (for example, three seconds or more).

[0100] The server also applies if signs of inattention (e.g., a blank expression) are detected.

[0101] Step 8:

[0102] The server sends a warning instruction to the terminal when signs of drowsiness or inattention are detected.

[0103] The server records this warning event in a database.

[0104] Step 9:

[0105] When the terminal receives a warning instruction from the server, it issues a warning to the driver via a voice message (for example, "Caution: You may be drowsy while driving. Please take a break.") or a text message.

[0106] Step 10:

[0107] The server stores all warning history and analysis results in a database and provides a dashboard that allows administrators to monitor in real time via the Internet.

[0108] Step 11:

[0109] Managers can monitor and manage the status of multiple drivers in real time through a provided dashboard.

[0110] Example 1

[0111] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0112] Many accidents occur due to drivers in the transportation industry falling asleep or being distracted while driving. This not only results in personal injury and property damage, and compromises safety, but also leads to economic losses. Therefore, a system is needed that can monitor driver attention in real time, quickly detect signs of drowsiness or inattention, and issue appropriate warnings.

[0113] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0114] In this invention, the server includes means for capturing the driver's face, means for analyzing the captured facial image and evaluating the driver's eye opening / closing state and facial expression, means for detecting signs of drowsiness or inattention of the driver, means for issuing a warning when signs of drowsiness or inattention are detected, and means for storing all warning history and analysis results in a database that an administrator can monitor in real time via the Internet. This makes it possible to monitor the driver's attention in real time and prevent drowsy driving and inattentive driving.

[0115] "Driver" refers to a person who performs driving duties, especially a person who operates a vehicle in the transportation industry.

[0116] "Facial capture means" means a camera device or equivalent installed in a vehicle for capturing an image of the driver's face.

[0117] "Means for analyzing images" refers to image analysis algorithms and software for evaluating the open / closed state of the eyes and changes in facial expression based on acquired facial images.

[0118] "Means for detecting signs of drowsiness or inattention" refers to algorithms or software that determine whether the driver is drowsy or inattentive from analyzed image data.

[0119] "Warning means" refers to devices or software that alert the driver via voice or text message if signs of drowsiness or inattention are detected.

[0120] "Means for storing in a database" refers to a database system for storing all warning history and analysis results.

[0121] "Means that allow an administrator to monitor in real time via the Internet" refers to a dashboard or web application that allows an administrator to monitor and manage the status of multiple drivers in real time via the Internet.

[0122] "Camera device" means an optical device mounted on a vehicle for capturing an image of the driver's face.

[0123] This invention is a system for preventing drivers in the transportation industry from falling asleep or being distracted while driving. The system includes a means for capturing the driver's face, a means for analyzing the captured facial image, a means for detecting signs of drowsiness or inattention, a means for issuing a warning, and a means for allowing a manager to monitor in real time.

[0124] Hardware Configuration

[0125] The terminal includes a camera device mounted on the vehicle, which periodically captures images of the driver's face and transmits the captured images to a server.

[0126] The server is a combination of a high-performance computer and database system, equipped with specialized software for running image analysis algorithms.

[0127] The administrator dashboard is a web-based application that allows real-time data monitoring and management over the Internet.

[0128] Software Configuration

[0129] The server includes the following software modules:

[0130] 1. Image capture module: processes image data received from the terminal.

[0131] 2. Image analysis module: The received image data is fed into an algorithm to evaluate the eye opening / closing status and changes in facial expression.

[0132] 3. Drowsiness detection algorithm: Based on the results of image analysis, it detects signs of drowsiness or inattention.

[0133] 4. Warning generation module: Generates warning instructions when signs of drowsiness or inattention are detected.

[0134] 5. Data storage module: All warning history and analysis results are stored in a database.

[0135] 6. Dashboard module: Provides a real-time monitoring interface for administrators.

[0136] System Operation

[0137] The user enters the vehicle, activates the terminal (camera device), and authenticates themselves. The terminal sends the authentication information to the server, and the authentication is confirmed.

[0138] The device periodically (for example, every second) captures an image of the driver's face and sends it to the server.

[0139] The server then inputs the received facial images into an analysis algorithm that evaluates the eye status and changes in facial expression to determine the driver's attention level and detect signs of drowsiness or inattention.

[0140] If the server detects signs of drowsiness or inattention, it sends a warning instruction to the terminal.

[0141] Based on the warning instructions received, the device will issue a warning to the driver via voice and text message.

[0142] The server stores all warning history and analysis results in a database, allowing administrators to monitor and manage the status of multiple drivers in real time using a dashboard connected via the Internet.

[0143] Specific examples

[0144] For example, if a driver keeps their eyes closed for three seconds while driving, here's a specific example:

[0145] 1. The user starts driving, and the device (camera device) captures facial images every second and sends them to the server.

[0146] 2. The server analyzes the captured facial images and determines that the eyes are closed in the first image.

[0147] 3. Next, the eyes are recognized as closed in the next two images (total of 3 seconds).

[0148] 4. The server detects signs of drowsiness because the eyes remain closed for more than 3 seconds.

[0149] 5. The server sends a warning instruction to the terminal, and the terminal issues a voice warning saying, "Caution: You may be drowsy while driving. Please take a break."

[0150] 6. The server stores this warning event in a database and makes it available for administrators to view in real time on a dashboard.

[0151] Prompt Sentence Examples

[0152] "Please explain the procedures and give specific examples of systems to prevent drivers from falling asleep or being distracted while driving in the transportation industry."

[0153] The system is designed to provide a concrete form to support the technical scope of an invention and to help others accurately understand and practice the invention.

[0154] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0155] Step 1:

[0156] The user gets into the vehicle, turns on the terminal (camera device), and authenticates himself / herself.

[0157] Input: User authentication information (e.g., facial recognition data or ID input)

[0158] Output: The device sends the user's authentication information to the server.

[0159] Specific operation: When the user turns their face towards the device, the camera captures an image and obtains authentication information.

[0160] Step 2:

[0161] The terminal sends the received authentication information to the server.

[0162] Input: User credentials captured by the device

[0163] Output: Authentication information sent to the server.

[0164] Specific operation: The terminal's sending function automatically sends the authentication information to the server.

[0165] Step 3:

[0166] The server compares the received authentication information with a database and returns the authentication result to the terminal.

[0167] Input: User credentials received by the server

[0168] Output: The authentication result (success or failure) is returned to the terminal.

[0169] What happens: The server searches its database and finds a record that matches the authentication information.

[0170] Step 4:

[0171] The device captures an image of the driver's face every second and sends it to a server.

[0172] Input: A face image captured by the device

[0173] Output: The image is sent to the server.

[0174] Specific operation: The device's camera operates once per second and transmits the captured images via the network.

[0175] Step 5:

[0176] The facial images received by the server are fed into an image analysis algorithm to evaluate the open / closed state of the eyes and changes in facial expression.

[0177] Input: A face image received by the server

[0178] Output: Eye status and facial expression evaluation results

[0179] What happens: The server runs image analysis software and analyzes each frame.

[0180] Step 6:

[0181] Based on the analysis results, the server determines the driver's level of attention and detects signs of drowsiness or inattention.

[0182] Input: Eye status and facial expression evaluation results obtained from the server

[0183] Output: Presence or absence of signs of drowsiness or inattention

[0184] Specific operation: The server monitors the image analysis results over a certain period of time and detects abnormalities.

[0185] Step 7:

[0186] If the server detects signs of drowsiness or inattention, it generates a warning instruction and sends it to the terminal.

[0187] Input: Detected signs of drowsiness or inattention

[0188] Output: Warning instruction (e.g. "Caution: You may be drowsy while driving. Please take a break.")

[0189] Specific operation: The server generates a warning message and sends it to the terminal via the network.

[0190] Step 8:

[0191] Based on the warning instructions received by the device, the device will warn the driver via voice and text messages.

[0192] Input: Warning instruction from the server

[0193] Output: Voice and text message alerts

[0194] Specific operation: The device will display a warning using audio equipment or a display, and will notify you by voice.

[0195] Step 9:

[0196] The server stores all warning history and analysis results in a database.

[0197] Input: Analysis results and warning history

[0198] Output: Warning history and analysis results stored in a database

[0199] Specific operation: The server adds and updates the analysis results and warning information to the database.

[0200] Step 10:

[0201] The server displays all warning history and real-time driver status through an administrator dashboard.

[0202] Input: Stored alert history and real-time data

[0203] Output: Information displayed on the admin dashboard

[0204] What it does: The server processes the collected data in real time and displays it in a web interface.

[0205] This series of processes makes it possible to monitor the driver's attention level in real time, quickly detect signs of drowsiness or inattention, and prevent accidents before they occur.

[0206] (Application example 1)

[0207] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0208] In conventional vehicle operation, there are systems that prevent drivers from falling asleep or being distracted while driving, but these systems do not provide comprehensive safety measures such as driver authentication or response measures when a warning is issued, such as vehicle speed control. As a result, risks remain if the driver does not respond immediately after receiving a warning. The present invention aims to solve these problems by not only monitoring the driver's state in real time and detecting signs of inattention or drowsiness, but also providing appropriate warning measures and additional safety measures if no response is received.

[0209] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0210] In this invention, the server includes means for capturing the driver's face, means for analyzing the captured facial image and evaluating the driver's eye opening / closing state and facial expression, means for detecting signs of driver drowsiness or inattention, means for issuing a warning when signs of drowsiness or inattention are detected, means for accumulating all of this data and enabling an administrator to monitor it in real time, means for authenticating the driver, and means for automatically reducing the vehicle speed if the warning is not responded to. This makes it possible to comprehensively monitor the driver's condition and improve safety.

[0211] "Means for capturing the driver's face" refers to an apparatus and method for periodically capturing images of the driver's face using a camera device mounted on the vehicle.

[0212] "Means for analyzing captured facial images and evaluating the driver's eye opening / closing state and facial expression" refers to technology and algorithms that use captured facial images to analyze the driver's eye opening / closing state and changes in facial expression in real time.

[0213] "Means for detecting signs of driver drowsiness or inattention" refers to technology and algorithms that identify signs of drowsiness or inattention based on analyzed data on eye status and facial expressions, and trigger necessary actions.

[0214] "Means for issuing a warning when signs of drowsiness or inattention are detected" refers to a device and method that issues a warning to the driver via voice or text message when the driver shows signs of drowsiness or inattention.

[0215] "A means of accumulating all this data and allowing administrators to monitor it in real time" refers to a system that stores driver status data centrally on a server and allows administrators to monitor that data in real time via a dashboard over the Internet.

[0216] "Means for authenticating a driver" refers to technology and devices that authenticate the driver when he or she gets into a vehicle and transmit the authentication information to a server for confirmation.

[0217] "Means for automatically reducing vehicle speed when warnings are not responded to" refers to mechanisms and methods that automatically reduce vehicle speed to maintain a safe driving environment if the driver does not respond to a warning.

[0218] The present invention is an advanced monitoring system for preventing drivers from falling asleep at the wheel or from driving carelessly. Next, an embodiment of the present invention will be described in detail.

[0219] System Configuration

[0220] The system includes the following components:

[0221] 1. Means for capturing the driver's face: Images of the driver's face are periodically captured using a camera device mounted on the vehicle.

[0222] 2. A means for analyzing the captured facial image and evaluating the driver's eye opening / closing state and facial expression: Using an image analysis algorithm on the server, the driver's eye opening / closing state and facial expression are evaluated from the captured image.

[0223] 3. Means of driver authentication: When the driver gets into the vehicle, personal authentication is performed and the information is sent to the server.

[0224] 4. Means for detecting signs of driver drowsiness or inattention: An algorithm is used to detect signs of drowsiness or inattention, such as when the driver opens and closes their eyes repeatedly within a specified period of time.

[0225] 5. Means of issuing a warning when signs of drowsiness or inattention are detected: Based on the detection results, a warning instruction is sent to the device and a warning is issued by voice or text message.

[0226] 6. Means for automatically reducing vehicle speed when warning is not responded to: If a warning is not responded to, the speed of the vehicle is automatically reduced.

[0227] 7. A means of accumulating all this data and allowing administrators to monitor it in real time: All warning history and analysis results are stored in a database, and an administrator dashboard is provided that is connected via the Internet.

[0228] Hardware and Software Configuration

[0229] Hardware: camera devices, vehicle control units, network connectivity devices

[0230] Software: Facial recognition algorithm, eye open / close state analysis algorithm, facial expression analysis algorithm, database management system, warning generation system, vehicle speed control program, administrator dashboard

[0231] Data processing and calculation

[0232] The server receives image data sent from the camera device and analyzes facial recognition, eye status, and facial expressions using image processing libraries such as TensorFlow and OpenCV. The server comprehensively evaluates this data and detects signs of drowsiness or inattention.

[0233] If signs of drowsiness or inattention are detected, the server sends a warning instruction to the vehicle's warning system, which then alerts the occupants via smartphone or head-mounted display. If the warning is not responded to, the vehicle control unit automatically reduces the speed.

[0234] Specific examples

[0235] For example, consider a case where a driver keeps their eyes closed for three seconds while driving. The camera device captures an image of their face every second and sends it to the server. The server analyzes the image and recognizes that their eyes are closed. If the server determines that their eyes are still closed for the next two seconds, it issues a warning that the driver may be drowsy. If the driver does not respond to this warning, the vehicle's speed will be automatically reduced to ensure safety.

[0236] Prompt Sentence Examples

[0237] "Develop an algorithm that analyzes facial images captured periodically by the camera to assess eye status and facial expression. We hope to create a real-time surveillance application, including a system that issues an alert if it detects signs of drowsiness or inattention."

[0238] By implementing the invention in this manner, it is possible to prevent drivers from falling asleep or driving carelessly, thereby providing a safe driving environment.

[0239] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0240] Step 1:

[0241] The user gets into the vehicle, activates the terminal (camera device), and authenticates himself / herself.

[0242] Input: User authentication information (e.g., biometrics or ID card)

[0243] Data processing: The device sends the authentication information to the server and authenticates the driver.

[0244] Output: Authentication result (success or failure)

[0245] Specific operation: The device captures the user's face with a camera and checks whether it matches a familiar driver.

[0246] Step 2:

[0247] The device periodically (for example, every second) captures an image of the driver's face and sends it to the server.

[0248] Input: Image data obtained from the camera device (driver's face)

[0249] Data processing: Images are captured and sent to the server

[0250] Output: Latest facial image data stored on the server

[0251] Specific operation: The device's camera captures the driver's face every second and sends the image to the server.

[0252] Step 3:

[0253] The server inputs the received facial images into an analysis algorithm to evaluate whether the eyes are open or closed and changes in facial expression.

[0254] Input: Facial image data sent to the server

[0255] Data calculation: Uses face recognition and facial expression analysis algorithms to assess eye status and facial expressions

[0256] Output: Evaluation results of eye opening / closing status and facial expression

[0257] How it works: The server uses a facial recognition algorithm to detect whether the eyes are open or closed, and based on this, determines signs of drowsiness or inattention.

[0258] Step 4:

[0259] The server detects signs of drowsiness or inattention based on the eye status and facial expression evaluation results.

[0260] Input: Eye status and facial expression evaluation results

[0261] Data calculation: Algorithms that detect signs of drowsiness or inattention based on predetermined criteria

[0262] Output: Detected signs of drowsiness or inattention (detected / not detected)

[0263] Specific operation: The server analyzes data such as the time the eyes are closed and periods of expressionlessness to detect drowsiness or inattention.

[0264] Step 5:

[0265] If the server detects signs of drowsiness or inattention, it sends a warning instruction to the terminal.

[0266] Input: Detected signs of drowsiness or inattention

[0267] Data processing: Generate warning instructions and send them to the device

[0268] Output: Warning instructions (audio warning or text message)

[0269] Specific operation: The server generates a warning such as "Caution: You may be drowsy while driving. Please take a break," and sends it to the terminal.

[0270] Step 6:

[0271] Based on the warning instructions received, the device will issue a warning to the driver via voice and text message.

[0272] Input: Warning instruction sent from the server

[0273] Data processing: Sending alerts in the form of voice synthesis or text display

[0274] Output: Driver warning (voice or text message)

[0275] Specific operation: The device will emit a warning message such as "Caution" using a speaker, for example.

[0276] Step 7:

[0277] If the server does not respond to the warning, it will issue an instruction to automatically reduce the vehicle's speed.

[0278] Input: Response status after warning (with / without response)

[0279] Data calculation: Generation of speed control instructions based on a countermeasure algorithm in the event of no response

[0280] Output: Speed ​​reduction command sent to vehicle control unit

[0281] Specific operation: The server checks that there is no response for a certain period of time and then issues an instruction to reduce the vehicle's speed.

[0282] Step 8:

[0283] The server stores all alert history and analysis results in a database and provides a dashboard for administrators.

[0284] Input: Warning history, analysis results

[0285] Data processing: Data storage in a database and dashboard provision via the internet

[0286] Output: Real-time monitorable user interface

[0287] Specific operation: The server accumulates all data and displays it on a dashboard for administrators to monitor in real time.

[0288] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0289] In order to prevent drivers from falling asleep or being distracted while driving, the present invention provides a system that combines an emotion engine, which is composed of the following elements:

[0290] 1. A way to capture the driver's face

[0291] The terminal includes a camera device mounted on the vehicle, which periodically captures the driver's face and transmits the captured image to a server to obtain information such as the eye status and facial expression.

[0292] 2. A means of analyzing captured facial images to assess the driver's eye status and facial expression.

[0293] The server then inputs the captured facial image data into an image analysis algorithm, which evaluates the driver's attention level by evaluating factors such as eye status, facial color, and changes in facial expression.

[0294] 3. Emotion engine that recognizes the driver's emotions

[0295] The server inputs the data obtained from image analysis into an emotion engine to recognize the driver's emotional state, which evaluates, for example, the driver's stress level, fatigue, and irritability.

[0296] 4. Measures to detect signs of drowsiness and inattention

[0297] The server combines the results of both image analysis and the emotion engine to comprehensively detect signs of drowsiness or inattention, such as when the eyes are closed for a certain period of time or when emotional states are detected in combination with signs such as a blank expression.

[0298] 5. A means of issuing warnings if signs of drowsiness or inattention are detected

[0299] If the server detects signs of drowsiness or inattention, it sends a warning instruction to the device. Based on this instruction, the device issues a warning to the driver by voice or text message. Depending on the driver's condition, the warning content may be something like "Caution: You may be drowsy at the wheel. Please take a break" or "Caution: Stress is building. Please stop and take a break."

[0300] 6. A means to store all data and enable administrators to monitor it in real time

[0301] The server stores all warning history and analysis results in a database and provides a dashboard that administrators can access via the internet, allowing them to monitor and manage the status of multiple drivers in real time.

[0302] Explaining program processing in natural language

[0303] Step 1: Initial Setup and Authentication

[0304] The user gets into the vehicle and turns on the terminal (camera device).

[0305] The user enters authentication information, and the terminal transmits the information to the server.

[0306] The server verifies the authentication information and notifies the terminal that the authentication was successful.

[0307] Step 2: Data Acquisition

[0308] The device periodically (e.g., every second) captures an image of the driver's face and sends it to the server.

[0309] Step 3: Data analysis

[0310] The server inputs the received facial images into an image analysis algorithm to evaluate the open / closed state of the eyes, facial expressions, and changes in facial color.

[0311] Step 4: Emotion Recognition

[0312] The server inputs the image analysis results into an emotion engine to identify the driver's emotional state.

[0313] The emotion engine assesses stress levels, fatigue, irritability, etc.

[0314] Step 5: Hazard detection

[0315] The server combines the analysis results, such as when the eyes are closed for a certain period of time (for example, more than three seconds) or when the face is expressionless, with the emotion engine's evaluation to detect signs of drowsiness or inattention.

[0316] Step 6: Warning occurs

[0317] The server sends a warning instruction to the terminal when it detects signs of drowsiness or inattention.

[0318] The terminal receives the warning instruction and issues a warning by voice or text message.

[0319] Step 7: Data storage and management

[0320] The server stores all warning history and analysis results in a database and provides a dashboard for administrators to monitor in real time.

[0321] Specific examples

[0322] For example, consider the case where a driver keeps their eyes closed for three seconds while driving and their facial expression is neutral.

[0323] 1. The user gets into the vehicle and turns on the terminal (camera device).

[0324] 2. The user enters their authentication information and the device sends it to the server.

[0325] 3. The server verifies the authentication information and notifies the terminal that authentication was successful.

[0326] 4. The device captures a facial image every second and sends it to the server.

[0327] 5. The server uses image analysis algorithms to analyze the eye status and facial expressions.

[0328] 6. The server is an emotion engine that evaluates the driver's emotional state (e.g., fatigue, stress).

[0329] 7. The server detects closed eyes for more than 3 seconds and a neutral facial expression, and combines this with the emotion engine's evaluation to recognize signs of drowsiness.

[0330] 8. The server sends a warning instruction to the terminal and generates a message warning of the possibility of drowsiness.

[0331] 9. The device will issue a voice warning saying, "Caution: You may be falling asleep at the wheel. Please take a break."

[0332] 10. The server stores this warning event in a database and allows administrators to view the situation in real time on a dashboard.

[0333] This specific example makes it possible to provide a safe driving environment by detecting drowsy or inattentive driving by the driver at an early stage and issuing a prompt warning. Furthermore, by combining it with an emotion engine, it becomes possible to make comprehensive judgments that take into account the driver's stress and fatigue state.

[0334] The processing flow will be explained below.

[0335] Step 1:

[0336] The user gets into the vehicle and turns on the terminal (camera device).

[0337] The device displays a screen for entering the driver's authentication information.

[0338] Step 2:

[0339] The user enters their authentication information (e.g., ID and password) into the terminal.

[0340] The terminal transmits this authentication information to the server.

[0341] Step 3:

[0342] The server compares the received authentication information with a database and authenticates the driver.

[0343] If the authentication is successful, the server notifies the terminal that the driver authentication was successful.

[0344] Step 4:

[0345] The device will indicate that driver authentication was successful and prepare to enter capture mode.

[0346] Step 5:

[0347] The terminal uses a camera device to capture an image of the driver's face at regular intervals (e.g., every second).

[0348] The terminal transmits the captured images to the server in real time.

[0349] Step 6:

[0350] The server inputs the received facial images into an image analysis algorithm to analyze the eye status and facial expressions.

[0351] The server inputs the image analysis results into an emotion engine to identify the driver's emotional state.

[0352] Step 7:

[0353] The server combines the analysis results, such as when the eyes are closed for a certain period of time (for example, more than three seconds) or when the face is expressionless, with the emotion engine's evaluation to detect signs of drowsiness or inattention.

[0354] Step 8:

[0355] If the server detects signs of drowsiness or inattention, it sends a warning instruction to the terminal.

[0356] The server records this warning event in a database.

[0357] Step 9:

[0358] When the terminal receives a warning instruction from the server, it issues a warning to the driver via voice or text message.

[0359] For example, it may notify you with a message such as, "Caution: You may be drowsy while driving. Please take a break."

[0360] Step 10:

[0361] The server stores all warning history and analysis results in a database and provides a dashboard that allows administrators to monitor in real time.

[0362] Step 11:

[0363] Managers can monitor and manage the status of multiple drivers in real time through a provided dashboard.

[0364] Example 2

[0365] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0366] Currently, there are systems in place to prevent drivers from falling asleep or being distracted while driving, but most of these systems simply rely on eye movements and changes in facial expression. This makes it difficult to comprehensively detect hazards that take into account the driver's emotional state (stress level, fatigue, etc.). Furthermore, there is no system in place to monitor warning history and analysis results in real time. This has led to the problem of insufficient measures being taken to ensure driver safety.

[0367] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for capturing the driver's face, a means for analyzing the captured facial image and evaluating the driver's eye opening / closing state and facial expression, a means for recognizing the driver's emotional state, a means for detecting signs of the driver's drowsiness or inattention, a means for issuing a warning when signs of drowsiness or inattention are detected, and a means for accumulating warning history and analysis results and allowing an administrator to monitor them in real time. This enables comprehensive hazard detection that takes into account not only the driver's eye opening / closing state and changes in facial expression, but also their emotional state. Furthermore, accumulating warning history and analysis results and allowing an administrator to monitor them in real time enables a rapid response, further ensuring the safety of the driver.

[0368] The "means for capturing the driver's face" is a device for periodically acquiring an image of the driver's face using a camera device mounted on the vehicle.

[0369] "Means for analyzing captured facial images and evaluating the driver's eye opening / closing status and facial expression" refers to a device or program that inputs the acquired facial image data into an image analysis algorithm to analyze the eye opening / closing status, facial color, changes in facial expression, etc.

[0370] The "means for recognizing the driver's emotional state" is a device or program that includes an emotion engine for identifying the driver's emotional state, such as stress level, fatigue, or irritability, based on data obtained from image analysis.

[0371] The "means for detecting signs of driver drowsiness or inattention" refers to a device or program that uses an algorithm that integrates the results of image analysis with the evaluation of an emotion engine, and evaluates the eye opening / closing status, changes in facial expression, and emotional state within a certain period of time to determine signs of drowsiness or inattention.

[0372] "Means for issuing a warning when signs of drowsiness or inattention are detected" means a device or program that issues a warning to the driver via voice or text message when signs of drowsiness or inattention are detected.

[0373] "Means for storing warning histories and analysis results and allowing administrators to monitor them in real time" refers to a device or program that stores all warning histories and analysis results in a database and provides a dashboard that administrators can access via the Internet.

[0374] This invention is a safe driving support system that combines facial image analysis and emotion recognition to prevent drivers from falling asleep or being careless while driving. This system is composed of the following elements.

[0375] 1. How to capture the driver's face:

[0376] The terminal includes a camera device mounted on the vehicle, which periodically captures the driver's face and transmits the captured image to a server to obtain information such as the eye status and facial expression.

[0377] 2. Means of analyzing the captured facial image:

[0378] The server inputs the received facial image data into an image analysis algorithm, possibly using software such as OpenCV or TensorFlow. The server evaluates the eye status, facial color, and changes in facial expression to analyze the driver's attention level.

[0379] 3. How to recognize driver emotions:

[0380] The server inputs the data obtained from image analysis into an emotion engine to recognize the driver's emotional state. The emotion engine can use AI models such as Microsoft's Azure Emotion API or Google Cloud's Face API. This allows the system to evaluate the driver's stress level, fatigue, irritability, etc.

[0381] 4. Measures to detect signs of drowsiness or inattention:

[0382] The server integrates the image analysis results with the emotion engine's evaluation to detect signs of drowsiness or inattention. Specifically, it uses an algorithm to determine whether the driver is drowsy or inattentive based on conditions such as eyes closed for a certain period of time (e.g., three seconds or more) or a blank expression.

[0383] 5. Means of issuing a warning if signs of drowsiness or inattention are detected:

[0384] If the server detects signs of drowsiness or inattention, it sends a warning instruction to the device. Based on this instruction, the device issues a warning to the driver by voice or text message. The warning content might be something like, "Caution: You may be drowsy at the wheel. Please take a break."

[0385] 6. A way to store all data and allow administrators to monitor it in real time:

[0386] The server stores all warning history and analysis results in a database and provides a dashboard that administrators can access via the internet, allowing them to monitor and manage the status of multiple drivers in real time.

[0387] Specific examples

[0388] For example, consider the case where a driver keeps their eyes closed for three seconds while driving and their facial expression is neutral.

[0389] 1. The user gets into the vehicle and turns on the terminal (camera device).

[0390] 2. The user enters their authentication information and the device sends it to the server.

[0391] 3. The server verifies the authentication information and notifies the terminal that authentication was successful.

[0392] 4. The device captures a facial image every second and sends it to the server.

[0393] 5. The server uses image analysis algorithms to analyze the eye status and facial expressions.

[0394] 6. The server is an emotion engine that evaluates the driver's emotional state (e.g., fatigue, stress).

[0395] 7. The server detects closed eyes for more than 3 seconds and a neutral facial expression, and combines this with the emotion engine's evaluation to recognize signs of drowsiness.

[0396] 8. The server sends a warning instruction to the terminal and generates a message warning of the possibility of drowsiness.

[0397] 9. The device will issue a voice warning saying, "Caution: You may be falling asleep at the wheel. Please take a break."

[0398] 10. The server stores this warning event in a database and allows administrators to view the situation in real time on a dashboard.

[0399] Example prompts for generative AI models

[0400] A detailed description of the system can be automatically generated by using the following prompts to input data into the generative AI model:

[0401] Based on the following information, generate a detailed description of a system that prevents drivers from falling asleep or being distracted while driving:

[0402] The system consists of a camera device, a server, and an emotion engine.

[0403] 1. The vehicle-mounted camera periodically captures the driver's face and sends it to a server.

[0404] 2. The server analyzes the image and evaluates the eye status and facial expression.

[0405] 3. The emotion engine identifies the driver's emotional state based on the analysis results.

[0406] 4. The server detects signs of drowsiness or inattention, such as closed eyes for a period of time or a blank expression.

[0407] 5. The server sends a warning instruction to the device and warns the driver via voice or text message.

[0408] 6. All data is stored on the server and can be viewed on a dashboard where administrators can monitor in real time.

[0409] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0410] Step 1: Initial Setup and Authentication

[0411] The user gets into the vehicle and turns on the terminal (camera device). The terminal checks the connection between the camera device and the network. The user enters authentication information (e.g., ID and password) as input, which the terminal sends to the server.

[0412] The server checks the received authentication information against the database to see if the authentication was successful, and notifies the terminal that the authentication was successful.

[0413] Step 2: Data Acquisition

[0414] After successful authentication, the device periodically (e.g., every second) captures an image of the driver's face using a camera device, and the captured face image data is generated as input.

[0415] The terminal compresses the captured image data and sends it to the server. The output compressed data is sent and reaches the server.

[0416] Step 3: Image analysis

[0417] The server feeds the received facial image data into an image analysis algorithm, which receives the compressed facial image data as input.

[0418] The server uses software such as OpenCV and TensorFlow to analyze eye status, facial color, and facial expression changes, and generates data on eye status and facial expressions as output.

[0419] Step 4: Emotion Recognition

[0420] The server inputs the data obtained from the image analysis into the emotion engine, providing data such as eye opening and closing status and facial expressions.

[0421] The emotion engine uses AI models (e.g., Azure Emotion API, Google Cloud Face API) to identify the driver's emotional state (e.g., stress level, fatigue, irritability). As an output, data about the driver's emotional state is generated.

[0422] Step 5: Hazard detection

[0423] The server integrates data on eye opening and closing and facial expressions with the emotion engine's evaluation. Analysis data and emotional state data are integrated as input.

[0424] The server runs an algorithm to detect signs of drowsiness or inattention. Specifically, it evaluates whether the eyes are closed for a certain period of time (e.g., more than 3 seconds) or whether the face is neutral. As an output, it generates warning data about signs of drowsiness or inattention.

[0425] Step 6: Warning occurs

[0426] The server sends a warning instruction to the terminal when it detects signs of drowsiness or inattention. As input, a warning instruction is generated.

[0427] The device receives the warning instruction and issues a warning via voice or text message. Specifically, it uses a generative AI model to generate a prompt sentence: "Caution: You may be drowsy at the wheel. Please take a break." As an output, the warning message is provided to the driver.

[0428] Step 7: Data storage and management

[0429] The server stores all the warning history and analysis results in a database. As input, the warning data and analysis data are stored in the database.

[0430] The server provides a dashboard that can be accessed by the administrator. Specifically, it builds an interface that allows the administrator to monitor the driver status in real time. As an output, a dashboard that can be accessed by the administrator is created.

[0431] (Application example 2)

[0432] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0433] Improving safety is a key issue for modern self-driving vehicles. Driver drowsiness and inattention can lead to serious accidents, so technology is needed to detect these early and provide appropriate warnings. Furthermore, there is a lack of systems that can make comprehensive judgments by taking into account the driver's emotional state, and this needs to be addressed.

[0434] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing the driver's face, means for analyzing the captured facial image and evaluating the driver's eye opening / closing state and facial expression, means for recognizing the driver's emotional state using an emotion engine, means for detecting signs of the driver's drowsiness or inattention, means for issuing a warning when signs of drowsiness or inattention are detected, means for accumulating all data so that an administrator can monitor it in real time, and means for transmitting data to a cloud server for analysis and issuing a warning instruction. This enables early detection of the driver's drowsiness or inattention and appropriate warning, as well as comprehensive judgment taking the driver's emotional state into consideration.

[0435] "Means for capturing the driver's face" refers to a device or sensor for acquiring an image of the driver's face, and generally refers to a camera device.

[0436] "Means for analyzing the captured facial image and evaluating the driver's eye opening / closing state and facial expression" refers to a device or algorithm that analyzes the driver's eye opening / closing state and changes in facial expression based on the acquired facial image.

[0437] "Means for recognizing the driver's emotional state using an emotion engine" refers to an algorithm or system for assessing the driver's emotional state (e.g., stress level, fatigue, irritability, etc.) based on analyzed facial information.

[0438] The "means for detecting signs of driver drowsiness or inattention" is a device or program for detecting signs of drowsiness or inattention by integrating the analysis results of the driver's eye opening / closing state and facial expressions with the evaluation of the emotion engine.

[0439] A "means for issuing a warning when signs of drowsiness or inattention are detected" is a device or system for issuing a warning to the driver via voice or text message based on detected signs of drowsiness or inattention.

[0440] "A means for storing all data and allowing administrators to monitor in real time" refers to a system that stores warning history and analysis results in a database and provides a dashboard that administrators can access and monitor in real time via the Internet.

[0441] The "means for transmitting data to a cloud server and issuing analysis and warning instructions" refers to a communication device and system for transmitting a driver's facial image and analysis data to a cloud server and issuing a warning instruction based on the analysis results of the cloud server.

[0442] This invention provides a system that combines an emotion engine to prevent drivers from falling asleep or being distracted while driving. This system is composed of the following elements:

[0443] Driver face capture methods:

[0444] A camera device is installed in the vehicle to periodically capture the driver's face, and the camera device is responsible for sending the captured facial images to a cloud server.

[0445] A means of analyzing captured facial images to assess the driver's eye status and facial expression:

[0446] The cloud server inputs the received facial image data into an image analysis algorithm (e.g., TensorFlow, OpenCV), which evaluates the eye status, facial color, and changes in facial expression to analyze the driver's attention level.

[0447] Using an emotion engine to recognize the driver's emotional state:

[0448] The cloud server inputs the data obtained from image analysis into an emotion engine (e.g., Microsoft Azure Emotion API) to recognize the driver's emotional state. The emotion engine evaluates the driver's stress level, fatigue, irritability, etc.

[0449] Ways to detect signs of driver drowsiness or inattention:

[0450] The cloud server combines the analysis results, such as when the eyes are closed for a certain period of time (e.g., more than 3 seconds) or when the face is expressionless, with the emotion engine's evaluation to comprehensively detect signs of drowsiness or inattention.

[0451] To alert you if signs of drowsiness or inattention are detected:

[0452] If the cloud server detects signs of drowsiness or inattention, it sends a warning instruction to the device, which then issues a voice or text message to the driver.

[0453] A way to store all data and allow administrators to monitor it in real time:

[0454] The cloud server stores all warning history and analysis results in a database and provides a dashboard that administrators can access via the internet, allowing them to monitor and manage the status of multiple drivers in real time.

[0455] Program processing:

[0456] The system periodically captures various data and sends it to a cloud server. The cloud server then runs the received data through an image analysis algorithm to evaluate eye status and facial expressions. It then uses an emotion engine to recognize the driver's emotional state and integrates the analysis results to detect signs of drowsiness or inattention. If signs are detected, the server sends a warning instruction to the device, which then issues a voice or text warning. Finally, all data is stored in a database, allowing administrators to monitor it in real time.

[0457] Examples:

[0458] Consider a case where a driver becomes tired while driving and closes their eyes for more than three seconds. At this time, a camera device captures the driver's face and sends the data to a cloud server. The server uses an image analysis algorithm to evaluate whether the eyes are open or closed, and an emotion engine to recognize the driver's fatigue state. The cloud server integrates these analysis results, detects signs of drowsiness, and sends a warning instruction to the device. The device issues an audio warning saying, "Caution: You may be drowsy while driving. Please take a break." This series of data is stored in a cloud database, allowing administrators to monitor it in real time.

[0459] Example prompt sentence:

[0460] "Images of the driver's face are captured every second and sent to a cloud server. The server uses image analysis algorithms and an emotion engine to evaluate eye status and facial expressions. If signs of drowsy driving or inattention are detected, the cloud server sends warning instructions to the device and warns via voice and text messages. All data is stored in a cloud database and can be monitored in real time by administrators."

[0461] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0462] Step 1:

[0463] The device is started in the vehicle and the user enters authentication information. The authentication information (e.g., facial recognition data or password) is sent from the device to the cloud server. The server verifies the authentication information and notifies the device that authentication was successful.

[0464] Input: User credentials

[0465] Output: Authentication success notification

[0466] Specific operation: When a user enters a vehicle, the camera device performs facial recognition. The authentication information is sent to the cloud server, and if the authentication is successful, the device is notified.

[0467] Step 2:

[0468] The device periodically (e.g., every second) captures the driver's facial image using a camera device, and the captured facial image is sent to a cloud server.

[0469] Input: Driver's face image

[0470] Output: Sending facial image data to the cloud server

[0471] Specific operation: The device's camera takes a picture of the driver's face every second and sends the image data to a cloud server.

[0472] Step 3:

[0473] The server then inputs the received facial images into an image analysis algorithm (e.g., TensorFlow, OpenCV), which processes the images to evaluate eye status and facial expressions.

[0474] Input: Facial image data

[0475] Output: Eye open / close status and facial expression analysis results

[0476] How it works: The cloud server uses image analysis algorithms to evaluate eye status and facial expressions.

[0477] Step 4:

[0478] The server inputs the data into an emotion engine (e.g., Microsoft Azure Emotion API) based on the image analysis results and evaluates the driver's emotional state (stress level, fatigue level, etc.).

[0479] Input: Image analysis results

[0480] Output: Emotional state evaluation result

[0481] Specific operation: The cloud server inputs the image analysis results into the emotion engine to evaluate the driver's stress level and fatigue state.

[0482] Step 5:

[0483] The server combines the analysis results, such as when the eyes are closed for a certain period of time (for example, more than three seconds) or when the face is expressionless, with the emotion engine's evaluation to detect signs of drowsiness or inattention.

[0484] Input: Eye status and emotional state evaluation results

[0485] Output: Detected signs of drowsiness and inattention

[0486] Specific operation: The cloud server integrates the evaluation results of eye opening / closing status and emotional state to detect signs of drowsiness or inattention.

[0487] Step 6:

[0488] If the server detects signs of drowsiness or inattention, it sends warning instructions to the device, which then issues voice and text message warnings based on the instructions.

[0489] Input: Result of detecting signs of drowsiness and inattention

[0490] Output: Warning instructions and warning messages

[0491] Specific operation: The cloud server detects signs of drowsiness and sends a warning instruction to the device. The device then issues a warning saying, "Caution: You may be drowsy at the wheel. Please take a break."

[0492] Step 7:

[0493] The server stores all warning history and analysis results in a database and provides an internet-accessible dashboard for administrators, allowing them to monitor and manage the status of multiple drivers in real time.

[0494] Input: Warning history and analysis results

[0495] Output: Save to database and reflect on dashboard

[0496] Specific operation: The cloud server stores warning events and analysis results in a database, and administrators can monitor them in real time through a dashboard.

[0497] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0498] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search<url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0499] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0500] [Second embodiment]

[0501] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0502] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0503] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0504] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0505] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0506] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0507] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0508] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0509] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0510] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0511] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0512] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0513] This invention is a system for preventing drivers in the transportation industry from falling asleep or being distracted while driving, and is composed of the following elements:

[0514] 1. A way to capture the driver's face

[0515] The terminal includes a camera device mounted on the vehicle. This camera device periodically captures the driver's face and sends the captured images to a server, thereby obtaining images of the driver's eye status and facial expressions.

[0516] 2. A means of analyzing captured facial images to assess the driver's eye status and facial expression.

[0517] The server then inputs the captured facial image data into an image analysis algorithm, which evaluates the driver's condition by assessing eye status, facial color, and changes in facial expression.

[0518] 3. Measures to detect signs of drowsiness and inattention

[0519] The server detects signs of drowsiness or inattention based on the results of image analysis, such as when the eyes remain closed for a certain period of time or when the facial expression becomes neutral. These criteria are built into the image analysis algorithm.

[0520] 4. A means of issuing warnings if signs of drowsiness or inattention are detected

[0521] When the server detects signs of drowsiness or inattention, it sends a warning instruction to the device. Based on this instruction, the device issues a warning to the driver by voice or text message. An example of a voice warning is, "Caution: You may be drowsy at the wheel. Please take a break."

[0522] 5. A means to accumulate all this data and allow administrators to monitor it in real time

[0523] The server stores all warning history and analysis results in a database and provides an administrator dashboard connected via the internet, which allows administrators to monitor and manage the status of multiple drivers in real time.

[0524] Explaining program processing in natural language

[0525] Step 1: Initial Setup and Authentication

[0526] The user gets into the vehicle, starts up the terminal (camera device), and authenticates himself / herself.

[0527] The device sends authentication information to the server to verify that it is the driver.

[0528] Step 2: Data Acquisition

[0529] The device periodically (for example, every second) captures an image of the driver's face and sends it to the server.

[0530] Step 3: Data analysis

[0531] The server inputs the received facial images into an analysis algorithm to evaluate whether the eyes are open or closed and changes in facial expression.

[0532] The server assesses the driver's level of attention and detects signs of drowsiness or inattention.

[0533] Step 4: Warning occurs

[0534] If the server detects signs of drowsiness or inattention, it sends a warning instruction to the terminal.

[0535] Step 5: Warning Notification

[0536] The device will warn the driver via voice and text messages based on the received warning instructions.

[0537] Step 6: Data storage and management

[0538] The server stores all warning history and analysis results in a database.

[0539] Managers can monitor and manage the status of multiple drivers in real time using an internet-based dashboard.

[0540] Specific examples

[0541] For example, consider a situation where a driver keeps their eyes closed for three seconds while driving.

[0542] 1. The user starts driving, and the device (camera device) captures facial images every second and sends them to the server.

[0543] 2. The server analyzes the captured facial images and determines that the eyes are closed in the first image.

[0544] 3. Next, the eyes are recognized as closed in the next two images (total of 3 seconds).

[0545] 4. The server detects signs of drowsiness because the eyes remain closed for more than 3 seconds.

[0546] 5. The server sends a warning instruction to the terminal, and the terminal issues a voice warning saying, "Caution: You may be drowsy while driving. Please take a break."

[0547] 6. The server stores this warning event in a database and makes it available for administrators to view in real time on a dashboard.

[0548] This series of steps enables the system to detect drowsy or careless driving by the driver in real time and issue a prompt warning, contributing to maintaining a safe driving environment.

[0549] The processing flow will be explained below.

[0550] Step 1:

[0551] The user gets into the vehicle and turns on the terminal (camera device).

[0552] The device displays a screen for entering the driver's authentication information.

[0553] Step 2:

[0554] The user enters their authentication information (e.g., ID and password) into the terminal.

[0555] The terminal transmits this authentication information to the server.

[0556] Step 3:

[0557] The server compares the received authentication information with a database and authenticates the driver.

[0558] If the authentication is successful, the server returns a message to the terminal indicating that the driver authentication was successful.

[0559] Step 4:

[0560] The device will indicate that driver authentication was successful and prepare to enter capture mode.

[0561] Step 5:

[0562] The terminal uses a camera device to capture an image of the driver's face at regular intervals (e.g., every second).

[0563] The terminal transmits the captured images to the server in real time.

[0564] Step 6:

[0565] The server then feeds the received facial images into an image analysis algorithm.

[0566] The server analyzes the eye status and facial expressions to assess the driver's level of attention.

[0567] Step 7:

[0568] The server detects signs of drowsiness if the driver's eyes remain closed for a certain period of time (for example, three seconds or more).

[0569] The server also applies if signs of inattention (e.g., a blank expression) are detected.

[0570] Step 8:

[0571] The server sends a warning instruction to the terminal when signs of drowsiness or inattention are detected.

[0572] The server records this warning event in a database.

[0573] Step 9:

[0574] When the terminal receives a warning instruction from the server, it issues a warning to the driver via a voice message (for example, "Caution: You may be drowsy while driving. Please take a break.") or a text message.

[0575] Step 10:

[0576] The server stores all warning history and analysis results in a database and provides a dashboard that allows administrators to monitor in real time via the Internet.

[0577] Step 11:

[0578] Managers can monitor and manage the status of multiple drivers in real time through a provided dashboard.

[0579] Example 1

[0580] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0581] Many accidents occur due to drivers in the transportation industry falling asleep or being distracted while driving. This not only results in personal injury and property damage, and compromises safety, but also leads to economic losses. Therefore, a system is needed that can monitor driver attention in real time, quickly detect signs of drowsiness or inattention, and issue appropriate warnings.

[0582] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0583] In this invention, the server includes means for capturing the driver's face, means for analyzing the captured facial image and evaluating the driver's eye opening / closing state and facial expression, means for detecting signs of drowsiness or inattention of the driver, means for issuing a warning when signs of drowsiness or inattention are detected, and means for storing all warning history and analysis results in a database that an administrator can monitor in real time via the Internet. This makes it possible to monitor the driver's attention in real time and prevent drowsy driving and inattentive driving.

[0584] "Driver" refers to a person who performs driving duties, especially a person who operates a vehicle in the transportation industry.

[0585] "Facial capture means" means a camera device or equivalent installed in a vehicle for capturing an image of the driver's face.

[0586] "Means for analyzing images" refers to image analysis algorithms and software for evaluating the open / closed state of the eyes and changes in facial expression based on acquired facial images.

[0587] "Means for detecting signs of drowsiness or inattention" refers to algorithms or software that determine whether the driver is drowsy or inattentive from analyzed image data.

[0588] "Warning means" refers to devices or software that alert the driver via voice or text message if signs of drowsiness or inattention are detected.

[0589] "Means for storing in a database" refers to a database system for storing all warning history and analysis results.

[0590] "Means that allow an administrator to monitor in real time via the Internet" refers to a dashboard or web application that allows an administrator to monitor and manage the status of multiple drivers in real time via the Internet.

[0591] "Camera device" means an optical device mounted on a vehicle for capturing an image of the driver's face.

[0592] This invention is a system for preventing drivers in the transportation industry from falling asleep or being distracted while driving. The system includes a means for capturing the driver's face, a means for analyzing the captured facial image, a means for detecting signs of drowsiness or inattention, a means for issuing a warning, and a means for allowing a manager to monitor in real time.

[0593] Hardware Configuration

[0594] The terminal includes a camera device mounted on the vehicle, which periodically captures images of the driver's face and transmits the captured images to a server.

[0595] The server is a combination of a high-performance computer and database system, equipped with specialized software for running image analysis algorithms.

[0596] The administrator dashboard is a web-based application that allows real-time data monitoring and management over the Internet.

[0597] Software Configuration

[0598] The server includes the following software modules:

[0599] 1. Image capture module: processes image data received from the terminal.

[0600] 2. Image analysis module: The received image data is fed into an algorithm to evaluate the eye opening / closing status and changes in facial expression.

[0601] 3. Drowsiness detection algorithm: Based on the results of image analysis, it detects signs of drowsiness or inattention.

[0602] 4. Warning generation module: Generates warning instructions when signs of drowsiness or inattention are detected.

[0603] 5. Data storage module: All warning history and analysis results are stored in a database.

[0604] 6. Dashboard module: Provides a real-time monitoring interface for administrators.

[0605] System Operation

[0606] The user enters the vehicle, activates the terminal (camera device), and authenticates themselves. The terminal sends the authentication information to the server, and the authentication is confirmed.

[0607] The device periodically (for example, every second) captures an image of the driver's face and sends it to the server.

[0608] The server then inputs the received facial images into an analysis algorithm that evaluates the eye status and changes in facial expression to determine the driver's attention level and detect signs of drowsiness or inattention.

[0609] If the server detects signs of drowsiness or inattention, it sends a warning instruction to the terminal.

[0610] Based on the warning instructions received, the device will issue a warning to the driver via voice and text message.

[0611] The server stores all warning history and analysis results in a database, allowing administrators to monitor and manage the status of multiple drivers in real time using a dashboard connected via the Internet.

[0612] Specific examples

[0613] For example, if a driver keeps their eyes closed for three seconds while driving, here's a specific example:

[0614] 1. The user starts driving, and the device (camera device) captures facial images every second and sends them to the server.

[0615] 2. The server analyzes the captured facial images and determines that the eyes are closed in the first image.

[0616] 3. Next, the eyes are recognized as closed in the next two images (total of 3 seconds).

[0617] 4. The server detects signs of drowsiness because the eyes remain closed for more than 3 seconds.

[0618] 5. The server sends a warning instruction to the terminal, and the terminal issues a voice warning saying, "Caution: You may be drowsy while driving. Please take a break."

[0619] 6. The server stores this warning event in a database and makes it available for administrators to view in real time on a dashboard.

[0620] Prompt Sentence Examples

[0621] "Please explain the procedures and give specific examples of systems to prevent drivers from falling asleep or being distracted while driving in the transportation industry."

[0622] The system is designed to provide a concrete form to support the technical scope of an invention and to help others accurately understand and practice the invention.

[0623] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0624] Step 1:

[0625] The user gets into the vehicle, turns on the terminal (camera device), and authenticates himself / herself.

[0626] Input: User authentication information (e.g., facial recognition data or ID input)

[0627] Output: The device sends the user's authentication information to the server.

[0628] Specific operation: When the user turns their face towards the device, the camera captures an image and obtains authentication information.

[0629] Step 2:

[0630] The terminal sends the received authentication information to the server.

[0631] Input: User credentials captured by the device

[0632] Output: Authentication information sent to the server.

[0633] Specific operation: The terminal's sending function automatically sends the authentication information to the server.

[0634] Step 3:

[0635] The server compares the received authentication information with a database and returns the authentication result to the terminal.

[0636] Input: User credentials received by the server

[0637] Output: The authentication result (success or failure) is returned to the terminal.

[0638] What happens: The server searches its database and finds a record that matches the authentication information.

[0639] Step 4:

[0640] The device captures an image of the driver's face every second and sends it to a server.

[0641] Input: A face image captured by the device

[0642] Output: The image is sent to the server.

[0643] Specific operation: The device's camera operates once per second and transmits the captured images via the network.

[0644] Step 5:

[0645] The facial images received by the server are fed into an image analysis algorithm to evaluate the open / closed state of the eyes and changes in facial expression.

[0646] Input: A face image received by the server

[0647] Output: Eye status and facial expression evaluation results

[0648] What happens: The server runs image analysis software and analyzes each frame.

[0649] Step 6:

[0650] Based on the analysis results, the server determines the driver's level of attention and detects signs of drowsiness or inattention.

[0651] Input: Eye status and facial expression evaluation results obtained from the server

[0652] Output: Presence or absence of signs of drowsiness or inattention

[0653] Specific operation: The server monitors the image analysis results over a certain period of time and detects abnormalities.

[0654] Step 7:

[0655] If the server detects signs of drowsiness or inattention, it generates a warning instruction and sends it to the terminal.

[0656] Input: Detected signs of drowsiness or inattention

[0657] Output: Warning instruction (e.g. "Caution: You may be drowsy while driving. Please take a break.")

[0658] Specific operation: The server generates a warning message and sends it to the terminal via the network.

[0659] Step 8:

[0660] Based on the warning instructions received by the device, the device will warn the driver via voice and text messages.

[0661] Input: Warning instruction from the server

[0662] Output: Voice and text message alerts

[0663] Specific operation: The device will display a warning using audio equipment or a display, and will notify you by voice.

[0664] Step 9:

[0665] The server stores all warning history and analysis results in a database.

[0666] Input: Analysis results and warning history

[0667] Output: Warning history and analysis results stored in a database

[0668] Specific operation: The server adds and updates the analysis results and warning information to the database.

[0669] Step 10:

[0670] The server displays all warning history and real-time driver status through an administrator dashboard.

[0671] Input: Stored alert history and real-time data

[0672] Output: Information displayed on the admin dashboard

[0673] What it does: The server processes the collected data in real time and displays it in a web interface.

[0674] This series of processes makes it possible to monitor the driver's attention level in real time, quickly detect signs of drowsiness or inattention, and prevent accidents before they occur.

[0675] (Application example 1)

[0676] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0677] In conventional vehicle operation, there are systems that prevent drivers from falling asleep or being distracted while driving, but these systems do not provide comprehensive safety measures such as driver authentication or response measures when a warning is issued, such as vehicle speed control. As a result, risks remain if the driver does not respond immediately after receiving a warning. The present invention aims to solve these problems by not only monitoring the driver's state in real time and detecting signs of inattention or drowsiness, but also providing appropriate warning measures and additional safety measures if no response is received.

[0678] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0679] In this invention, the server includes means for capturing the driver's face, means for analyzing the captured facial image and evaluating the driver's eye opening / closing state and facial expression, means for detecting signs of driver drowsiness or inattention, means for issuing a warning when signs of drowsiness or inattention are detected, means for accumulating all of this data and enabling an administrator to monitor it in real time, means for authenticating the driver, and means for automatically reducing the vehicle speed if the warning is not responded to. This makes it possible to comprehensively monitor the driver's condition and improve safety.

[0680] "Means for capturing the driver's face" refers to an apparatus and method for periodically capturing images of the driver's face using a camera device mounted on the vehicle.

[0681] "Means for analyzing captured facial images and evaluating the driver's eye opening / closing state and facial expression" refers to technology and algorithms that use captured facial images to analyze the driver's eye opening / closing state and changes in facial expression in real time.

[0682] "Means for detecting signs of driver drowsiness or inattention" refers to technology and algorithms that identify signs of drowsiness or inattention based on analyzed data on eye status and facial expressions, and trigger necessary actions.

[0683] "Means for issuing a warning when signs of drowsiness or inattention are detected" refers to a device and method that issues a warning to the driver via voice or text message when the driver shows signs of drowsiness or inattention.

[0684] "A means of accumulating all this data and allowing administrators to monitor it in real time" refers to a system that stores driver status data centrally on a server and allows administrators to monitor that data in real time via a dashboard over the Internet.

[0685] "Means for authenticating a driver" refers to technology and devices that authenticate the driver when he or she gets into a vehicle and transmit the authentication information to a server for confirmation.

[0686] "Means for automatically reducing vehicle speed when warnings are not responded to" refers to mechanisms and methods that automatically reduce vehicle speed to maintain a safe driving environment if the driver does not respond to a warning.

[0687] The present invention is an advanced monitoring system for preventing drivers from falling asleep at the wheel or from driving carelessly. Next, an embodiment of the present invention will be described in detail.

[0688] System Configuration

[0689] The system includes the following components:

[0690] 1. Means for capturing the driver's face: Images of the driver's face are periodically captured using a camera device mounted on the vehicle.

[0691] 2. A means for analyzing the captured facial image and evaluating the driver's eye opening / closing state and facial expression: Using an image analysis algorithm on the server, the driver's eye opening / closing state and facial expression are evaluated from the captured image.

[0692] 3. Means of driver authentication: When the driver gets into the vehicle, personal authentication is performed and the information is sent to the server.

[0693] 4. Means for detecting signs of driver drowsiness or inattention: An algorithm is used to detect signs of drowsiness or inattention, such as when the driver opens and closes their eyes repeatedly within a specified period of time.

[0694] 5. Means of issuing a warning when signs of drowsiness or inattention are detected: Based on the detection results, a warning instruction is sent to the device and a warning is issued by voice or text message.

[0695] 6. Means for automatically reducing vehicle speed when warning is not responded to: If a warning is not responded to, the speed of the vehicle is automatically reduced.

[0696] 7. A means of accumulating all this data and allowing administrators to monitor it in real time: All warning history and analysis results are stored in a database, and an administrator dashboard is provided that is connected via the Internet.

[0697] Hardware and Software Configuration

[0698] Hardware: camera devices, vehicle control units, network connectivity devices

[0699] Software: Facial recognition algorithm, eye open / close state analysis algorithm, facial expression analysis algorithm, database management system, warning generation system, vehicle speed control program, administrator dashboard

[0700] Data processing and calculation

[0701] The server receives image data sent from the camera device and analyzes facial recognition, eye status, and facial expressions using image processing libraries such as TensorFlow and OpenCV. The server comprehensively evaluates this data and detects signs of drowsiness or inattention.

[0702] If signs of drowsiness or inattention are detected, the server sends a warning instruction to the vehicle's warning system, which then alerts the occupants via smartphone or head-mounted display. If the warning is not responded to, the vehicle control unit automatically reduces the speed.

[0703] Specific examples

[0704] For example, consider a case where a driver keeps their eyes closed for three seconds while driving. The camera device captures an image of their face every second and sends it to the server. The server analyzes the image and recognizes that their eyes are closed. If the server determines that their eyes are still closed for the next two seconds, it issues a warning that the driver may be drowsy. If the driver does not respond to this warning, the vehicle's speed will be automatically reduced to ensure safety.

[0705] Prompt Sentence Examples

[0706] "Develop an algorithm that analyzes facial images captured periodically by the camera to assess eye status and facial expression. We hope to create a real-time surveillance application, including a system that issues an alert if it detects signs of drowsiness or inattention."

[0707] By implementing the invention in this manner, it is possible to prevent drivers from falling asleep or driving carelessly, thereby providing a safe driving environment.

[0708] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0709] Step 1:

[0710] The user gets into the vehicle, activates the terminal (camera device), and authenticates himself / herself.

[0711] Input: User authentication information (e.g., biometrics or ID card)

[0712] Data processing: The device sends the authentication information to the server and authenticates the driver.

[0713] Output: Authentication result (success or failure)

[0714] Specific operation: The device captures the user's face with a camera and checks whether it matches a familiar driver.

[0715] Step 2:

[0716] The device periodically (for example, every second) captures an image of the driver's face and sends it to the server.

[0717] Input: Image data obtained from the camera device (driver's face)

[0718] Data processing: Images are captured and sent to the server

[0719] Output: Latest facial image data stored on the server

[0720] Specific operation: The device's camera captures the driver's face every second and sends the image to the server.

[0721] Step 3:

[0722] The server inputs the received facial images into an analysis algorithm to evaluate whether the eyes are open or closed and changes in facial expression.

[0723] Input: Facial image data sent to the server

[0724] Data calculation: Uses face recognition and facial expression analysis algorithms to assess eye status and facial expressions

[0725] Output: Evaluation results of eye opening / closing status and facial expression

[0726] How it works: The server uses a facial recognition algorithm to detect whether the eyes are open or closed, and based on this, determines signs of drowsiness or inattention.

[0727] Step 4:

[0728] The server detects signs of drowsiness or inattention based on the eye status and facial expression evaluation results.

[0729] Input: Eye status and facial expression evaluation results

[0730] Data calculation: Algorithms that detect signs of drowsiness or inattention based on predetermined criteria

[0731] Output: Detected signs of drowsiness or inattention (detected / not detected)

[0732] Specific operation: The server analyzes data such as the time the eyes are closed and periods of expressionlessness to detect drowsiness or inattention.

[0733] Step 5:

[0734] If the server detects signs of drowsiness or inattention, it sends a warning instruction to the terminal.

[0735] Input: Detected signs of drowsiness or inattention

[0736] Data processing: Generate warning instructions and send them to the device

[0737] Output: Warning instructions (audio warning or text message)

[0738] Specific operation: The server generates a warning such as "Caution: You may be drowsy while driving. Please take a break," and sends it to the terminal.

[0739] Step 6:

[0740] Based on the warning instructions received, the device will issue a warning to the driver via voice and text message.

[0741] Input: Warning instruction sent from the server

[0742] Data processing: Sending alerts in the form of voice synthesis or text display

[0743] Output: Driver warning (voice or text message)

[0744] Specific operation: The device will emit a warning message such as "Caution" using a speaker, for example.

[0745] Step 7:

[0746] If the server does not respond to the warning, it will issue an instruction to automatically reduce the vehicle's speed.

[0747] Input: Response status after warning (with / without response)

[0748] Data calculation: Generation of speed control instructions based on a countermeasure algorithm in the event of no response

[0749] Output: Speed ​​reduction command sent to vehicle control unit

[0750] Specific operation: The server checks that there is no response for a certain period of time and then issues an instruction to reduce the vehicle's speed.

[0751] Step 8:

[0752] The server stores all alert history and analysis results in a database and provides a dashboard for administrators.

[0753] Input: Warning history, analysis results

[0754] Data processing: Data storage in a database and dashboard provision via the internet

[0755] Output: Real-time monitorable user interface

[0756] Specific operation: The server accumulates all data and displays it on a dashboard for administrators to monitor in real time.

[0757] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0758] In order to prevent drivers from falling asleep or being distracted while driving, the present invention provides a system that combines an emotion engine, which is composed of the following elements:

[0759] 1. A way to capture the driver's face

[0760] The terminal includes a camera device mounted on the vehicle, which periodically captures the driver's face and transmits the captured image to a server to obtain information such as the eye status and facial expression.

[0761] 2. A means of analyzing captured facial images to assess the driver's eye status and facial expression.

[0762] The server then inputs the captured facial image data into an image analysis algorithm, which evaluates the driver's attention level by evaluating factors such as eye status, facial color, and changes in facial expression.

[0763] 3. Emotion engine that recognizes the driver's emotions

[0764] The server inputs the data obtained from image analysis into an emotion engine to recognize the driver's emotional state, which evaluates, for example, the driver's stress level, fatigue, and irritability.

[0765] 4. Measures to detect signs of drowsiness and inattention

[0766] The server combines the results of both image analysis and the emotion engine to comprehensively detect signs of drowsiness or inattention, such as when the eyes are closed for a certain period of time or when emotional states are detected in combination with signs such as a blank expression.

[0767] 5. A means of issuing warnings if signs of drowsiness or inattention are detected

[0768] If the server detects signs of drowsiness or inattention, it sends a warning instruction to the device. Based on this instruction, the device issues a warning to the driver by voice or text message. Depending on the driver's condition, the warning content may be something like "Caution: You may be drowsy at the wheel. Please take a break" or "Caution: Stress is building. Please stop and take a break."

[0769] 6. A means to store all data and enable administrators to monitor it in real time

[0770] The server stores all warning history and analysis results in a database and provides a dashboard that administrators can access via the internet, allowing them to monitor and manage the status of multiple drivers in real time.

[0771] Explaining program processing in natural language

[0772] Step 1: Initial Setup and Authentication

[0773] The user gets into the vehicle and turns on the terminal (camera device).

[0774] The user enters authentication information, and the terminal transmits the information to the server.

[0775] The server verifies the authentication information and notifies the terminal that the authentication was successful.

[0776] Step 2: Data Acquisition

[0777] The device periodically (e.g., every second) captures an image of the driver's face and sends it to the server.

[0778] Step 3: Data analysis

[0779] The server inputs the received facial images into an image analysis algorithm to evaluate the open / closed state of the eyes, facial expressions, and changes in facial color.

[0780] Step 4: Emotion Recognition

[0781] The server inputs the image analysis results into an emotion engine to identify the driver's emotional state.

[0782] The emotion engine assesses stress levels, fatigue, irritability, etc.

[0783] Step 5: Hazard detection

[0784] The server combines the analysis results, such as when the eyes are closed for a certain period of time (for example, more than three seconds) or when the face is expressionless, with the emotion engine's evaluation to detect signs of drowsiness or inattention.

[0785] Step 6: Warning occurs

[0786] The server sends a warning instruction to the terminal when it detects signs of drowsiness or inattention.

[0787] The terminal receives the warning instruction and issues a warning by voice or text message.

[0788] Step 7: Data storage and management

[0789] The server stores all warning history and analysis results in a database and provides a dashboard for administrators to monitor in real time.

[0790] Specific examples

[0791] For example, consider the case where a driver keeps their eyes closed for three seconds while driving and their facial expression is neutral.

[0792] 1. The user gets into the vehicle and turns on the terminal (camera device).

[0793] 2. The user enters their authentication information and the device sends it to the server.

[0794] 3. The server verifies the authentication information and notifies the terminal that authentication was successful.

[0795] 4. The device captures a facial image every second and sends it to the server.

[0796] 5. The server uses image analysis algorithms to analyze the eye status and facial expressions.

[0797] 6. The server is an emotion engine that evaluates the driver's emotional state (e.g., fatigue, stress).

[0798] 7. The server detects closed eyes for more than 3 seconds and a neutral facial expression, and combines this with the emotion engine's evaluation to recognize signs of drowsiness.

[0799] 8. The server sends a warning instruction to the terminal and generates a message warning of the possibility of drowsiness.

[0800] 9. The device will issue a voice warning saying, "Caution: You may be falling asleep at the wheel. Please take a break."

[0801] 10. The server stores this warning event in a database and allows administrators to view the situation in real time on a dashboard.

[0802] This specific example makes it possible to provide a safe driving environment by detecting drowsy or inattentive driving by the driver at an early stage and issuing a prompt warning. Furthermore, by combining it with an emotion engine, it becomes possible to make comprehensive judgments that take into account the driver's stress and fatigue state.

[0803] The processing flow will be explained below.

[0804] Step 1:

[0805] The user gets into the vehicle and turns on the terminal (camera device).

[0806] The device displays a screen for entering the driver's authentication information.

[0807] Step 2:

[0808] The user enters their authentication information (e.g., ID and password) into the terminal.

[0809] The terminal transmits this authentication information to the server.

[0810] Step 3:

[0811] The server compares the received authentication information with a database and authenticates the driver.

[0812] If the authentication is successful, the server notifies the terminal that the driver authentication was successful.

[0813] Step 4:

[0814] The device will indicate that driver authentication was successful and prepare to enter capture mode.

[0815] Step 5:

[0816] The terminal uses a camera device to capture an image of the driver's face at regular intervals (e.g., every second).

[0817] The terminal transmits the captured images to the server in real time.

[0818] Step 6:

[0819] The server inputs the received facial images into an image analysis algorithm to analyze the eye status and facial expressions.

[0820] The server inputs the image analysis results into an emotion engine to identify the driver's emotional state.

[0821] Step 7:

[0822] The server combines the analysis results, such as when the eyes are closed for a certain period of time (for example, more than three seconds) or when the face is expressionless, with the emotion engine's evaluation to detect signs of drowsiness or inattention.

[0823] Step 8:

[0824] If the server detects signs of drowsiness or inattention, it sends a warning instruction to the terminal.

[0825] The server records this warning event in a database.

[0826] Step 9:

[0827] When the terminal receives a warning instruction from the server, it issues a warning to the driver via voice or text message.

[0828] For example, it may notify you with a message such as, "Caution: You may be drowsy while driving. Please take a break."

[0829] Step 10:

[0830] The server stores all warning history and analysis results in a database and provides a dashboard that allows administrators to monitor in real time.

[0831] Step 11:

[0832] Managers can monitor and manage the status of multiple drivers in real time through a provided dashboard.

[0833] Example 2

[0834] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0835] Currently, there are systems in place to prevent drivers from falling asleep or being distracted while driving, but most of these systems simply rely on eye movements and changes in facial expression. This makes it difficult to comprehensively detect hazards that take into account the driver's emotional state (stress level, fatigue, etc.). Furthermore, there is no system in place to monitor warning history and analysis results in real time. This has led to the problem of insufficient measures being taken to ensure driver safety.

[0836] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for capturing the driver's face, a means for analyzing the captured facial image and evaluating the driver's eye opening / closing state and facial expression, a means for recognizing the driver's emotional state, a means for detecting signs of the driver's drowsiness or inattention, a means for issuing a warning when signs of drowsiness or inattention are detected, and a means for accumulating warning history and analysis results and allowing an administrator to monitor them in real time. This enables comprehensive hazard detection that takes into account not only the driver's eye opening / closing state and changes in facial expression, but also their emotional state. Furthermore, accumulating warning history and analysis results and allowing an administrator to monitor them in real time enables a rapid response, further ensuring the safety of the driver.

[0837] The "means for capturing the driver's face" is a device for periodically acquiring an image of the driver's face using a camera device mounted on the vehicle.

[0838] "Means for analyzing captured facial images and evaluating the driver's eye opening / closing status and facial expression" refers to a device or program that inputs the acquired facial image data into an image analysis algorithm to analyze the eye opening / closing status, facial color, changes in facial expression, etc.

[0839] The "means for recognizing the driver's emotional state" is a device or program that includes an emotion engine for identifying the driver's emotional state, such as stress level, fatigue, or irritability, based on data obtained from image analysis.

[0840] The "means for detecting signs of driver drowsiness or inattention" refers to a device or program that uses an algorithm that integrates the results of image analysis with the evaluation of an emotion engine, and evaluates the eye opening / closing status, changes in facial expression, and emotional state within a certain period of time to determine signs of drowsiness or inattention.

[0841] "Means for issuing a warning when signs of drowsiness or inattention are detected" means a device or program that issues a warning to the driver via voice or text message when signs of drowsiness or inattention are detected.

[0842] "Means for storing warning histories and analysis results and allowing administrators to monitor them in real time" refers to a device or program that stores all warning histories and analysis results in a database and provides a dashboard that administrators can access via the Internet.

[0843] This invention is a safe driving support system that combines facial image analysis and emotion recognition to prevent drivers from falling asleep or being careless while driving. This system is composed of the following elements.

[0844] 1. How to capture the driver's face:

[0845] The terminal includes a camera device mounted on the vehicle, which periodically captures the driver's face and transmits the captured image to a server to obtain information such as the eye status and facial expression.

[0846] 2. Means of analyzing the captured facial image:

[0847] The server inputs the received facial image data into an image analysis algorithm, possibly using software such as OpenCV or TensorFlow. The server evaluates the eye status, facial color, and changes in facial expression to analyze the driver's attention level.

[0848] 3. How to recognize driver emotions:

[0849] The server inputs the data obtained from image analysis into an emotion engine to recognize the driver's emotional state. The emotion engine can use AI models such as Microsoft's Azure Emotion API or Google Cloud's Face API. This allows the system to evaluate the driver's stress level, fatigue, irritability, etc.

[0850] 4. Measures to detect signs of drowsiness or inattention:

[0851] The server integrates the image analysis results with the emotion engine's evaluation to detect signs of drowsiness or inattention. Specifically, it uses an algorithm to determine whether the driver is drowsy or inattentive based on conditions such as eyes closed for a certain period of time (e.g., three seconds or more) or a blank expression.

[0852] 5. Means of issuing a warning if signs of drowsiness or inattention are detected:

[0853] If the server detects signs of drowsiness or inattention, it sends a warning instruction to the device. Based on this instruction, the device issues a warning to the driver by voice or text message. The warning content might be something like, "Caution: You may be drowsy at the wheel. Please take a break."

[0854] 6. A way to store all data and allow administrators to monitor it in real time:

[0855] The server stores all warning history and analysis results in a database and provides a dashboard that administrators can access via the internet, allowing them to monitor and manage the status of multiple drivers in real time.

[0856] Specific examples

[0857] For example, consider the case where a driver keeps their eyes closed for three seconds while driving and their facial expression is neutral.

[0858] 1. The user gets into the vehicle and turns on the terminal (camera device).

[0859] 2. The user enters their authentication information and the device sends it to the server.

[0860] 3. The server verifies the authentication information and notifies the terminal that authentication was successful.

[0861] 4. The device captures a facial image every second and sends it to the server.

[0862] 5. The server uses image analysis algorithms to analyze the eye status and facial expressions.

[0863] 6. The server is an emotion engine that evaluates the driver's emotional state (e.g., fatigue, stress).

[0864] 7. The server detects closed eyes for more than 3 seconds and a neutral facial expression, and combines this with the emotion engine's evaluation to recognize signs of drowsiness.

[0865] 8. The server sends a warning instruction to the terminal and generates a message warning of the possibility of drowsiness.

[0866] 9. The device will issue a voice warning saying, "Caution: You may be falling asleep at the wheel. Please take a break."

[0867] 10. The server stores this warning event in a database and allows administrators to view the situation in real time on a dashboard.

[0868] Example prompts for generative AI models

[0869] A detailed description of the system can be automatically generated by using the following prompts to input data into the generative AI model:

[0870] Based on the following information, generate a detailed description of a system that prevents drivers from falling asleep or being distracted while driving:

[0871] The system consists of a camera device, a server, and an emotion engine.

[0872] 1. The vehicle-mounted camera periodically captures the driver's face and sends it to a server.

[0873] 2. The server analyzes the image and evaluates the eye status and facial expression.

[0874] 3. The emotion engine identifies the driver's emotional state based on the analysis results.

[0875] 4. The server detects signs of drowsiness or inattention, such as closed eyes for a period of time or a blank expression.

[0876] 5. The server sends a warning instruction to the device and warns the driver via voice or text message.

[0877] 6. All data is stored on the server and can be viewed on a dashboard where administrators can monitor in real time.

[0878] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0879] Step 1: Initial Setup and Authentication

[0880] The user gets into the vehicle and turns on the terminal (camera device). The terminal checks the connection between the camera device and the network. The user enters authentication information (e.g., ID and password) as input, which the terminal sends to the server.

[0881] The server checks the received authentication information against the database to see if the authentication was successful, and notifies the terminal that the authentication was successful.

[0882] Step 2: Data Acquisition

[0883] After successful authentication, the device periodically (e.g., every second) captures an image of the driver's face using a camera device, and the captured face image data is generated as input.

[0884] The terminal compresses the captured image data and sends it to the server. The output compressed data is sent and reaches the server.

[0885] Step 3: Image analysis

[0886] The server feeds the received facial image data into an image analysis algorithm, which receives the compressed facial image data as input.

[0887] The server uses software such as OpenCV and TensorFlow to analyze eye status, facial color, and facial expression changes, and generates data on eye status and facial expressions as output.

[0888] Step 4: Emotion Recognition

[0889] The server inputs the data obtained from the image analysis into the emotion engine, providing data such as eye opening and closing status and facial expressions.

[0890] The emotion engine uses AI models (e.g., Azure Emotion API, Google Cloud Face API) to identify the driver's emotional state (e.g., stress level, fatigue, irritability). As an output, data about the driver's emotional state is generated.

[0891] Step 5: Hazard detection

[0892] The server integrates data on eye opening and closing and facial expressions with the emotion engine's evaluation. Analysis data and emotional state data are integrated as input.

[0893] The server runs an algorithm to detect signs of drowsiness or inattention. Specifically, it evaluates whether the eyes are closed for a certain period of time (e.g., more than 3 seconds) or whether the face is neutral. As an output, it generates warning data about signs of drowsiness or inattention.

[0894] Step 6: Warning occurs

[0895] The server sends a warning instruction to the terminal when it detects signs of drowsiness or inattention. As input, a warning instruction is generated.

[0896] The device receives the warning instruction and issues a warning via voice or text message. Specifically, it uses a generative AI model to generate a prompt sentence: "Caution: You may be drowsy at the wheel. Please take a break." As an output, the warning message is provided to the driver.

[0897] Step 7: Data storage and management

[0898] The server stores all the warning history and analysis results in a database. As input, the warning data and analysis data are stored in the database.

[0899] The server provides a dashboard that can be accessed by the administrator. Specifically, it builds an interface that allows the administrator to monitor the driver status in real time. As an output, a dashboard that can be accessed by the administrator is created.

[0900] (Application example 2)

[0901] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0902] Improving safety is a key issue for modern self-driving vehicles. Driver drowsiness and inattention can lead to serious accidents, so technology is needed to detect these early and provide appropriate warnings. Furthermore, there is a lack of systems that can make comprehensive judgments by taking into account the driver's emotional state, and this needs to be addressed.

[0903] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing the driver's face, means for analyzing the captured facial image and evaluating the driver's eye opening / closing state and facial expression, means for recognizing the driver's emotional state using an emotion engine, means for detecting signs of the driver's drowsiness or inattention, means for issuing a warning when signs of drowsiness or inattention are detected, means for accumulating all data so that an administrator can monitor it in real time, and means for transmitting data to a cloud server for analysis and issuing a warning instruction. This enables early detection of the driver's drowsiness or inattention and appropriate warning, as well as comprehensive judgment taking the driver's emotional state into consideration.

[0904] "Means for capturing the driver's face" refers to a device or sensor for acquiring an image of the driver's face, and generally refers to a camera device.

[0905] "Means for analyzing the captured facial image and evaluating the driver's eye opening / closing state and facial expression" refers to a device or algorithm that analyzes the driver's eye opening / closing state and changes in facial expression based on the acquired facial image.

[0906] "Means for recognizing the driver's emotional state using an emotion engine" refers to an algorithm or system for assessing the driver's emotional state (e.g., stress level, fatigue, irritability, etc.) based on analyzed facial information.

[0907] The "means for detecting signs of driver drowsiness or inattention" is a device or program for detecting signs of drowsiness or inattention by integrating the analysis results of the driver's eye opening / closing state and facial expressions with the evaluation of the emotion engine.

[0908] A "means for issuing a warning when signs of drowsiness or inattention are detected" is a device or system for issuing a warning to the driver via voice or text message based on detected signs of drowsiness or inattention.

[0909] "A means for storing all data and allowing administrators to monitor in real time" refers to a system that stores warning history and analysis results in a database and provides a dashboard that administrators can access and monitor in real time via the Internet.

[0910] The "means for transmitting data to a cloud server and issuing analysis and warning instructions" refers to a communication device and system for transmitting a driver's facial image and analysis data to a cloud server and issuing a warning instruction based on the analysis results of the cloud server.

[0911] This invention provides a system that combines an emotion engine to prevent drivers from falling asleep or being distracted while driving. This system is composed of the following elements:

[0912] Driver face capture methods:

[0913] A camera device is installed in the vehicle to periodically capture the driver's face, and the camera device is responsible for sending the captured facial images to a cloud server.

[0914] A means of analyzing captured facial images to assess the driver's eye status and facial expression:

[0915] The cloud server inputs the received facial image data into an image analysis algorithm (e.g., TensorFlow, OpenCV), which evaluates the eye status, facial color, and changes in facial expression to analyze the driver's attention level.

[0916] Using an emotion engine to recognize the driver's emotional state:

[0917] The cloud server inputs the data obtained from image analysis into an emotion engine (e.g., Microsoft Azure Emotion API) to recognize the driver's emotional state. The emotion engine evaluates the driver's stress level, fatigue, irritability, etc.

[0918] Ways to detect signs of driver drowsiness or inattention:

[0919] The cloud server combines the analysis results, such as when the eyes are closed for a certain period of time (e.g., more than 3 seconds) or when the face is expressionless, with the emotion engine's evaluation to comprehensively detect signs of drowsiness or inattention.

[0920] To alert you if signs of drowsiness or inattention are detected:

[0921] If the cloud server detects signs of drowsiness or inattention, it sends a warning instruction to the device, which then issues a voice or text message to the driver.

[0922] A way to store all data and allow administrators to monitor it in real time:

[0923] The cloud server stores all warning history and analysis results in a database and provides a dashboard that administrators can access via the internet, allowing them to monitor and manage the status of multiple drivers in real time.

[0924] Program processing:

[0925] The system periodically captures various data and sends it to a cloud server. The cloud server then runs the received data through an image analysis algorithm to evaluate eye status and facial expressions. It then uses an emotion engine to recognize the driver's emotional state and integrates the analysis results to detect signs of drowsiness or inattention. If signs are detected, the server sends a warning instruction to the device, which then issues a voice or text warning. Finally, all data is stored in a database, allowing administrators to monitor it in real time.

[0926] Examples:

[0927] Consider a case where a driver becomes tired while driving and closes their eyes for more than three seconds. At this time, a camera device captures the driver's face and sends the data to a cloud server. The server uses an image analysis algorithm to evaluate whether the eyes are open or closed, and an emotion engine to recognize the driver's fatigue state. The cloud server integrates these analysis results, detects signs of drowsiness, and sends a warning instruction to the device. The device issues an audio warning saying, "Caution: You may be drowsy while driving. Please take a break." This series of data is stored in a cloud database, allowing administrators to monitor it in real time.

[0928] Example prompt sentence:

[0929] "Images of the driver's face are captured every second and sent to a cloud server. The server uses image analysis algorithms and an emotion engine to evaluate eye status and facial expressions. If signs of drowsy driving or inattention are detected, the cloud server sends warning instructions to the device and warns via voice and text messages. All data is stored in a cloud database and can be monitored in real time by administrators."

[0930] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0931] Step 1:

[0932] The device is started in the vehicle and the user enters authentication information. The authentication information (e.g., facial recognition data or password) is sent from the device to the cloud server. The server verifies the authentication information and notifies the device that authentication was successful.

[0933] Input: User credentials

[0934] Output: Authentication success notification

[0935] Specific operation: When a user enters a vehicle, the camera device performs facial recognition. The authentication information is sent to the cloud server, and if the authentication is successful, the device is notified.

[0936] Step 2:

[0937] The device periodically (e.g., every second) captures the driver's facial image using a camera device, and the captured facial image is sent to a cloud server.

[0938] Input: Driver's face image

[0939] Output: Sending facial image data to the cloud server

[0940] Specific operation: The device's camera takes a picture of the driver's face every second and sends the image data to a cloud server.

[0941] Step 3:

[0942] The server then inputs the received facial images into an image analysis algorithm (e.g., TensorFlow, OpenCV), which processes the images to evaluate eye status and facial expressions.

[0943] Input: Facial image data

[0944] Output: Eye open / close status and facial expression analysis results

[0945] How it works: The cloud server uses image analysis algorithms to evaluate eye status and facial expressions.

[0946] Step 4:

[0947] The server inputs the data into an emotion engine (e.g., Microsoft Azure Emotion API) based on the image analysis results and evaluates the driver's emotional state (stress level, fatigue level, etc.).

[0948] Input: Image analysis results

[0949] Output: Emotional state evaluation result

[0950] Specific operation: The cloud server inputs the image analysis results into the emotion engine to evaluate the driver's stress level and fatigue state.

[0951] Step 5:

[0952] The server combines the analysis results, such as when the eyes are closed for a certain period of time (for example, more than three seconds) or when the face is expressionless, with the emotion engine's evaluation to detect signs of drowsiness or inattention.

[0953] Input: Eye status and emotional state evaluation results

[0954] Output: Detected signs of drowsiness and inattention

[0955] Specific operation: The cloud server integrates the evaluation results of eye opening / closing status and emotional state to detect signs of drowsiness or inattention.

[0956] Step 6:

[0957] If the server detects signs of drowsiness or inattention, it sends warning instructions to the device, which then issues voice and text message warnings based on the instructions.

[0958] Input: Result of detecting signs of drowsiness and inattention

[0959] Output: Warning instructions and warning messages

[0960] Specific operation: The cloud server detects signs of drowsiness and sends a warning instruction to the device. The device then issues a warning saying, "Caution: You may be drowsy at the wheel. Please take a break."

[0961] Step 7:

[0962] The server stores all warning history and analysis results in a database and provides an internet-accessible dashboard for administrators, allowing them to monitor and manage the status of multiple drivers in real time.

[0963] Input: Warning history and analysis results

[0964] Output: Save to database and reflect on dashboard

[0965] Specific operation: The cloud server stores warning events and analysis results in a database, and administrators can monitor them in real time through a dashboard.

[0966] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0967] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0968] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0969] [Third embodiment]

[0970] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0971] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0972] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0973] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0974] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0975] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0976] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0977] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0978] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0979] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0980] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0981] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0982] This invention is a system for preventing drivers in the transportation industry from falling asleep or being distracted while driving, and is composed of the following elements:

[0983] 1. A way to capture the driver's face

[0984] The terminal includes a camera device mounted on the vehicle. This camera device periodically captures the driver's face and sends the captured images to a server, thereby obtaining images of the driver's eye status and facial expressions.

[0985] 2. A means of analyzing captured facial images to assess the driver's eye status and facial expression.

[0986] The server then inputs the captured facial image data into an image analysis algorithm, which evaluates the driver's condition by assessing eye status, facial color, and changes in facial expression.

[0987] 3. Measures to detect signs of drowsiness and inattention

[0988] The server detects signs of drowsiness or inattention based on the results of image analysis, such as when the eyes remain closed for a certain period of time or when the facial expression becomes neutral. These criteria are built into the image analysis algorithm.

[0989] 4. A means of issuing warnings if signs of drowsiness or inattention are detected

[0990] When the server detects signs of drowsiness or inattention, it sends a warning instruction to the device. Based on this instruction, the device issues a warning to the driver by voice or text message. An example of a voice warning is, "Caution: You may be drowsy at the wheel. Please take a break."

[0991] 5. A means to accumulate all this data and allow administrators to monitor it in real time

[0992] The server stores all warning history and analysis results in a database and provides an administrator dashboard connected via the internet, which allows administrators to monitor and manage the status of multiple drivers in real time.

[0993] Explaining program processing in natural language

[0994] Step 1: Initial Setup and Authentication

[0995] The user gets into the vehicle, starts up the terminal (camera device), and authenticates himself / herself.

[0996] The device sends authentication information to the server to verify that it is the driver.

[0997] Step 2: Data Acquisition

[0998] The device periodically (for example, every second) captures an image of the driver's face and sends it to the server.

[0999] Step 3: Data analysis

[1000] The server inputs the received facial images into an analysis algorithm to evaluate whether the eyes are open or closed and changes in facial expression.

[1001] The server assesses the driver's level of attention and detects signs of drowsiness or inattention.

[1002] Step 4: Warning occurs

[1003] If the server detects signs of drowsiness or inattention, it sends a warning instruction to the terminal.

[1004] Step 5: Warning Notification

[1005] The device will warn the driver via voice and text messages based on the received warning instructions.

[1006] Step 6: Data storage and management

[1007] The server stores all warning history and analysis results in a database.

[1008] Managers can monitor and manage the status of multiple drivers in real time using an internet-based dashboard.

[1009] Specific examples

[1010] For example, consider a situation where a driver keeps their eyes closed for three seconds while driving.

[1011] 1. The user starts driving, and the device (camera device) captures facial images every second and sends them to the server.

[1012] 2. The server analyzes the captured facial images and determines that the eyes are closed in the first image.

[1013] 3. Next, the eyes are recognized as closed in the next two images (total of 3 seconds).

[1014] 4. The server detects signs of drowsiness because the eyes remain closed for more than 3 seconds.

[1015] 5. The server sends a warning instruction to the terminal, and the terminal issues a voice warning saying, "Caution: You may be drowsy while driving. Please take a break."

[1016] 6. The server stores this warning event in a database and makes it available for administrators to view in real time on a dashboard.

[1017] This series of steps enables the system to detect drowsy or careless driving by the driver in real time and issue a prompt warning, contributing to maintaining a safe driving environment.

[1018] The processing flow will be explained below.

[1019] Step 1:

[1020] The user gets into the vehicle and turns on the terminal (camera device).

[1021] The device displays a screen for entering the driver's authentication information.

[1022] Step 2:

[1023] The user enters their authentication information (e.g., ID and password) into the terminal.

[1024] The terminal transmits this authentication information to the server.

[1025] Step 3:

[1026] The server compares the received authentication information with a database and authenticates the driver.

[1027] If the authentication is successful, the server returns a message to the terminal indicating that the driver authentication was successful.

[1028] Step 4:

[1029] The device will indicate that driver authentication was successful and prepare to enter capture mode.

[1030] Step 5:

[1031] The terminal uses a camera device to capture an image of the driver's face at regular intervals (e.g., every second).

[1032] The terminal transmits the captured images to the server in real time.

[1033] Step 6:

[1034] The server then feeds the received facial images into an image analysis algorithm.

[1035] The server analyzes the eye status and facial expressions to assess the driver's level of attention.

[1036] Step 7:

[1037] The server detects signs of drowsiness if the driver's eyes remain closed for a certain period of time (for example, three seconds or more).

[1038] The server also applies if signs of inattention (e.g., a blank expression) are detected.

[1039] Step 8:

[1040] The server sends a warning instruction to the terminal when signs of drowsiness or inattention are detected.

[1041] The server records this warning event in a database.

[1042] Step 9:

[1043] When the terminal receives a warning instruction from the server, it issues a warning to the driver via a voice message (for example, "Caution: You may be drowsy while driving. Please take a break.") or a text message.

[1044] Step 10:

[1045] The server stores all warning history and analysis results in a database and provides a dashboard that allows administrators to monitor in real time via the Internet.

[1046] Step 11:

[1047] Managers can monitor and manage the status of multiple drivers in real time through a provided dashboard.

[1048] Example 1

[1049] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1050] Many accidents occur due to drivers in the transportation industry falling asleep or being distracted while driving. This not only results in personal injury and property damage, and compromises safety, but also leads to economic losses. Therefore, a system is needed that can monitor driver attention in real time, quickly detect signs of drowsiness or inattention, and issue appropriate warnings.

[1051] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1052] In this invention, the server includes means for capturing the driver's face, means for analyzing the captured facial image and evaluating the driver's eye opening / closing state and facial expression, means for detecting signs of drowsiness or inattention of the driver, means for issuing a warning when signs of drowsiness or inattention are detected, and means for storing all warning history and analysis results in a database that an administrator can monitor in real time via the Internet. This makes it possible to monitor the driver's attention in real time and prevent drowsy driving and inattentive driving.

[1053] "Driver" refers to a person who performs driving duties, especially a person who operates a vehicle in the transportation industry.

[1054] "Facial capture means" means a camera device or equivalent installed in a vehicle for capturing an image of the driver's face.

[1055] "Means for analyzing images" refers to image analysis algorithms and software for evaluating the open / closed state of the eyes and changes in facial expression based on acquired facial images.

[1056] "Means for detecting signs of drowsiness or inattention" refers to algorithms or software that determine whether the driver is drowsy or inattentive from analyzed image data.

[1057] "Warning means" refers to devices or software that alert the driver via voice or text message if signs of drowsiness or inattention are detected.

[1058] "Means for storing in a database" refers to a database system for storing all warning history and analysis results.

[1059] "Means that allow an administrator to monitor in real time via the Internet" refers to a dashboard or web application that allows an administrator to monitor and manage the status of multiple drivers in real time via the Internet.

[1060] "Camera device" means an optical device mounted on a vehicle for capturing an image of the driver's face.

[1061] This invention is a system for preventing drivers in the transportation industry from falling asleep or being distracted while driving. The system includes a means for capturing the driver's face, a means for analyzing the captured facial image, a means for detecting signs of drowsiness or inattention, a means for issuing a warning, and a means for allowing a manager to monitor in real time.

[1062] Hardware Configuration

[1063] The terminal includes a camera device mounted on the vehicle, which periodically captures images of the driver's face and transmits the captured images to a server.

[1064] The server is a combination of a high-performance computer and database system, equipped with specialized software for running image analysis algorithms.

[1065] The administrator dashboard is a web-based application that allows real-time data monitoring and management over the Internet.

[1066] Software Configuration

[1067] The server includes the following software modules:

[1068] 1. Image capture module: processes image data received from the terminal.

[1069] 2. Image analysis module: The received image data is fed into an algorithm to evaluate the eye opening / closing status and changes in facial expression.

[1070] 3. Drowsiness detection algorithm: Based on the results of image analysis, it detects signs of drowsiness or inattention.

[1071] 4. Warning generation module: Generates warning instructions when signs of drowsiness or inattention are detected.

[1072] 5. Data storage module: All warning history and analysis results are stored in a database.

[1073] 6. Dashboard module: Provides a real-time monitoring interface for administrators.

[1074] System Operation

[1075] The user enters the vehicle, activates the terminal (camera device), and authenticates themselves. The terminal sends the authentication information to the server, and the authentication is confirmed.

[1076] The device periodically (for example, every second) captures an image of the driver's face and sends it to the server.

[1077] The server then inputs the received facial images into an analysis algorithm that evaluates the eye status and changes in facial expression to determine the driver's attention level and detect signs of drowsiness or inattention.

[1078] If the server detects signs of drowsiness or inattention, it sends a warning instruction to the terminal.

[1079] Based on the warning instructions received, the device will issue a warning to the driver via voice and text message.

[1080] The server stores all warning history and analysis results in a database, allowing administrators to monitor and manage the status of multiple drivers in real time using a dashboard connected via the Internet.

[1081] Specific examples

[1082] For example, if a driver keeps their eyes closed for three seconds while driving, here's a specific example:

[1083] 1. The user starts driving, and the device (camera device) captures facial images every second and sends them to the server.

[1084] 2. The server analyzes the captured facial images and determines that the eyes are closed in the first image.

[1085] 3. Next, the eyes are recognized as closed in the next two images (total of 3 seconds).

[1086] 4. The server detects signs of drowsiness because the eyes remain closed for more than 3 seconds.

[1087] 5. The server sends a warning instruction to the terminal, and the terminal issues a voice warning saying, "Caution: You may be drowsy while driving. Please take a break."

[1088] 6. The server stores this warning event in a database and makes it available for administrators to view in real time on a dashboard.

[1089] Prompt Sentence Examples

[1090] "Please explain the procedures and give specific examples of systems to prevent drivers from falling asleep or being distracted while driving in the transportation industry."

[1091] The system is designed to provide a concrete form to support the technical scope of an invention and to help others accurately understand and practice the invention.

[1092] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1093] Step 1:

[1094] The user gets into the vehicle, turns on the terminal (camera device), and authenticates himself / herself.

[1095] Input: User authentication information (e.g., facial recognition data or ID input)

[1096] Output: The device sends the user's authentication information to the server.

[1097] Specific operation: When the user turns their face towards the device, the camera captures an image and obtains authentication information.

[1098] Step 2:

[1099] The terminal sends the received authentication information to the server.

[1100] Input: User credentials captured by the device

[1101] Output: Authentication information sent to the server.

[1102] Specific operation: The terminal's sending function automatically sends the authentication information to the server.

[1103] Step 3:

[1104] The server compares the received authentication information with a database and returns the authentication result to the terminal.

[1105] Input: User credentials received by the server

[1106] Output: The authentication result (success or failure) is returned to the terminal.

[1107] What happens: The server searches its database and finds a record that matches the authentication information.

[1108] Step 4:

[1109] The device captures an image of the driver's face every second and sends it to a server.

[1110] Input: A face image captured by the device

[1111] Output: The image is sent to the server.

[1112] Specific operation: The device's camera operates once per second and transmits the captured images via the network.

[1113] Step 5:

[1114] The facial images received by the server are fed into an image analysis algorithm to evaluate the open / closed state of the eyes and changes in facial expression.

[1115] Input: A face image received by the server

[1116] Output: Eye status and facial expression evaluation results

[1117] What happens: The server runs image analysis software and analyzes each frame.

[1118] Step 6:

[1119] Based on the analysis results, the server determines the driver's level of attention and detects signs of drowsiness or inattention.

[1120] Input: Eye status and facial expression evaluation results obtained from the server

[1121] Output: Presence or absence of signs of drowsiness or inattention

[1122] Specific operation: The server monitors the image analysis results over a certain period of time and detects abnormalities.

[1123] Step 7:

[1124] If the server detects signs of drowsiness or inattention, it generates a warning instruction and sends it to the terminal.

[1125] Input: Detected signs of drowsiness or inattention

[1126] Output: Warning instruction (e.g. "Caution: You may be drowsy while driving. Please take a break.")

[1127] Specific operation: The server generates a warning message and sends it to the terminal via the network.

[1128] Step 8:

[1129] Based on the warning instructions received by the device, the device will warn the driver via voice and text messages.

[1130] Input: Warning instruction from the server

[1131] Output: Voice and text message alerts

[1132] Specific operation: The device will display a warning using audio equipment or a display, and will notify you by voice.

[1133] Step 9:

[1134] The server stores all warning history and analysis results in a database.

[1135] Input: Analysis results and warning history

[1136] Output: Warning history and analysis results stored in a database

[1137] Specific operation: The server adds and updates the analysis results and warning information to the database.

[1138] Step 10:

[1139] The server displays all warning history and real-time driver status through an administrator dashboard.

[1140] Input: Stored alert history and real-time data

[1141] Output: Information displayed on the admin dashboard

[1142] What it does: The server processes the collected data in real time and displays it in a web interface.

[1143] This series of processes makes it possible to monitor the driver's attention level in real time, quickly detect signs of drowsiness or inattention, and prevent accidents before they occur.

[1144] (Application example 1)

[1145] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1146] In conventional vehicle operation, there are systems that prevent drivers from falling asleep or being distracted while driving, but these systems do not provide comprehensive safety measures such as driver authentication or response measures when a warning is issued, such as vehicle speed control. As a result, risks remain if the driver does not respond immediately after receiving a warning. The present invention aims to solve these problems by not only monitoring the driver's state in real time and detecting signs of inattention or drowsiness, but also providing appropriate warning measures and additional safety measures if no response is received.

[1147] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1148] In this invention, the server includes means for capturing the driver's face, means for analyzing the captured facial image and evaluating the driver's eye opening / closing state and facial expression, means for detecting signs of driver drowsiness or inattention, means for issuing a warning when signs of drowsiness or inattention are detected, means for accumulating all of this data and enabling an administrator to monitor it in real time, means for authenticating the driver, and means for automatically reducing the vehicle speed if the warning is not responded to. This makes it possible to comprehensively monitor the driver's condition and improve safety.

[1149] "Means for capturing the driver's face" refers to an apparatus and method for periodically capturing images of the driver's face using a camera device mounted on the vehicle.

[1150] "Means for analyzing captured facial images and evaluating the driver's eye opening / closing state and facial expression" refers to technology and algorithms that use captured facial images to analyze the driver's eye opening / closing state and changes in facial expression in real time.

[1151] "Means for detecting signs of driver drowsiness or inattention" refers to technology and algorithms that identify signs of drowsiness or inattention based on analyzed data on eye status and facial expressions, and trigger necessary actions.

[1152] "Means for issuing a warning when signs of drowsiness or inattention are detected" refers to a device and method that issues a warning to the driver via voice or text message when the driver shows signs of drowsiness or inattention.

[1153] "A means of accumulating all this data and allowing administrators to monitor it in real time" refers to a system that stores driver status data centrally on a server and allows administrators to monitor that data in real time via a dashboard over the Internet.

[1154] "Means for authenticating a driver" refers to technology and devices that authenticate the driver when he or she gets into a vehicle and transmit the authentication information to a server for confirmation.

[1155] "Means for automatically reducing vehicle speed when warnings are not responded to" refers to mechanisms and methods that automatically reduce vehicle speed to maintain a safe driving environment if the driver does not respond to a warning.

[1156] The present invention is an advanced monitoring system for preventing drivers from falling asleep at the wheel or from driving carelessly. Next, an embodiment of the present invention will be described in detail.

[1157] System Configuration

[1158] The system includes the following components:

[1159] 1. Means for capturing the driver's face: Images of the driver's face are periodically captured using a camera device mounted on the vehicle.

[1160] 2. A means for analyzing the captured facial image and evaluating the driver's eye opening / closing state and facial expression: Using an image analysis algorithm on the server, the driver's eye opening / closing state and facial expression are evaluated from the captured image.

[1161] 3. Means of driver authentication: When the driver gets into the vehicle, personal authentication is performed and the information is sent to the server.

[1162] 4. Means for detecting signs of driver drowsiness or inattention: An algorithm is used to detect signs of drowsiness or inattention, such as when the driver opens and closes their eyes repeatedly within a specified period of time.

[1163] 5. Means of issuing a warning when signs of drowsiness or inattention are detected: Based on the detection results, a warning instruction is sent to the device and a warning is issued by voice or text message.

[1164] 6. Means for automatically reducing vehicle speed when warning is not responded to: If a warning is not responded to, the speed of the vehicle is automatically reduced.

[1165] 7. A means of accumulating all this data and allowing administrators to monitor it in real time: All warning history and analysis results are stored in a database, and an administrator dashboard is provided that is connected via the Internet.

[1166] Hardware and Software Configuration

[1167] Hardware: camera devices, vehicle control units, network connectivity devices

[1168] Software: Facial recognition algorithm, eye open / close state analysis algorithm, facial expression analysis algorithm, database management system, warning generation system, vehicle speed control program, administrator dashboard

[1169] Data processing and calculation

[1170] The server receives image data sent from the camera device and analyzes facial recognition, eye status, and facial expressions using image processing libraries such as TensorFlow and OpenCV. The server comprehensively evaluates this data and detects signs of drowsiness or inattention.

[1171] If signs of drowsiness or inattention are detected, the server sends a warning instruction to the vehicle's warning system, which then alerts the occupants via smartphone or head-mounted display. If the warning is not responded to, the vehicle control unit automatically reduces the speed.

[1172] Specific examples

[1173] For example, consider a case where a driver keeps their eyes closed for three seconds while driving. The camera device captures an image of their face every second and sends it to the server. The server analyzes the image and recognizes that their eyes are closed. If the server determines that their eyes are still closed for the next two seconds, it issues a warning that the driver may be drowsy. If the driver does not respond to this warning, the vehicle's speed will be automatically reduced to ensure safety.

[1174] Prompt Sentence Examples

[1175] "Develop an algorithm that analyzes facial images captured periodically by the camera to assess eye status and facial expression. We hope to create a real-time surveillance application, including a system that issues an alert if it detects signs of drowsiness or inattention."

[1176] By implementing the invention in this manner, it is possible to prevent drivers from falling asleep or driving carelessly, thereby providing a safe driving environment.

[1177] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1178] Step 1:

[1179] The user gets into the vehicle, activates the terminal (camera device), and authenticates himself / herself.

[1180] Input: User authentication information (e.g., biometrics or ID card)

[1181] Data processing: The device sends the authentication information to the server and authenticates the driver.

[1182] Output: Authentication result (success or failure)

[1183] Specific operation: The device captures the user's face with a camera and checks whether it matches a familiar driver.

[1184] Step 2:

[1185] The device periodically (for example, every second) captures an image of the driver's face and sends it to the server.

[1186] Input: Image data obtained from the camera device (driver's face)

[1187] Data processing: Images are captured and sent to the server

[1188] Output: Latest facial image data stored on the server

[1189] Specific operation: The device's camera captures the driver's face every second and sends the image to the server.

[1190] Step 3:

[1191] The server inputs the received facial images into an analysis algorithm to evaluate whether the eyes are open or closed and changes in facial expression.

[1192] Input: Facial image data sent to the server

[1193] Data calculation: Uses face recognition and facial expression analysis algorithms to assess eye status and facial expressions

[1194] Output: Evaluation results of eye opening / closing status and facial expression

[1195] How it works: The server uses a facial recognition algorithm to detect whether the eyes are open or closed, and based on this, determines signs of drowsiness or inattention.

[1196] Step 4:

[1197] The server detects signs of drowsiness or inattention based on the eye status and facial expression evaluation results.

[1198] Input: Eye status and facial expression evaluation results

[1199] Data calculation: Algorithms that detect signs of drowsiness or inattention based on predetermined criteria

[1200] Output: Detected signs of drowsiness or inattention (detected / not detected)

[1201] Specific operation: The server analyzes data such as the time the eyes are closed and periods of expressionlessness to detect drowsiness or inattention.

[1202] Step 5:

[1203] If the server detects signs of drowsiness or inattention, it sends a warning instruction to the terminal.

[1204] Input: Detected signs of drowsiness or inattention

[1205] Data processing: Generate warning instructions and send them to the device

[1206] Output: Warning instructions (audio warning or text message)

[1207] Specific operation: The server generates a warning such as "Caution: You may be drowsy while driving. Please take a break," and sends it to the terminal.

[1208] Step 6:

[1209] Based on the warning instructions received, the device will issue a warning to the driver via voice and text message.

[1210] Input: Warning instruction sent from the server

[1211] Data processing: Sending alerts in the form of voice synthesis or text display

[1212] Output: Driver warning (voice or text message)

[1213] Specific operation: The device will emit a warning message such as "Caution" using a speaker, for example.

[1214] Step 7:

[1215] If the server does not respond to the warning, it will issue an instruction to automatically reduce the vehicle's speed.

[1216] Input: Response status after warning (with / without response)

[1217] Data calculation: Generation of speed control instructions based on a countermeasure algorithm in the event of no response

[1218] Output: Speed ​​reduction command sent to vehicle control unit

[1219] Specific operation: The server checks that there is no response for a certain period of time and then issues an instruction to reduce the vehicle's speed.

[1220] Step 8:

[1221] The server stores all alert history and analysis results in a database and provides a dashboard for administrators.

[1222] Input: Warning history, analysis results

[1223] Data processing: Data storage in a database and dashboard provision via the internet

[1224] Output: Real-time monitorable user interface

[1225] Specific operation: The server accumulates all data and displays it on a dashboard for administrators to monitor in real time.

[1226] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1227] In order to prevent drivers from falling asleep or being distracted while driving, the present invention provides a system that combines an emotion engine, which is composed of the following elements:

[1228] 1. A way to capture the driver's face

[1229] The terminal includes a camera device mounted on the vehicle, which periodically captures the driver's face and transmits the captured image to a server to obtain information such as the eye status and facial expression.

[1230] 2. A means of analyzing captured facial images to assess the driver's eye status and facial expression.

[1231] The server then inputs the captured facial image data into an image analysis algorithm, which evaluates the driver's attention level by evaluating factors such as eye status, facial color, and changes in facial expression.

[1232] 3. Emotion engine that recognizes the driver's emotions

[1233] The server inputs the data obtained from image analysis into an emotion engine to recognize the driver's emotional state, which evaluates, for example, the driver's stress level, fatigue, and irritability.

[1234] 4. Measures to detect signs of drowsiness and inattention

[1235] The server combines the results of both image analysis and the emotion engine to comprehensively detect signs of drowsiness or inattention, such as when the eyes are closed for a certain period of time or when emotional states are detected in combination with signs such as a blank expression.

[1236] 5. A means of issuing warnings if signs of drowsiness or inattention are detected

[1237] If the server detects signs of drowsiness or inattention, it sends a warning instruction to the device. Based on this instruction, the device issues a warning to the driver by voice or text message. Depending on the driver's condition, the warning content may be something like "Caution: You may be drowsy at the wheel. Please take a break" or "Caution: Stress is building. Please stop and take a break."

[1238] 6. A means to store all data and enable administrators to monitor it in real time

[1239] The server stores all warning history and analysis results in a database and provides a dashboard that administrators can access via the internet, allowing them to monitor and manage the status of multiple drivers in real time.

[1240] Explaining program processing in natural language

[1241] Step 1: Initial Setup and Authentication

[1242] The user gets into the vehicle and turns on the terminal (camera device).

[1243] The user enters authentication information, and the terminal transmits the information to the server.

[1244] The server verifies the authentication information and notifies the terminal that the authentication was successful.

[1245] Step 2: Data Acquisition

[1246] The device periodically (e.g., every second) captures an image of the driver's face and sends it to the server.

[1247] Step 3: Data analysis

[1248] The server inputs the received facial images into an image analysis algorithm to evaluate the open / closed state of the eyes, facial expressions, and changes in facial color.

[1249] Step 4: Emotion Recognition

[1250] The server inputs the image analysis results into an emotion engine to identify the driver's emotional state.

[1251] The emotion engine assesses stress levels, fatigue, irritability, etc.

[1252] Step 5: Hazard detection

[1253] The server combines the analysis results, such as when the eyes are closed for a certain period of time (for example, more than three seconds) or when the face is expressionless, with the emotion engine's evaluation to detect signs of drowsiness or inattention.

[1254] Step 6: Warning occurs

[1255] The server sends a warning instruction to the terminal when it detects signs of drowsiness or inattention.

[1256] The terminal receives the warning instruction and issues a warning by voice or text message.

[1257] Step 7: Data storage and management

[1258] The server stores all warning history and analysis results in a database and provides a dashboard for administrators to monitor in real time.

[1259] Specific examples

[1260] For example, consider the case where a driver keeps their eyes closed for three seconds while driving and their facial expression is neutral.

[1261] 1. The user gets into the vehicle and turns on the terminal (camera device).

[1262] 2. The user enters their authentication information and the device sends it to the server.

[1263] 3. The server verifies the authentication information and notifies the terminal that authentication was successful.

[1264] 4. The device captures a facial image every second and sends it to the server.

[1265] 5. The server uses image analysis algorithms to analyze the eye status and facial expressions.

[1266] 6. The server is an emotion engine that evaluates the driver's emotional state (e.g., fatigue, stress).

[1267] 7. The server detects closed eyes for more than 3 seconds and a neutral facial expression, and combines this with the emotion engine's evaluation to recognize signs of drowsiness.

[1268] 8. The server sends a warning instruction to the terminal and generates a message warning of the possibility of drowsiness.

[1269] 9. The device will issue a voice warning saying, "Caution: You may be falling asleep at the wheel. Please take a break."

[1270] 10. The server stores this warning event in a database and allows administrators to view the situation in real time on a dashboard.

[1271] This specific example makes it possible to provide a safe driving environment by detecting drowsy or inattentive driving by the driver at an early stage and issuing a prompt warning. Furthermore, by combining it with an emotion engine, it becomes possible to make comprehensive judgments that take into account the driver's stress and fatigue state.

[1272] The processing flow will be explained below.

[1273] Step 1:

[1274] The user gets into the vehicle and turns on the terminal (camera device).

[1275] The device displays a screen for entering the driver's authentication information.

[1276] Step 2:

[1277] The user enters their authentication information (e.g., ID and password) into the terminal.

[1278] The terminal transmits this authentication information to the server.

[1279] Step 3:

[1280] The server compares the received authentication information with a database and authenticates the driver.

[1281] If the authentication is successful, the server notifies the terminal that the driver authentication was successful.

[1282] Step 4:

[1283] The device will indicate that driver authentication was successful and prepare to enter capture mode.

[1284] Step 5:

[1285] The terminal uses a camera device to capture an image of the driver's face at regular intervals (e.g., every second).

[1286] The terminal transmits the captured images to the server in real time.

[1287] Step 6:

[1288] The server inputs the received facial images into an image analysis algorithm to analyze the eye status and facial expressions.

[1289] The server inputs the image analysis results into an emotion engine to identify the driver's emotional state.

[1290] Step 7:

[1291] The server combines the analysis results, such as when the eyes are closed for a certain period of time (for example, more than three seconds) or when the face is expressionless, with the emotion engine's evaluation to detect signs of drowsiness or inattention.

[1292] Step 8:

[1293] If the server detects signs of drowsiness or inattention, it sends a warning instruction to the terminal.

[1294] The server records this warning event in a database.

[1295] Step 9:

[1296] When the terminal receives a warning instruction from the server, it issues a warning to the driver via voice or text message.

[1297] For example, it may notify you with a message such as, "Caution: You may be drowsy while driving. Please take a break."

[1298] Step 10:

[1299] The server stores all warning history and analysis results in a database and provides a dashboard that allows administrators to monitor in real time.

[1300] Step 11:

[1301] Managers can monitor and manage the status of multiple drivers in real time through a provided dashboard.

[1302] Example 2

[1303] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1304] Currently, there are systems in place to prevent drivers from falling asleep or being distracted while driving, but most of these systems simply rely on eye movements and changes in facial expression. This makes it difficult to comprehensively detect hazards that take into account the driver's emotional state (stress level, fatigue, etc.). Furthermore, there is no system in place to monitor warning history and analysis results in real time. This has led to the problem of insufficient measures being taken to ensure driver safety.

[1305] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for capturing the driver's face, a means for analyzing the captured facial image and evaluating the driver's eye opening / closing state and facial expression, a means for recognizing the driver's emotional state, a means for detecting signs of the driver's drowsiness or inattention, a means for issuing a warning when signs of drowsiness or inattention are detected, and a means for accumulating warning history and analysis results and allowing an administrator to monitor them in real time. This enables comprehensive hazard detection that takes into account not only the driver's eye opening / closing state and changes in facial expression, but also their emotional state. Furthermore, accumulating warning history and analysis results and allowing an administrator to monitor them in real time enables a rapid response, further ensuring the safety of the driver.

[1306] The "means for capturing the driver's face" is a device for periodically acquiring an image of the driver's face using a camera device mounted on the vehicle.

[1307] "Means for analyzing captured facial images and evaluating the driver's eye opening / closing status and facial expression" refers to a device or program that inputs the acquired facial image data into an image analysis algorithm to analyze the eye opening / closing status, facial color, changes in facial expression, etc.

[1308] The "means for recognizing the driver's emotional state" is a device or program that includes an emotion engine for identifying the driver's emotional state, such as stress level, fatigue, or irritability, based on data obtained from image analysis.

[1309] The "means for detecting signs of driver drowsiness or inattention" refers to a device or program that uses an algorithm that integrates the results of image analysis with the evaluation of an emotion engine, and evaluates the eye opening / closing status, changes in facial expression, and emotional state within a certain period of time to determine signs of drowsiness or inattention.

[1310] "Means for issuing a warning when signs of drowsiness or inattention are detected" means a device or program that issues a warning to the driver via voice or text message when signs of drowsiness or inattention are detected.

[1311] "Means for storing warning histories and analysis results and allowing administrators to monitor them in real time" refers to a device or program that stores all warning histories and analysis results in a database and provides a dashboard that administrators can access via the Internet.

[1312] This invention is a safe driving support system that combines facial image analysis and emotion recognition to prevent drivers from falling asleep or being careless while driving. This system is composed of the following elements.

[1313] 1. How to capture the driver's face:

[1314] The terminal includes a camera device mounted on the vehicle, which periodically captures the driver's face and transmits the captured image to a server to obtain information such as the eye status and facial expression.

[1315] 2. Means of analyzing the captured facial image:

[1316] The server inputs the received facial image data into an image analysis algorithm, possibly using software such as OpenCV or TensorFlow. The server evaluates the eye status, facial color, and changes in facial expression to analyze the driver's attention level.

[1317] 3. How to recognize driver emotions:

[1318] The server inputs the data obtained from image analysis into an emotion engine to recognize the driver's emotional state. The emotion engine can use AI models such as Microsoft's Azure Emotion API or Google Cloud's Face API. This allows the system to evaluate the driver's stress level, fatigue, irritability, etc.

[1319] 4. Measures to detect signs of drowsiness or inattention:

[1320] The server integrates the image analysis results with the emotion engine's evaluation to detect signs of drowsiness or inattention. Specifically, it uses an algorithm to determine whether the driver is drowsy or inattentive based on conditions such as eyes closed for a certain period of time (e.g., three seconds or more) or a blank expression.

[1321] 5. Means of issuing a warning if signs of drowsiness or inattention are detected:

[1322] If the server detects signs of drowsiness or inattention, it sends a warning instruction to the device. Based on this instruction, the device issues a warning to the driver by voice or text message. The warning content might be something like, "Caution: You may be drowsy at the wheel. Please take a break."

[1323] 6. A way to store all data and allow administrators to monitor it in real time:

[1324] The server stores all warning history and analysis results in a database and provides a dashboard that administrators can access via the internet, allowing them to monitor and manage the status of multiple drivers in real time.

[1325] Specific examples

[1326] For example, consider the case where a driver keeps their eyes closed for three seconds while driving and their facial expression is neutral.

[1327] 1. The user gets into the vehicle and turns on the terminal (camera device).

[1328] 2. The user enters their authentication information and the device sends it to the server.

[1329] 3. The server verifies the authentication information and notifies the terminal that authentication was successful.

[1330] 4. The device captures a facial image every second and sends it to the server.

[1331] 5. The server uses image analysis algorithms to analyze the eye status and facial expressions.

[1332] 6. The server is an emotion engine that evaluates the driver's emotional state (e.g., fatigue, stress).

[1333] 7. The server detects closed eyes for more than 3 seconds and a neutral facial expression, and combines this with the emotion engine's evaluation to recognize signs of drowsiness.

[1334] 8. The server sends a warning instruction to the terminal and generates a message warning of the possibility of drowsiness.

[1335] 9. The device will issue a voice warning saying, "Caution: You may be falling asleep at the wheel. Please take a break."

[1336] 10. The server stores this warning event in a database and allows administrators to view the situation in real time on a dashboard.

[1337] Example prompts for generative AI models

[1338] A detailed description of the system can be automatically generated by using the following prompts to input data into the generative AI model:

[1339] Based on the following information, generate a detailed description of a system that prevents drivers from falling asleep or being distracted while driving:

[1340] The system consists of a camera device, a server, and an emotion engine.

[1341] 1. The vehicle-mounted camera periodically captures the driver's face and sends it to a server.

[1342] 2. The server analyzes the image and evaluates the eye status and facial expression.

[1343] 3. The emotion engine identifies the driver's emotional state based on the analysis results.

[1344] 4. The server detects signs of drowsiness or inattention, such as closed eyes for a period of time or a blank expression.

[1345] 5. The server sends a warning instruction to the device and warns the driver via voice or text message.

[1346] 6. All data is stored on the server and can be viewed on a dashboard where administrators can monitor in real time.

[1347] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1348] Step 1: Initial Setup and Authentication

[1349] The user gets into the vehicle and turns on the terminal (camera device). The terminal checks the connection between the camera device and the network. The user enters authentication information (e.g., ID and password) as input, which the terminal sends to the server.

[1350] The server checks the received authentication information against the database to see if the authentication was successful, and notifies the terminal that the authentication was successful.

[1351] Step 2: Data Acquisition

[1352] After successful authentication, the device periodically (e.g., every second) captures an image of the driver's face using a camera device, and the captured face image data is generated as input.

[1353] The terminal compresses the captured image data and sends it to the server. The output compressed data is sent and reaches the server.

[1354] Step 3: Image analysis

[1355] The server feeds the received facial image data into an image analysis algorithm, which receives the compressed facial image data as input.

[1356] The server uses software such as OpenCV and TensorFlow to analyze eye status, facial color, and facial expression changes, and generates data on eye status and facial expressions as output.

[1357] Step 4: Emotion Recognition

[1358] The server inputs the data obtained from the image analysis into the emotion engine, providing data such as eye opening and closing status and facial expressions.

[1359] The emotion engine uses AI models (e.g., Azure Emotion API, Google Cloud Face API) to identify the driver's emotional state (e.g., stress level, fatigue, irritability). As an output, data about the driver's emotional state is generated.

[1360] Step 5: Hazard detection

[1361] The server integrates data on eye opening and closing and facial expressions with the emotion engine's evaluation. Analysis data and emotional state data are integrated as input.

[1362] The server runs an algorithm to detect signs of drowsiness or inattention. Specifically, it evaluates whether the eyes are closed for a certain period of time (e.g., more than 3 seconds) or whether the face is neutral. As an output, it generates warning data about signs of drowsiness or inattention.

[1363] Step 6: Warning occurs

[1364] The server sends a warning instruction to the terminal when it detects signs of drowsiness or inattention. As input, a warning instruction is generated.

[1365] The device receives the warning instruction and issues a warning via voice or text message. Specifically, it uses a generative AI model to generate a prompt sentence: "Caution: You may be drowsy at the wheel. Please take a break." As an output, the warning message is provided to the driver.

[1366] Step 7: Data storage and management

[1367] The server stores all the warning history and analysis results in a database. As input, the warning data and analysis data are stored in the database.

[1368] The server provides a dashboard that can be accessed by the administrator. Specifically, it builds an interface that allows the administrator to monitor the driver status in real time. As an output, a dashboard that can be accessed by the administrator is created.

[1369] (Application example 2)

[1370] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1371] Improving safety is a key issue for modern self-driving vehicles. Driver drowsiness and inattention can lead to serious accidents, so technology is needed to detect these early and provide appropriate warnings. Furthermore, there is a lack of systems that can make comprehensive judgments by taking into account the driver's emotional state, and this needs to be addressed.

[1372] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing the driver's face, means for analyzing the captured facial image and evaluating the driver's eye opening / closing state and facial expression, means for recognizing the driver's emotional state using an emotion engine, means for detecting signs of the driver's drowsiness or inattention, means for issuing a warning when signs of drowsiness or inattention are detected, means for accumulating all data so that an administrator can monitor it in real time, and means for transmitting data to a cloud server for analysis and issuing a warning instruction. This enables early detection of the driver's drowsiness or inattention and appropriate warning, as well as comprehensive judgment taking the driver's emotional state into consideration.

[1373] "Means for capturing the driver's face" refers to a device or sensor for acquiring an image of the driver's face, and generally refers to a camera device.

[1374] "Means for analyzing the captured facial image and evaluating the driver's eye opening / closing state and facial expression" refers to a device or algorithm that analyzes the driver's eye opening / closing state and changes in facial expression based on the acquired facial image.

[1375] "Means for recognizing the driver's emotional state using an emotion engine" refers to an algorithm or system for assessing the driver's emotional state (e.g., stress level, fatigue, irritability, etc.) based on analyzed facial information.

[1376] The "means for detecting signs of driver drowsiness or inattention" is a device or program for detecting signs of drowsiness or inattention by integrating the analysis results of the driver's eye opening / closing state and facial expressions with the evaluation of the emotion engine.

[1377] A "means for issuing a warning when signs of drowsiness or inattention are detected" is a device or system for issuing a warning to the driver via voice or text message based on detected signs of drowsiness or inattention.

[1378] "A means for storing all data and allowing administrators to monitor in real time" refers to a system that stores warning history and analysis results in a database and provides a dashboard that administrators can access and monitor in real time via the Internet.

[1379] The "means for transmitting data to a cloud server and issuing analysis and warning instructions" refers to a communication device and system for transmitting a driver's facial image and analysis data to a cloud server and issuing a warning instruction based on the analysis results of the cloud server.

[1380] This invention provides a system that combines an emotion engine to prevent drivers from falling asleep or being distracted while driving. This system is composed of the following elements:

[1381] Driver face capture methods:

[1382] A camera device is installed in the vehicle to periodically capture the driver's face, and the camera device is responsible for sending the captured facial images to a cloud server.

[1383] A means of analyzing captured facial images to assess the driver's eye status and facial expression:

[1384] The cloud server inputs the received facial image data into an image analysis algorithm (e.g., TensorFlow, OpenCV), which evaluates the eye status, facial color, and changes in facial expression to analyze the driver's attention level.

[1385] Using an emotion engine to recognize the driver's emotional state:

[1386] The cloud server inputs the data obtained from image analysis into an emotion engine (e.g., Microsoft Azure Emotion API) to recognize the driver's emotional state. The emotion engine evaluates the driver's stress level, fatigue, irritability, etc.

[1387] Ways to detect signs of driver drowsiness or inattention:

[1388] The cloud server combines the analysis results, such as when the eyes are closed for a certain period of time (e.g., more than 3 seconds) or when the face is expressionless, with the emotion engine's evaluation to comprehensively detect signs of drowsiness or inattention.

[1389] To alert you if signs of drowsiness or inattention are detected:

[1390] If the cloud server detects signs of drowsiness or inattention, it sends a warning instruction to the device, which then issues a voice or text message to the driver.

[1391] A way to store all data and allow administrators to monitor it in real time:

[1392] The cloud server stores all warning history and analysis results in a database and provides a dashboard that administrators can access via the internet, allowing them to monitor and manage the status of multiple drivers in real time.

[1393] Program processing:

[1394] The system periodically captures various data and sends it to a cloud server. The cloud server then runs the received data through an image analysis algorithm to evaluate eye status and facial expressions. It then uses an emotion engine to recognize the driver's emotional state and integrates the analysis results to detect signs of drowsiness or inattention. If signs are detected, the server sends a warning instruction to the device, which then issues a voice or text warning. Finally, all data is stored in a database, allowing administrators to monitor it in real time.

[1395] Examples:

[1396] Consider a case where a driver becomes tired while driving and closes their eyes for more than three seconds. At this time, a camera device captures the driver's face and sends the data to a cloud server. The server uses an image analysis algorithm to evaluate whether the eyes are open or closed, and an emotion engine to recognize the driver's fatigue state. The cloud server integrates these analysis results, detects signs of drowsiness, and sends a warning instruction to the device. The device issues an audio warning saying, "Caution: You may be drowsy while driving. Please take a break." This series of data is stored in a cloud database, allowing administrators to monitor it in real time.

[1397] Example prompt sentence:

[1398] "Images of the driver's face are captured every second and sent to a cloud server. The server uses image analysis algorithms and an emotion engine to evaluate eye status and facial expressions. If signs of drowsy driving or inattention are detected, the cloud server sends warning instructions to the device and warns via voice and text messages. All data is stored in a cloud database and can be monitored in real time by administrators."

[1399] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1400] Step 1:

[1401] The device is started in the vehicle and the user enters authentication information. The authentication information (e.g., facial recognition data or password) is sent from the device to the cloud server. The server verifies the authentication information and notifies the device that authentication was successful.

[1402] Input: User credentials

[1403] Output: Authentication success notification

[1404] Specific operation: When a user enters a vehicle, the camera device performs facial recognition. The authentication information is sent to the cloud server, and if the authentication is successful, the device is notified.

[1405] Step 2:

[1406] The device periodically (e.g., every second) captures the driver's facial image using a camera device, and the captured facial image is sent to a cloud server.

[1407] Input: Driver's face image

[1408] Output: Sending facial image data to the cloud server

[1409] Specific operation: The device's camera takes a picture of the driver's face every second and sends the image data to a cloud server.

[1410] Step 3:

[1411] The server then inputs the received facial images into an image analysis algorithm (e.g., TensorFlow, OpenCV), which processes the images to evaluate eye status and facial expressions.

[1412] Input: Facial image data

[1413] Output: Eye open / close status and facial expression analysis results

[1414] How it works: The cloud server uses image analysis algorithms to evaluate eye status and facial expressions.

[1415] Step 4:

[1416] The server inputs the data into an emotion engine (e.g., Microsoft Azure Emotion API) based on the image analysis results and evaluates the driver's emotional state (stress level, fatigue level, etc.).

[1417] Input: Image analysis results

[1418] Output: Emotional state evaluation result

[1419] Specific operation: The cloud server inputs the image analysis results into the emotion engine to evaluate the driver's stress level and fatigue state.

[1420] Step 5:

[1421] The server combines the analysis results, such as when the eyes are closed for a certain period of time (for example, more than three seconds) or when the face is expressionless, with the emotion engine's evaluation to detect signs of drowsiness or inattention.

[1422] Input: Eye status and emotional state evaluation results

[1423] Output: Detected signs of drowsiness and inattention

[1424] Specific operation: The cloud server integrates the evaluation results of eye opening / closing status and emotional state to detect signs of drowsiness or inattention.

[1425] Step 6:

[1426] If the server detects signs of drowsiness or inattention, it sends warning instructions to the device, which then issues voice and text message warnings based on the instructions.

[1427] Input: Result of detecting signs of drowsiness and inattention

[1428] Output: Warning instructions and warning messages

[1429] Specific operation: The cloud server detects signs of drowsiness and sends a warning instruction to the device. The device then issues a warning saying, "Caution: You may be drowsy at the wheel. Please take a break."

[1430] Step 7:

[1431] The server stores all warning history and analysis results in a database and provides an internet-accessible dashboard for administrators, allowing them to monitor and manage the status of multiple drivers in real time.

[1432] Input: Warning history and analysis results

[1433] Output: Save to database and reflect on dashboard

[1434] Specific operation: The cloud server stores warning events and analysis results in a database, and administrators can monitor them in real time through a dashboard.

[1435] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1436] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1437] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1438] [Fourth embodiment]

[1439] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1440] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1441] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1442] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1443] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1444] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1445] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1446] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1447] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1448] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1449] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1450] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1451] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1452] This invention is a system for preventing drivers in the transportation industry from falling asleep or being distracted while driving, and is composed of the following elements:

[1453] 1. A way to capture the driver's face

[1454] The terminal includes a camera device mounted on the vehicle. This camera device periodically captures the driver's face and sends the captured images to a server, thereby obtaining images of the driver's eye status and facial expressions.

[1455] 2. A means of analyzing captured facial images to assess the driver's eye status and facial expression.

[1456] The server then inputs the captured facial image data into an image analysis algorithm, which evaluates the driver's condition by assessing eye status, facial color, and changes in facial expression.

[1457] 3. Measures to detect signs of drowsiness and inattention

[1458] The server detects signs of drowsiness or inattention based on the results of image analysis, such as when the eyes remain closed for a certain period of time or when the facial expression becomes neutral. These criteria are built into the image analysis algorithm.

[1459] 4. A means of issuing warnings if signs of drowsiness or inattention are detected

[1460] When the server detects signs of drowsiness or inattention, it sends a warning instruction to the device. Based on this instruction, the device issues a warning to the driver by voice or text message. An example of a voice warning is, "Caution: You may be drowsy at the wheel. Please take a break."

[1461] 5. A means to accumulate all this data and allow administrators to monitor it in real time

[1462] The server stores all warning history and analysis results in a database and provides an administrator dashboard connected via the internet, which allows administrators to monitor and manage the status of multiple drivers in real time.

[1463] Explaining program processing in natural language

[1464] Step 1: Initial Setup and Authentication

[1465] The user gets into the vehicle, starts up the terminal (camera device), and authenticates himself / herself.

[1466] The device sends authentication information to the server to verify that it is the driver.

[1467] Step 2: Data Acquisition

[1468] The device periodically (for example, every second) captures an image of the driver's face and sends it to the server.

[1469] Step 3: Data analysis

[1470] The server inputs the received facial images into an analysis algorithm to evaluate whether the eyes are open or closed and changes in facial expression.

[1471] The server assesses the driver's level of attention and detects signs of drowsiness or inattention.

[1472] Step 4: Warning occurs

[1473] If the server detects signs of drowsiness or inattention, it sends a warning instruction to the terminal.

[1474] Step 5: Warning Notification

[1475] The device will warn the driver via voice and text messages based on the received warning instructions.

[1476] Step 6: Data storage and management

[1477] The server stores all warning history and analysis results in a database.

[1478] Managers can monitor and manage the status of multiple drivers in real time using an internet-based dashboard.

[1479] Specific examples

[1480] For example, consider a situation where a driver keeps their eyes closed for three seconds while driving.

[1481] 1. The user starts driving, and the device (camera device) captures facial images every second and sends them to the server.

[1482] 2. The server analyzes the captured facial images and determines that the eyes are closed in the first image.

[1483] 3. Next, the eyes are recognized as closed in the next two images (total of 3 seconds).

[1484] 4. The server detects signs of drowsiness because the eyes remain closed for more than 3 seconds.

[1485] 5. The server sends a warning instruction to the terminal, and the terminal issues a voice warning saying, "Caution: You may be drowsy while driving. Please take a break."

[1486] 6. The server stores this warning event in a database and makes it available for administrators to view in real time on a dashboard.

[1487] This series of steps enables the system to detect drowsy or careless driving by the driver in real time and issue a prompt warning, contributing to maintaining a safe driving environment.

[1488] The processing flow will be explained below.

[1489] Step 1:

[1490] The user gets into the vehicle and turns on the terminal (camera device).

[1491] The device displays a screen for entering the driver's authentication information.

[1492] Step 2:

[1493] The user enters their authentication information (e.g., ID and password) into the terminal.

[1494] The terminal transmits this authentication information to the server.

[1495] Step 3:

[1496] The server compares the received authentication information with a database and authenticates the driver.

[1497] If the authentication is successful, the server returns a message to the terminal indicating that the driver authentication was successful.

[1498] Step 4:

[1499] The device will indicate that driver authentication was successful and prepare to enter capture mode.

[1500] Step 5:

[1501] The terminal uses a camera device to capture an image of the driver's face at regular intervals (e.g., every second).

[1502] The terminal transmits the captured images to the server in real time.

[1503] Step 6:

[1504] The server then feeds the received facial images into an image analysis algorithm.

[1505] The server analyzes the eye status and facial expressions to assess the driver's level of attention.

[1506] Step 7:

[1507] The server detects signs of drowsiness if the driver's eyes remain closed for a certain period of time (for example, three seconds or more).

[1508] The server also applies if signs of inattention (e.g., a blank expression) are detected.

[1509] Step 8:

[1510] The server sends a warning instruction to the terminal when signs of drowsiness or inattention are detected.

[1511] The server records this warning event in a database.

[1512] Step 9:

[1513] When the terminal receives a warning instruction from the server, it issues a warning to the driver via a voice message (for example, "Caution: You may be drowsy while driving. Please take a break.") or a text message.

[1514] Step 10:

[1515] The server stores all warning history and analysis results in a database and provides a dashboard that allows administrators to monitor in real time via the Internet.

[1516] Step 11:

[1517] Managers can monitor and manage the status of multiple drivers in real time through a provided dashboard.

[1518] Example 1

[1519] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1520] Many accidents occur due to drivers in the transportation industry falling asleep or being distracted while driving. This not only results in personal injury and property damage, and compromises safety, but also leads to economic losses. Therefore, a system is needed that can monitor driver attention in real time, quickly detect signs of drowsiness or inattention, and issue appropriate warnings.

[1521] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1522] In this invention, the server includes means for capturing the driver's face, means for analyzing the captured facial image and evaluating the driver's eye opening / closing state and facial expression, means for detecting signs of drowsiness or inattention of the driver, means for issuing a warning when signs of drowsiness or inattention are detected, and means for storing all warning history and analysis results in a database that an administrator can monitor in real time via the Internet. This makes it possible to monitor the driver's attention in real time and prevent drowsy driving and inattentive driving.

[1523] "Driver" refers to a person who performs driving duties, especially a person who operates a vehicle in the transportation industry.

[1524] "Facial capture means" means a camera device or equivalent installed in a vehicle for capturing an image of the driver's face.

[1525] "Means for analyzing images" refers to image analysis algorithms and software for evaluating the open / closed state of the eyes and changes in facial expression based on acquired facial images.

[1526] "Means for detecting signs of drowsiness or inattention" refers to algorithms or software that determine whether the driver is drowsy or inattentive from analyzed image data.

[1527] "Warning means" refers to devices or software that alert the driver via voice or text message if signs of drowsiness or inattention are detected.

[1528] "Means for storing in a database" refers to a database system for storing all warning history and analysis results.

[1529] "Means that allow an administrator to monitor in real time via the Internet" refers to a dashboard or web application that allows an administrator to monitor and manage the status of multiple drivers in real time via the Internet.

[1530] "Camera device" means an optical device mounted on a vehicle for capturing an image of the driver's face.

[1531] This invention is a system for preventing drivers in the transportation industry from falling asleep or being distracted while driving. The system includes a means for capturing the driver's face, a means for analyzing the captured facial image, a means for detecting signs of drowsiness or inattention, a means for issuing a warning, and a means for allowing a manager to monitor in real time.

[1532] Hardware Configuration

[1533] The terminal includes a camera device mounted on the vehicle, which periodically captures images of the driver's face and transmits the captured images to a server.

[1534] The server is a combination of a high-performance computer and database system, equipped with specialized software for running image analysis algorithms.

[1535] The administrator dashboard is a web-based application that allows real-time data monitoring and management over the Internet.

[1536] Software Configuration

[1537] The server includes the following software modules:

[1538] 1. Image capture module: processes image data received from the terminal.

[1539] 2. Image analysis module: The received image data is fed into an algorithm to evaluate the eye opening / closing status and changes in facial expression.

[1540] 3. Drowsiness detection algorithm: Based on the results of image analysis, it detects signs of drowsiness or inattention.

[1541] 4. Warning generation module: Generates warning instructions when signs of drowsiness or inattention are detected.

[1542] 5. Data storage module: All warning history and analysis results are stored in a database.

[1543] 6. Dashboard module: Provides a real-time monitoring interface for administrators.

[1544] System Operation

[1545] The user enters the vehicle, activates the terminal (camera device), and authenticates themselves. The terminal sends the authentication information to the server, and the authentication is confirmed.

[1546] The device periodically (for example, every second) captures an image of the driver's face and sends it to the server.

[1547] The server then inputs the received facial images into an analysis algorithm that evaluates the eye status and changes in facial expression to determine the driver's attention level and detect signs of drowsiness or inattention.

[1548] If the server detects signs of drowsiness or inattention, it sends a warning instruction to the terminal.

[1549] Based on the warning instructions received, the device will issue a warning to the driver via voice and text message.

[1550] The server stores all warning history and analysis results in a database, allowing administrators to monitor and manage the status of multiple drivers in real time using a dashboard connected via the Internet.

[1551] Specific examples

[1552] For example, if a driver keeps their eyes closed for three seconds while driving, here's a specific example:

[1553] 1. The user starts driving, and the device (camera device) captures facial images every second and sends them to the server.

[1554] 2. The server analyzes the captured facial images and determines that the eyes are closed in the first image.

[1555] 3. Next, the eyes are recognized as closed in the next two images (total of 3 seconds).

[1556] 4. The server detects signs of drowsiness because the eyes remain closed for more than 3 seconds.

[1557] 5. The server sends a warning instruction to the terminal, and the terminal issues a voice warning saying, "Caution: You may be drowsy while driving. Please take a break."

[1558] 6. The server stores this warning event in a database and makes it available for administrators to view in real time on a dashboard.

[1559] Prompt Sentence Examples

[1560] "Please explain the procedures and give specific examples of systems to prevent drivers from falling asleep or being distracted while driving in the transportation industry."

[1561] The system is designed to provide a concrete form to support the technical scope of an invention and to help others accurately understand and practice the invention.

[1562] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1563] Step 1:

[1564] The user gets into the vehicle, turns on the terminal (camera device), and authenticates himself / herself.

[1565] Input: User authentication information (e.g., facial recognition data or ID input)

[1566] Output: The device sends the user's authentication information to the server.

[1567] Specific operation: When the user turns their face towards the device, the camera captures an image and obtains authentication information.

[1568] Step 2:

[1569] The terminal sends the received authentication information to the server.

[1570] Input: User credentials captured by the device

[1571] Output: Authentication information sent to the server.

[1572] Specific operation: The terminal's sending function automatically sends the authentication information to the server.

[1573] Step 3:

[1574] The server compares the received authentication information with a database and returns the authentication result to the terminal.

[1575] Input: User credentials received by the server

[1576] Output: The authentication result (success or failure) is returned to the terminal.

[1577] What happens: The server searches its database and finds a record that matches the authentication information.

[1578] Step 4:

[1579] The device captures an image of the driver's face every second and sends it to a server.

[1580] Input: A face image captured by the device

[1581] Output: The image is sent to the server.

[1582] Specific operation: The device's camera operates once per second and transmits the captured images via the network.

[1583] Step 5:

[1584] The facial images received by the server are fed into an image analysis algorithm to evaluate the open / closed state of the eyes and changes in facial expression.

[1585] Input: A face image received by the server

[1586] Output: Eye status and facial expression evaluation results

[1587] What happens: The server runs image analysis software and analyzes each frame.

[1588] Step 6:

[1589] Based on the analysis results, the server determines the driver's level of attention and detects signs of drowsiness or inattention.

[1590] Input: Eye status and facial expression evaluation results obtained from the server

[1591] Output: Presence or absence of signs of drowsiness or inattention

[1592] Specific operation: The server monitors the image analysis results over a certain period of time and detects abnormalities.

[1593] Step 7:

[1594] If the server detects signs of drowsiness or inattention, it generates a warning instruction and sends it to the terminal.

[1595] Input: Detected signs of drowsiness or inattention

[1596] Output: Warning instruction (e.g. "Caution: You may be drowsy while driving. Please take a break.")

[1597] Specific operation: The server generates a warning message and sends it to the terminal via the network.

[1598] Step 8:

[1599] Based on the warning instructions received by the device, the device will warn the driver via voice and text messages.

[1600] Input: Warning instruction from the server

[1601] Output: Voice and text message alerts

[1602] Specific operation: The device will display a warning using audio equipment or a display, and will notify you by voice.

[1603] Step 9:

[1604] The server stores all warning history and analysis results in a database.

[1605] Input: Analysis results and warning history

[1606] Output: Warning history and analysis results stored in a database

[1607] Specific operation: The server adds and updates the analysis results and warning information to the database.

[1608] Step 10:

[1609] The server displays all warning history and real-time driver status through an administrator dashboard.

[1610] Input: Stored alert history and real-time data

[1611] Output: Information displayed on the admin dashboard

[1612] What it does: The server processes the collected data in real time and displays it in a web interface.

[1613] This series of processes makes it possible to monitor the driver's attention level in real time, quickly detect signs of drowsiness or inattention, and prevent accidents before they occur.

[1614] (Application example 1)

[1615] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1616] In conventional vehicle operation, there are systems that prevent drivers from falling asleep or being distracted while driving, but these systems do not provide comprehensive safety measures such as driver authentication or response measures when a warning is issued, such as vehicle speed control. As a result, risks remain if the driver does not respond immediately after receiving a warning. The present invention aims to solve these problems by not only monitoring the driver's state in real time and detecting signs of inattention or drowsiness, but also providing appropriate warning measures and additional safety measures if no response is received.

[1617] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1618] In this invention, the server includes means for capturing the driver's face, means for analyzing the captured facial image and evaluating the driver's eye opening / closing state and facial expression, means for detecting signs of driver drowsiness or inattention, means for issuing a warning when signs of drowsiness or inattention are detected, means for accumulating all of this data and enabling an administrator to monitor it in real time, means for authenticating the driver, and means for automatically reducing the vehicle speed if the warning is not responded to. This makes it possible to comprehensively monitor the driver's condition and improve safety.

[1619] "Means for capturing the driver's face" refers to an apparatus and method for periodically capturing images of the driver's face using a camera device mounted on the vehicle.

[1620] "Means for analyzing captured facial images and evaluating the driver's eye opening / closing state and facial expression" refers to technology and algorithms that use captured facial images to analyze the driver's eye opening / closing state and changes in facial expression in real time.

[1621] "Means for detecting signs of driver drowsiness or inattention" refers to technology and algorithms that identify signs of drowsiness or inattention based on analyzed data on eye status and facial expressions, and trigger necessary actions.

[1622] "Means for issuing a warning when signs of drowsiness or inattention are detected" refers to a device and method that issues a warning to the driver via voice or text message when the driver shows signs of drowsiness or inattention.

[1623] "A means of accumulating all this data and allowing administrators to monitor it in real time" refers to a system that stores driver status data centrally on a server and allows administrators to monitor that data in real time via a dashboard over the Internet.

[1624] "Means for authenticating a driver" refers to technology and devices that authenticate the driver when he or she gets into a vehicle and transmit the authentication information to a server for confirmation.

[1625] "Means for automatically reducing vehicle speed when warnings are not responded to" refers to mechanisms and methods that automatically reduce vehicle speed to maintain a safe driving environment if the driver does not respond to a warning.

[1626] The present invention is an advanced monitoring system for preventing drivers from falling asleep at the wheel or from driving carelessly. Next, an embodiment of the present invention will be described in detail.

[1627] System Configuration

[1628] The system includes the following components:

[1629] 1. Means for capturing the driver's face: Images of the driver's face are periodically captured using a camera device mounted on the vehicle.

[1630] 2. A means for analyzing the captured facial image and evaluating the driver's eye opening / closing state and facial expression: Using an image analysis algorithm on the server, the driver's eye opening / closing state and facial expression are evaluated from the captured image.

[1631] 3. Means of driver authentication: When the driver gets into the vehicle, personal authentication is performed and the information is sent to the server.

[1632] 4. Means for detecting signs of driver drowsiness or inattention: An algorithm is used to detect signs of drowsiness or inattention, such as when the driver opens and closes their eyes repeatedly within a specified period of time.

[1633] 5. Means of issuing a warning when signs of drowsiness or inattention are detected: Based on the detection results, a warning instruction is sent to the device and a warning is issued by voice or text message.

[1634] 6. Means for automatically reducing vehicle speed when warning is not responded to: If a warning is not responded to, the speed of the vehicle is automatically reduced.

[1635] 7. A means of accumulating all this data and allowing administrators to monitor it in real time: All warning history and analysis results are stored in a database, and an administrator dashboard is provided that is connected via the Internet.

[1636] Hardware and Software Configuration

[1637] Hardware: camera devices, vehicle control units, network connectivity devices

[1638] Software: Facial recognition algorithm, eye open / close state analysis algorithm, facial expression analysis algorithm, database management system, warning generation system, vehicle speed control program, administrator dashboard

[1639] Data processing and calculation

[1640] The server receives image data sent from the camera device and analyzes facial recognition, eye status, and facial expressions using image processing libraries such as TensorFlow and OpenCV. The server comprehensively evaluates this data and detects signs of drowsiness or inattention.

[1641] If signs of drowsiness or inattention are detected, the server sends a warning instruction to the vehicle's warning system, which then alerts the occupants via smartphone or head-mounted display. If the warning is not responded to, the vehicle control unit automatically reduces the speed.

[1642] Specific examples

[1643] For example, consider a case where a driver keeps their eyes closed for three seconds while driving. The camera device captures an image of their face every second and sends it to the server. The server analyzes the image and recognizes that their eyes are closed. If the server determines that their eyes are still closed for the next two seconds, it issues a warning that the driver may be drowsy. If the driver does not respond to this warning, the vehicle's speed will be automatically reduced to ensure safety.

[1644] Prompt Sentence Examples

[1645] "Develop an algorithm that analyzes facial images captured periodically by the camera to assess eye status and facial expression. We hope to create a real-time surveillance application, including a system that issues an alert if it detects signs of drowsiness or inattention."

[1646] By implementing the invention in this manner, it is possible to prevent drivers from falling asleep or driving carelessly, thereby providing a safe driving environment.

[1647] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1648] Step 1:

[1649] The user gets into the vehicle, activates the terminal (camera device), and authenticates himself / herself.

[1650] Input: User authentication information (e.g., biometrics or ID card)

[1651] Data processing: The device sends the authentication information to the server and authenticates the driver.

[1652] Output: Authentication result (success or failure)

[1653] Specific operation: The device captures the user's face with a camera and checks whether it matches a familiar driver.

[1654] Step 2:

[1655] The device periodically (for example, every second) captures an image of the driver's face and sends it to the server.

[1656] Input: Image data obtained from the camera device (driver's face)

[1657] Data processing: Images are captured and sent to the server

[1658] Output: Latest facial image data stored on the server

[1659] Specific operation: The device's camera captures the driver's face every second and sends the image to the server.

[1660] Step 3:

[1661] The server inputs the received facial images into an analysis algorithm to evaluate whether the eyes are open or closed and changes in facial expression.

[1662] Input: Facial image data sent to the server

[1663] Data calculation: Uses face recognition and facial expression analysis algorithms to assess eye status and facial expressions

[1664] Output: Evaluation results of eye opening / closing status and facial expression

[1665] How it works: The server uses a facial recognition algorithm to detect whether the eyes are open or closed, and based on this, determines signs of drowsiness or inattention.

[1666] Step 4:

[1667] The server detects signs of drowsiness or inattention based on the eye status and facial expression evaluation results.

[1668] Input: Eye status and facial expression evaluation results

[1669] Data calculation: Algorithms that detect signs of drowsiness or inattention based on predetermined criteria

[1670] Output: Detected signs of drowsiness or inattention (detected / not detected)

[1671] Specific operation: The server analyzes data such as the time the eyes are closed and periods of expressionlessness to detect drowsiness or inattention.

[1672] Step 5:

[1673] If the server detects signs of drowsiness or inattention, it sends a warning instruction to the terminal.

[1674] Input: Detected signs of drowsiness or inattention

[1675] Data processing: Generate warning instructions and send them to the device

[1676] Output: Warning instructions (audio warning or text message)

[1677] Specific operation: The server generates a warning such as "Caution: You may be drowsy while driving. Please take a break," and sends it to the terminal.

[1678] Step 6:

[1679] Based on the warning instructions received, the device will issue a warning to the driver via voice and text message.

[1680] Input: Warning instruction sent from the server

[1681] Data processing: Sending alerts in the form of voice synthesis or text display

[1682] Output: Driver warning (voice or text message)

[1683] Specific operation: The device will emit a warning message such as "Caution" using a speaker, for example.

[1684] Step 7:

[1685] If the server does not respond to the warning, it will issue an instruction to automatically reduce the vehicle's speed.

[1686] Input: Response status after warning (with / without response)

[1687] Data calculation: Generation of speed control instructions based on a countermeasure algorithm in the event of no response

[1688] Output: Speed ​​reduction command sent to vehicle control unit

[1689] Specific operation: The server checks that there is no response for a certain period of time and then issues an instruction to reduce the vehicle's speed.

[1690] Step 8:

[1691] The server stores all alert history and analysis results in a database and provides a dashboard for administrators.

[1692] Input: Warning history, analysis results

[1693] Data processing: Data storage in a database and dashboard provision via the internet

[1694] Output: Real-time monitorable user interface

[1695] Specific operation: The server accumulates all data and displays it on a dashboard for administrators to monitor in real time.

[1696] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1697] In order to prevent drivers from falling asleep or being distracted while driving, the present invention provides a system that combines an emotion engine, which is composed of the following elements:

[1698] 1. A way to capture the driver's face

[1699] The terminal includes a camera device mounted on the vehicle, which periodically captures the driver's face and transmits the captured image to a server to obtain information such as the eye status and facial expression.

[1700] 2. A means of analyzing captured facial images to assess the driver's eye status and facial expression.

[1701] The server then inputs the captured facial image data into an image analysis algorithm, which evaluates the driver's attention level by evaluating factors such as eye status, facial color, and changes in facial expression.

[1702] 3. Emotion engine that recognizes the driver's emotions

[1703] The server inputs the data obtained from image analysis into an emotion engine to recognize the driver's emotional state, which evaluates, for example, the driver's stress level, fatigue, and irritability.

[1704] 4. Measures to detect signs of drowsiness and inattention

[1705] The server combines the results of both image analysis and the emotion engine to comprehensively detect signs of drowsiness or inattention, such as when the eyes are closed for a certain period of time or when emotional states are detected in combination with signs such as a blank expression.

[1706] 5. A means of issuing warnings if signs of drowsiness or inattention are detected

[1707] If the server detects signs of drowsiness or inattention, it sends a warning instruction to the device. Based on this instruction, the device issues a warning to the driver by voice or text message. Depending on the driver's condition, the warning content may be something like "Caution: You may be drowsy at the wheel. Please take a break" or "Caution: Stress is building. Please stop and take a break."

[1708] 6. A means to store all data and enable administrators to monitor it in real time

[1709] The server stores all warning history and analysis results in a database and provides a dashboard that administrators can access via the internet, allowing them to monitor and manage the status of multiple drivers in real time.

[1710] Explaining program processing in natural language

[1711] Step 1: Initial Setup and Authentication

[1712] The user gets into the vehicle and turns on the terminal (camera device).

[1713] The user enters authentication information, and the terminal transmits the information to the server.

[1714] The server verifies the authentication information and notifies the terminal that the authentication was successful.

[1715] Step 2: Data Acquisition

[1716] The device periodically (e.g., every second) captures an image of the driver's face and sends it to the server.

[1717] Step 3: Data analysis

[1718] The server inputs the received facial images into an image analysis algorithm to evaluate the open / closed state of the eyes, facial expressions, and changes in facial color.

[1719] Step 4: Emotion Recognition

[1720] The server inputs the image analysis results into an emotion engine to identify the driver's emotional state.

[1721] The emotion engine assesses stress levels, fatigue, irritability, etc.

[1722] Step 5: Hazard detection

[1723] The server combines the analysis results, such as when the eyes are closed for a certain period of time (for example, more than three seconds) or when the face is expressionless, with the emotion engine's evaluation to detect signs of drowsiness or inattention.

[1724] Step 6: Warning occurs

[1725] The server sends a warning instruction to the terminal when it detects signs of drowsiness or inattention.

[1726] The terminal receives the warning instruction and issues a warning by voice or text message.

[1727] Step 7: Data storage and management

[1728] The server stores all warning history and analysis results in a database and provides a dashboard for administrators to monitor in real time.

[1729] Specific examples

[1730] For example, consider the case where a driver keeps their eyes closed for three seconds while driving and their facial expression is neutral.

[1731] 1. The user gets into the vehicle and turns on the terminal (camera device).

[1732] 2. The user enters their authentication information and the device sends it to the server.

[1733] 3. The server verifies the authentication information and notifies the terminal that authentication was successful.

[1734] 4. The device captures a facial image every second and sends it to the server.

[1735] 5. The server uses image analysis algorithms to analyze the eye status and facial expressions.

[1736] 6. The server is an emotion engine that evaluates the driver's emotional state (e.g., fatigue, stress).

[1737] 7. The server detects closed eyes for more than 3 seconds and a neutral facial expression, and combines this with the emotion engine's evaluation to recognize signs of drowsiness.

[1738] 8. The server sends a warning instruction to the terminal and generates a message warning of the possibility of drowsiness.

[1739] 9. The device will issue a voice warning saying, "Caution: You may be falling asleep at the wheel. Please take a break."

[1740] 10. The server stores this warning event in a database and allows administrators to view the situation in real time on a dashboard.

[1741] This specific example makes it possible to provide a safe driving environment by detecting drowsy or inattentive driving by the driver at an early stage and issuing a prompt warning. Furthermore, by combining it with an emotion engine, it becomes possible to make comprehensive judgments that take into account the driver's stress and fatigue state.

[1742] The processing flow will be explained below.

[1743] Step 1:

[1744] The user gets into the vehicle and turns on the terminal (camera device).

[1745] The device displays a screen for entering the driver's authentication information.

[1746] Step 2:

[1747] The user enters their authentication information (e.g., ID and password) into the terminal.

[1748] The terminal transmits this authentication information to the server.

[1749] Step 3:

[1750] The server compares the received authentication information with a database and authenticates the driver.

[1751] If the authentication is successful, the server notifies the terminal that the driver authentication was successful.

[1752] Step 4:

[1753] The device will indicate that driver authentication was successful and prepare to enter capture mode.

[1754] Step 5:

[1755] The terminal uses a camera device to capture an image of the driver's face at regular intervals (e.g., every second).

[1756] The terminal transmits the captured images to the server in real time.

[1757] Step 6:

[1758] The server inputs the received facial images into an image analysis algorithm to analyze the eye status and facial expressions.

[1759] The server inputs the image analysis results into an emotion engine to identify the driver's emotional state.

[1760] Step 7:

[1761] The server combines the analysis results, such as when the eyes are closed for a certain period of time (for example, more than three seconds) or when the face is expressionless, with the emotion engine's evaluation to detect signs of drowsiness or inattention.

[1762] Step 8:

[1763] If the server detects signs of drowsiness or inattention, it sends a warning instruction to the terminal.

[1764] The server records this warning event in a database.

[1765] Step 9:

[1766] When the terminal receives a warning instruction from the server, it issues a warning to the driver via voice or text message.

[1767] For example, it may notify you with a message such as, "Caution: You may be drowsy while driving. Please take a break."

[1768] Step 10:

[1769] The server stores all warning history and analysis results in a database and provides a dashboard that allows administrators to monitor in real time.

[1770] Step 11:

[1771] Managers can monitor and manage the status of multiple drivers in real time through a provided dashboard.

[1772] Example 2

[1773] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1774] Currently, there are systems in place to prevent drivers from falling asleep or being distracted while driving, but most of these systems simply rely on eye movements and changes in facial expression. This makes it difficult to comprehensively detect hazards that take into account the driver's emotional state (stress level, fatigue, etc.). Furthermore, there is no system in place to monitor warning history and analysis results in real time. This has led to the problem of insufficient measures being taken to ensure driver safety.

[1775] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for capturing the driver's face, a means for analyzing the captured facial image and evaluating the driver's eye opening / closing state and facial expression, a means for recognizing the driver's emotional state, a means for detecting signs of the driver's drowsiness or inattention, a means for issuing a warning when signs of drowsiness or inattention are detected, and a means for accumulating warning history and analysis results and allowing an administrator to monitor them in real time. This enables comprehensive hazard detection that takes into account not only the driver's eye opening / closing state and changes in facial expression, but also their emotional state. Furthermore, accumulating warning history and analysis results and allowing an administrator to monitor them in real time enables a rapid response, further ensuring the safety of the driver.

[1776] The "means for capturing the driver's face" is a device for periodically acquiring an image of the driver's face using a camera device mounted on the vehicle.

[1777] "Means for analyzing captured facial images and evaluating the driver's eye opening / closing status and facial expression" refers to a device or program that inputs the acquired facial image data into an image analysis algorithm to analyze the eye opening / closing status, facial color, changes in facial expression, etc.

[1778] The "means for recognizing the driver's emotional state" is a device or program that includes an emotion engine for identifying the driver's emotional state, such as stress level, fatigue, or irritability, based on data obtained from image analysis.

[1779] The "means for detecting signs of driver drowsiness or inattention" refers to a device or program that uses an algorithm that integrates the results of image analysis with the evaluation of an emotion engine, and evaluates the eye opening / closing status, changes in facial expression, and emotional state within a certain period of time to determine signs of drowsiness or inattention.

[1780] "Means for issuing a warning when signs of drowsiness or inattention are detected" means a device or program that issues a warning to the driver via voice or text message when signs of drowsiness or inattention are detected.

[1781] "Means for storing warning histories and analysis results and allowing administrators to monitor them in real time" refers to a device or program that stores all warning histories and analysis results in a database and provides a dashboard that administrators can access via the Internet.

[1782] This invention is a safe driving support system that combines facial image analysis and emotion recognition to prevent drivers from falling asleep or being careless while driving. This system is composed of the following elements.

[1783] 1. How to capture the driver's face:

[1784] The terminal includes a camera device mounted on the vehicle, which periodically captures the driver's face and transmits the captured image to a server to obtain information such as the eye status and facial expression.

[1785] 2. Means of analyzing the captured facial image:

[1786] The server inputs the received facial image data into an image analysis algorithm, possibly using software such as OpenCV or TensorFlow. The server evaluates the eye status, facial color, and changes in facial expression to analyze the driver's attention level.

[1787] 3. How to recognize driver emotions:

[1788] The server inputs the data obtained from image analysis into an emotion engine to recognize the driver's emotional state. The emotion engine can use AI models such as Microsoft's Azure Emotion API or Google Cloud's Face API. This allows the system to evaluate the driver's stress level, fatigue, irritability, etc.

[1789] 4. Measures to detect signs of drowsiness or inattention:

[1790] The server integrates the image analysis results with the emotion engine's evaluation to detect signs of drowsiness or inattention. Specifically, it uses an algorithm to determine whether the driver is drowsy or inattentive based on conditions such as eyes closed for a certain period of time (e.g., three seconds or more) or a blank expression.

[1791] 5. Means of issuing a warning if signs of drowsiness or inattention are detected:

[1792] If the server detects signs of drowsiness or inattention, it sends a warning instruction to the device. Based on this instruction, the device issues a warning to the driver by voice or text message. The warning content might be something like, "Caution: You may be drowsy at the wheel. Please take a break."

[1793] 6. A way to store all data and allow administrators to monitor it in real time:

[1794] The server stores all warning history and analysis results in a database and provides a dashboard that administrators can access via the internet, allowing them to monitor and manage the status of multiple drivers in real time.

[1795] Specific examples

[1796] For example, consider the case where a driver keeps their eyes closed for three seconds while driving and their facial expression is neutral.

[1797] 1. The user gets into the vehicle and turns on the terminal (camera device).

[1798] 2. The user enters their authentication information and the device sends it to the server.

[1799] 3. The server verifies the authentication information and notifies the terminal that authentication was successful.

[1800] 4. The device captures a facial image every second and sends it to the server.

[1801] 5. The server uses image analysis algorithms to analyze the eye status and facial expressions.

[1802] 6. The server is an emotion engine that evaluates the driver's emotional state (e.g., fatigue, stress).

[1803] 7. The server detects closed eyes for more than 3 seconds and a neutral facial expression, and combines this with the emotion engine's evaluation to recognize signs of drowsiness.

[1804] 8. The server sends a warning instruction to the terminal and generates a message warning of the possibility of drowsiness.

[1805] 9. The device will issue a voice warning saying, "Caution: You may be falling asleep at the wheel. Please take a break."

[1806] 10. The server stores this warning event in a database and allows administrators to view the situation in real time on a dashboard.

[1807] Example prompts for generative AI models

[1808] A detailed description of the system can be automatically generated by using the following prompts to input data into the generative AI model:

[1809] Based on the following information, generate a detailed description of a system that prevents drivers from falling asleep or being distracted while driving:

[1810] The system consists of a camera device, a server, and an emotion engine.

[1811] 1. The vehicle-mounted camera periodically captures the driver's face and sends it to a server.

[1812] 2. The server analyzes the image and evaluates the eye status and facial expression.

[1813] 3. The emotion engine identifies the driver's emotional state based on the analysis results.

[1814] 4. The server detects signs of drowsiness or inattention, such as closed eyes for a period of time or a blank expression.

[1815] 5. The server sends a warning instruction to the device and warns the driver via voice or text message.

[1816] 6. All data is stored on the server and can be viewed on a dashboard where administrators can monitor in real time.

[1817] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1818] Step 1: Initial Setup and Authentication

[1819] The user gets into the vehicle and turns on the terminal (camera device). The terminal checks the connection between the camera device and the network. The user enters authentication information (e.g., ID and password) as input, which the terminal sends to the server.

[1820] The server checks the received authentication information against the database to see if the authentication was successful, and notifies the terminal that the authentication was successful.

[1821] Step 2: Data Acquisition

[1822] After successful authentication, the device periodically (e.g., every second) captures an image of the driver's face using a camera device, and the captured face image data is generated as input.

[1823] The terminal compresses the captured image data and sends it to the server. The output compressed data is sent and reaches the server.

[1824] Step 3: Image analysis

[1825] The server feeds the received facial image data into an image analysis algorithm, which receives the compressed facial image data as input.

[1826] The server uses software such as OpenCV and TensorFlow to analyze eye status, facial color, and facial expression changes, and generates data on eye status and facial expressions as output.

[1827] Step 4: Emotion Recognition

[1828] The server inputs the data obtained from the image analysis into the emotion engine, providing data such as eye opening and closing status and facial expressions.

[1829] The emotion engine uses AI models (e.g., Azure Emotion API, Google Cloud Face API) to identify the driver's emotional state (e.g., stress level, fatigue, irritability). As an output, data about the driver's emotional state is generated.

[1830] Step 5: Hazard detection

[1831] The server integrates data on eye opening and closing and facial expressions with the emotion engine's evaluation. Analysis data and emotional state data are integrated as input.

[1832] The server runs an algorithm to detect signs of drowsiness or inattention. Specifically, it evaluates whether the eyes are closed for a certain period of time (e.g., more than 3 seconds) or whether the face is neutral. As an output, it generates warning data about signs of drowsiness or inattention.

[1833] Step 6: Warning occurs

[1834] The server sends a warning instruction to the terminal when it detects signs of drowsiness or inattention. As input, a warning instruction is generated.

[1835] The device receives the warning instruction and issues a warning via voice or text message. Specifically, it uses a generative AI model to generate a prompt sentence: "Caution: You may be drowsy at the wheel. Please take a break." As an output, the warning message is provided to the driver.

[1836] Step 7: Data storage and management

[1837] The server stores all the warning history and analysis results in a database. As input, the warning data and analysis data are stored in the database.

[1838] The server provides a dashboard that can be accessed by the administrator. Specifically, it builds an interface that allows the administrator to monitor the driver status in real time. As an output, a dashboard that can be accessed by the administrator is created.

[1839] (Application example 2)

[1840] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1841] Improving safety is a key issue for modern self-driving vehicles. Driver drowsiness and inattention can lead to serious accidents, so technology is needed to detect these early and provide appropriate warnings. Furthermore, there is a lack of systems that can make comprehensive judgments by taking into account the driver's emotional state, and this needs to be addressed.

[1842] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing the driver's face, means for analyzing the captured facial image and evaluating the driver's eye opening / closing state and facial expression, means for recognizing the driver's emotional state using an emotion engine, means for detecting signs of the driver's drowsiness or inattention, means for issuing a warning when signs of drowsiness or inattention are detected, means for accumulating all data so that an administrator can monitor it in real time, and means for transmitting data to a cloud server for analysis and issuing a warning instruction. This enables early detection of the driver's drowsiness or inattention and appropriate warning, as well as comprehensive judgment taking the driver's emotional state into consideration.

[1843] "Means for capturing the driver's face" refers to a device or sensor for acquiring an image of the driver's face, and generally refers to a camera device.

[1844] "Means for analyzing the captured facial image and evaluating the driver's eye opening / closing state and facial expression" refers to a device or algorithm that analyzes the driver's eye opening / closing state and changes in facial expression based on the acquired facial image.

[1845] "Means for recognizing the driver's emotional state using an emotion engine" refers to an algorithm or system for assessing the driver's emotional state (e.g., stress level, fatigue, irritability, etc.) based on analyzed facial information.

[1846] The "means for detecting signs of driver drowsiness or inattention" is a device or program for detecting signs of drowsiness or inattention by integrating the analysis results of the driver's eye opening / closing state and facial expressions with the evaluation of the emotion engine.

[1847] A "means for issuing a warning when signs of drowsiness or inattention are detected" is a device or system for issuing a warning to the driver via voice or text message based on detected signs of drowsiness or inattention.

[1848] "A means for storing all data and allowing administrators to monitor in real time" refers to a system that stores warning history and analysis results in a database and provides a dashboard that administrators can access and monitor in real time via the Internet.

[1849] The "means for transmitting data to a cloud server and issuing analysis and warning instructions" refers to a communication device and system for transmitting a driver's facial image and analysis data to a cloud server and issuing a warning instruction based on the analysis results of the cloud server.

[1850] This invention provides a system that combines an emotion engine to prevent drivers from falling asleep or being distracted while driving. This system is composed of the following elements:

[1851] Driver face capture methods:

[1852] A camera device is installed in the vehicle to periodically capture the driver's face, and the camera device is responsible for sending the captured facial images to a cloud server.

[1853] A means of analyzing captured facial images to assess the driver's eye status and facial expression:

[1854] The cloud server inputs the received facial image data into an image analysis algorithm (e.g., TensorFlow, OpenCV), which evaluates the eye status, facial color, and changes in facial expression to analyze the driver's attention level.

[1855] Using an emotion engine to recognize the driver's emotional state:

[1856] The cloud server inputs the data obtained from image analysis into an emotion engine (e.g., Microsoft Azure Emotion API) to recognize the driver's emotional state. The emotion engine evaluates the driver's stress level, fatigue, irritability, etc.

[1857] Ways to detect signs of driver drowsiness or inattention:

[1858] The cloud server combines the analysis results, such as when the eyes are closed for a certain period of time (e.g., more than 3 seconds) or when the face is expressionless, with the emotion engine's evaluation to comprehensively detect signs of drowsiness or inattention.

[1859] To alert you if signs of drowsiness or inattention are detected:

[1860] If the cloud server detects signs of drowsiness or inattention, it sends a warning instruction to the device, which then issues a voice or text message to the driver.

[1861] A way to store all data and allow administrators to monitor it in real time:

[1862] The cloud server stores all warning history and analysis results in a database and provides a dashboard that administrators can access via the internet, allowing them to monitor and manage the status of multiple drivers in real time.

[1863] Program processing:

[1864] The system periodically captures various data and sends it to a cloud server. The cloud server then runs the received data through an image analysis algorithm to evaluate eye status and facial expressions. It then uses an emotion engine to recognize the driver's emotional state and integrates the analysis results to detect signs of drowsiness or inattention. If signs are detected, the server sends a warning instruction to the device, which then issues a voice or text warning. Finally, all data is stored in a database, allowing administrators to monitor it in real time.

[1865] Examples:

[1866] Consider a case where a driver becomes tired while driving and closes their eyes for more than three seconds. At this time, a camera device captures the driver's face and sends the data to a cloud server. The server uses an image analysis algorithm to evaluate whether the eyes are open or closed, and an emotion engine to recognize the driver's fatigue state. The cloud server integrates these analysis results, detects signs of drowsiness, and sends a warning instruction to the device. The device issues an audio warning saying, "Caution: You may be drowsy while driving. Please take a break." This series of data is stored in a cloud database, allowing administrators to monitor it in real time.

[1867] Example prompt sentence:

[1868] "Images of the driver's face are captured every second and sent to a cloud server. The server uses image analysis algorithms and an emotion engine to evaluate eye status and facial expressions. If signs of drowsy driving or inattention are detected, the cloud server sends warning instructions to the device and warns via voice and text messages. All data is stored in a cloud database and can be monitored in real time by administrators."

[1869] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1870] Step 1:

[1871] The device is started in the vehicle and the user enters authentication information. The authentication information (e.g., facial recognition data or password) is sent from the device to the cloud server. The server verifies the authentication information and notifies the device that authentication was successful.

[1872] Input: User credentials

[1873] Output: Authentication success notification

[1874] Specific operation: When a user enters a vehicle, the camera device performs facial recognition. The authentication information is sent to the cloud server, and if the authentication is successful, the device is notified.

[1875] Step 2:

[1876] The device periodically (e.g., every second) captures the driver's facial image using a camera device, and the captured facial image is sent to a cloud server.

[1877] Input: Driver's face image

[1878] Output: Sending facial image data to the cloud server

[1879] Specific operation: The device's camera takes a picture of the driver's face every second and sends the image data to a cloud server.

[1880] Step 3:

[1881] The server then inputs the received facial images into an image analysis algorithm (e.g., TensorFlow, OpenCV), which processes the images to evaluate eye status and facial expressions.

[1882] Input: Facial image data

[1883] Output: Eye open / close status and facial expression analysis results

[1884] How it works: The cloud server uses image analysis algorithms to evaluate eye status and facial expressions.

[1885] Step 4:

[1886] The server inputs the data into an emotion engine (e.g., Microsoft Azure Emotion API) based on the image analysis results and evaluates the driver's emotional state (stress level, fatigue level, etc.).

[1887] Input: Image analysis results

[1888] Output: Emotional state evaluation result

[1889] Specific operation: The cloud server inputs the image analysis results into the emotion engine to evaluate the driver's stress level and fatigue state.

[1890] Step 5:

[1891] The server combines the analysis results, such as when the eyes are closed for a certain period of time (for example, more than three seconds) or when the face is expressionless, with the emotion engine's evaluation to detect signs of drowsiness or inattention.

[1892] Input: Eye status and emotional state evaluation results

[1893] Output: Detected signs of drowsiness and inattention

[1894] Specific operation: The cloud server integrates the evaluation results of eye opening / closing status and emotional state to detect signs of drowsiness or inattention.

[1895] Step 6:

[1896] If the server detects signs of drowsiness or inattention, it sends warning instructions to the device, which then issues voice and text message warnings based on the instructions.

[1897] Input: Result of detecting signs of drowsiness and inattention

[1898] Output: Warning instructions and warning messages

[1899] Specific operation: The cloud server detects signs of drowsiness and sends a warning instruction to the device. The device then issues a warning saying, "Caution: You may be drowsy at the wheel. Please take a break."

[1900] Step 7:

[1901] The server stores all warning history and analysis results in a database and provides an internet-accessible dashboard for administrators, allowing them to monitor and manage the status of multiple drivers in real time.

[1902] Input: Warning history and analysis results

[1903] Output: Save to database and reflect on dashboard

[1904] Specific operation: The cloud server stores warning events and analysis results in a database, and administrators can monitor them in real time through a dashboard.

[1905] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1906] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1907] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1908] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1909] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1910] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1911] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1912] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1913] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1914] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1915] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1916] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1917] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1918] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1919] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1920] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1921] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1922] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1923] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1924] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1925] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1926] The following is further disclosed regarding the above embodiment.

[1927] (Claim 1)

[1928] a means for capturing the face of the driver;

[1929] A means for analyzing the captured facial image to evaluate the driver's eye opening / closing state and facial expression;

[1930] a means of detecting signs of driver drowsiness or inattention;

[1931] a means of issuing an alert if signs of drowsiness or inattention are detected;

[1932] A means to store all this data and allow administrators to monitor it in real time,

[1933] A system including:

[1934] (Claim 2)

[1935] 10. The system of claim 1, wherein the means for capturing the driver's face is a camera device mounted on the vehicle.

[1936] (Claim 3)

[1937] 2. The system according to claim 1, wherein the means for detecting signs of driver drowsiness or inattention uses an algorithm that detects when the driver's eyes are continuously opened or closed within a certain period of time.

[1938] (Claim 4)

[1939] 10. The system of claim 1, wherein the means for issuing the alert uses voice and text messages.

[1940] (Claim 5)

[1941] 2. The system according to claim 1, wherein the means for storing the data and allowing an administrator to monitor it in real time includes a dashboard that allows the administrator to monitor the driver's status via the Internet.

[1942] "Example 1"

[1943] (Claim 1)

[1944] a means for capturing the face of the driver;

[1945] A means for analyzing the captured facial image to evaluate the driver's eye opening / closing state and facial expression;

[1946] a means of detecting signs of driver drowsiness or inattention;

[1947] a means of issuing an alert if signs of drowsiness or inattention are detected;

[1948] All warning history and analysis results are stored in a database, allowing administrators to monitor in real time via the Internet.

[1949] A system including:

[1950] (Claim 2)

[1951] 10. The system of claim 1, wherein the means for capturing the driver's face is a camera device mounted on the vehicle.

[1952] (Claim 3)

[1953] 2. The system according to claim 1, wherein the means for detecting signs of driver drowsiness or inattention uses an algorithm that detects when the driver's eyes are continuously opened or closed within a certain period of time.

[1954] "Application Example 1"

[1955] (Claim 1)

[1956] a means for capturing the face of the driver;

[1957] A means for analyzing the captured facial image to evaluate the driver's eye opening / closing state and facial expression;

[1958] a means of detecting signs of driver drowsiness or inattention;

[1959] a means of issuing an alert if signs of drowsiness or inattention are detected;

[1960] A means to store all this data and allow administrators to monitor it in real time,

[1961] a means for authenticating a driver;

[1962] means for automatically reducing the speed of the vehicle if the warning goes unanswered;

[1963] A system including:

[1964] (Claim 2)

[1965] 10. The system of claim 1, w...

Claims

1. a means for capturing the face of the driver; A means for analyzing the captured facial image to evaluate the driver's eye opening / closing state and facial expression; a means of detecting signs of driver drowsiness or inattention; a means of issuing an alert if signs of drowsiness or inattention are detected; A means to store all this data and allow administrators to monitor it in real time, A system including:

2. The system of claim 1 , wherein the means for capturing the driver's face is a camera device mounted on the vehicle.

3. 2. The system according to claim 1, wherein the means for detecting signs of driver drowsiness or inattention uses an algorithm for detecting when the driver's eyes are continuously opened or closed within a certain period of time.

4. 10. The system of claim 1, wherein said means for issuing an alert uses voice and text messages.

5. 2. The system according to claim 1, wherein the means for storing the data and enabling a manager to monitor the data in real time includes a dashboard that enables the manager to monitor the driver's status via the Internet.

Images