system

Abstract

We provide the system. [Solution] A means for detecting driver intoxication or abnormal behavior by analyzing video data acquired in real time by an image acquisition device installed inside the vehicle, The means for stopping the vehicle's power source and switching to a semi-autonomous driving mode when an abnormality is detected by the means described above, After detecting the aforementioned anomaly, the vehicle is stopped in a safe location, and the anomaly is automatically notified to the relevant authorities. Before starting the vehicle, the system verifies the driver's qualifications and provides a means to prevent the vehicle from starting if the information is inappropriate or invalid. A means of monitoring the driver's behavior while driving and providing warnings or instructions for improvement if dangerous driving is detected, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In recent years, among traffic accidents caused by automobiles, accidents due to drunk driving or inappropriate license status of drivers still remain a social problem, and the establishment of means to prevent these is required. Also, incorrect driving behaviors such as mistakes or reverse driving by elderly drivers are increasing. To reduce such accidents, the introduction of a system related to driver state monitoring and prevention of unlicensed driving is essential.

Means for Solving the Problems

[0005] This invention provides a system that detects signs of drunk driving and abnormal behavior by acquiring video footage of the driver using an image acquisition device installed in the vehicle and analyzing it in real time using a generating AI. When an abnormality is detected, the system shuts off the vehicle's power source, stops the vehicle in a safe location in semi-autonomous driving mode, and automatically notifies the relevant authorities. It also has a function to check the driver's license information before driving begins and prevents the vehicle from starting if it is inappropriate. Furthermore, it continuously monitors the driver's behavior while driving and provides warnings or corrective instructions when danger is recognized, thereby aiming to prevent traffic accidents.

[0006] An "image acquisition device" is a device installed inside a vehicle that captures video of the driver in real time.

[0007] "Generative AI" refers to artificial intelligence technology that analyzes acquired video data to detect signs of driver intoxication or abnormal behavior.

[0008] "Drunk driving" refers to the act of operating a vehicle while under the influence of alcohol, and is prohibited by law.

[0009] "Abnormal behavior" refers to actions taken by a driver that deviate from normal driving procedures and may impair driving ability.

[0010] "To shut down the power source" means to stop the vehicle's power supply device, such as the engine or motor.

[0011] "Semi-autonomous driving mode" refers to a function that allows a vehicle to operate autonomously with minimal driver intervention and move to a safe location.

[0012] "Relevant organizations" typically refer to organizations that respond to vehicle malfunctions, such as the police or traffic management organizations.

[0013] "License information" refers to data used to prove a driver's qualifications to drive and is used to verify the validity of the license.

[0014] "Dangerous driving" refers to any action or operation by a driver that threatens traffic safety and has the potential to cause an accident.

[0015] A "warning or corrective action" is when the system provides a message to the driver to alert them to a recognized dangerous behavior and encourage them to correct it. [Brief explanation of the drawing]

[0016] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Embodiments for Carrying Out the Invention

[0017] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0018] First, the language used in the following description will be explained.

[0019] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), etc.

[0020] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0021] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disk (e.g., hard disk), or magnetic tape, etc.

[0022] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0024] [First Embodiment]

[0025] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0026] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0029] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0032] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0036] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0037] This invention provides a detailed system for detecting and responding to drunk driving and other abnormal behavior by drivers. The system mainly consists of various devices and a server installed inside the vehicle. An example of its operation is described below.

[0038] In an embodiment of the present invention, the terminal first captures video data in real time from a camera installed inside the vehicle. Then, it transmits this video data to a server for real-time analysis. The server uses generated AI to analyze the driver's facial expression, eye movements, and other behavioral characteristics through this video data to detect signs of drinking or abnormal behavior.

[0039] If an anomaly is detected, the server sends a command to the terminal to shut down the engine and activates semi-autonomous driving mode to safely operate the vehicle. At the same time, the server automatically notifies relevant organizations of the anomaly to facilitate a rapid response.

[0040] For license verification, the user scans their driver's license at the terminal before starting to drive, and its validity is verified by the server. If the license is found to be expired or invalid, the terminal will not allow the engine to start.

[0041] The server also continuously monitors the driver's behavior while driving, and if dangerous driving is detected, the terminal provides real-time warnings and corrective instructions. For example, in a real-world case, if lane departure is detected, the terminal will provide an audio warning to the user saying, "Please stay in your lane." This warning encourages the driver to immediately return to safe driving.

[0042] By implementing this system, we can provide a new vehicle driving support environment that can suppress dangerous behaviors such as drunk driving and prevent traffic accidents.

[0043] The following describes the processing flow.

[0044] Step 1:

[0045] The device acquires video data from the in-car camera in real time and sends that data to the server.

[0046] Step 2:

[0047] To analyze the video data received by the server, a generating AI is called in to analyze the driver's facial expression, eye condition, and other behavioral patterns.

[0048] Step 3:

[0049] Based on the data analyzed by the generated AI, the server determines whether the driver is showing signs of intoxication or exhibiting abnormal behavior.

[0050] Step 4:

[0051] If an abnormality is detected, the server sends an engine stop command and a command to switch to semi-autonomous driving mode to the terminal.

[0052] Step 5:

[0053] The terminal, following the received commands, shuts off the vehicle's power source, activates semi-autonomous driving mode, and stops in a safe location.

[0054] Step 6:

[0055] The server automatically notifies the relevant authorities when it detects an anomaly and sends a message containing details such as the vehicle's location.

[0056] Step 7:

[0057] Before the user starts driving, they scan their driver's license at a terminal, the server receives the information, and verifies the validity of the license.

[0058] Step 8:

[0059] If the server detects that the license has expired or is invalid, it sends an engine start prohibition command to the terminal. The terminal executes this command and prevents the engine from starting.

[0060] Step 9:

[0061] While driving, the server continuously monitors the driver's behavior and, if it determines that dangerous driving is occurring, provides the user with warnings and instructions for improvement via the terminal.

[0062] (Example 1)

[0063] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0064] The challenge lies in preventing traffic accidents caused by drunk driving or abnormal behavior by drivers, and ensuring the safety of drivers and those around them. It is also necessary to prevent accidents and problems caused by drivers operating vehicles without verifying their own driving qualifications. Furthermore, it is crucial to monitor dangerous driving in real time and provide appropriate guidance as needed.

[0065] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0066] In this invention, the server includes means for analyzing video data acquired in real time by a camera installed in the vehicle to detect drunk driving or abnormal behavior by the driver; means for analyzing the driver's biological characteristics by analyzing image data using a generative AI model; and means for transmitting signals to a terminal to gradually decelerate and safely stop the vehicle. This makes it possible to quickly detect drunk driving or abnormal behavior and provide a safe driving environment.

[0067] A "camera" is a device installed inside a vehicle to acquire video data of the interior, including the driver.

[0068] "Real-time analysis" is a process that processes acquired data immediately to derive some kind of judgment or result.

[0069] A "propulsion system" is a system that supplies and controls power, including various devices necessary to move a vehicle forward.

[0070] "Semi-autonomous driving mode" is a driving mode that automatically controls the basic driving operations of the vehicle while minimizing driver intervention.

[0071] "Related facilities" refers to transportation and safety organizations that receive and respond to information related to traffic management and safety management.

[0072] "Authentication information" refers to information that can be used to verify a driver's qualifications and licenses.

[0073] A "generative AI model" is a model that uses artificial intelligence algorithms to analyze data and recognize its features.

[0074] "Biological characteristics" refer to information that describes individual physical features of the driver, such as their facial complexion and eye movements.

[0075] This invention provides a system to prevent drunk driving and abnormal behavior by drivers, thereby supporting safe driving. The system mainly consists of a terminal installed inside the vehicle and a server that performs advanced analysis.

[0076] The terminal uses a camera installed inside the vehicle to capture video data of the driver in real time. Specifically, high-resolution camera equipment is used, and this video data is continuously transmitted to a server. Wireless technologies such as Bluetooth and Wi-Fi are used for communication.

[0077] The server analyzes the received video data and uses a generated AI model to evaluate the driver's biometric characteristics. The server utilizes high-performance hardware such as NVIDIA GPUs and Intel processors. This analysis uses the prompt message "Detect signs of alcohol intoxication from the driver's facial expression and eye movements" to detect signs of alcohol intoxication or abnormal behavior. Based on the analysis results, if an abnormality is detected, the system instructs the driver to stop the propulsion system and activates semi-autonomous driving mode.

[0078] As an example, consider a situation where a driver feels drowsy while driving and their eyes begin to close. The system instantly detects the eye movement and warns the driver in real time via the device, "Please keep your eyes open." This warning allows the driver to quickly regain their focus and return their attention to the road.

[0079] Furthermore, before starting the vehicle, the user enters their authentication information into the terminal, and the server verifies the validity of that information. If the authentication information is valid, the terminal immediately allows the engine to start; however, if it is invalid, it notifies the user that "authentication could not be verified" and restricts the vehicle's operation.

[0080] Thus, the system of the present invention can comprehensively support driver safety through real-time data analysis utilizing a generative AI model.

[0081] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0082] Step 1:

[0083] The terminal activates a camera installed inside the vehicle and captures video data of the driver. The input at this time is a real-time video signal, which is converted into digital data frame by frame. As output, the acquired video data is sent to the server. Specifically, the terminal transfers the compressed video data to the server via Bluetooth or Wi-Fi.

[0084] Step 2:

[0085] The server takes the received video data as input and performs analysis using a generating AI model. During this process, it measures biometric information such as eye movements and facial color, and applies the prompt message, "Detect signs of intoxication from the driver's facial color and eye movements." For data processing, specific features are extracted from the video frames and classified and evaluated by the AI ​​model. The output is the analysis results, such as signs of intoxication or abnormal behavior.

[0086] Step 3:

[0087] If abnormal behavior is detected, the server generates a command to shut down the propulsion system and outputs it to the terminal. Specifically, it creates a signal that instructs the optimal deceleration and stopping process, taking into account the vehicle's current speed and location information. When this signal is received by the terminal, the stepwise deceleration procedure begins.

[0088] Step 4:

[0089] Based on instructions from the server, the terminal directly controls the propulsion system to switch the vehicle into semi-autonomous driving mode. Its input is the server's control instructions, and its output is a vehicle speed control signal. Specifically, the control unit is activated, adjusting the engine output and gradually bringing the vehicle to a safe stop.

[0090] Step 5:

[0091] After the vehicle stops, the server notifies the relevant facilities of the abnormal situation. The input at this time is the detected abnormal behavior and related information, and the output is an abnormal situation notification message which is sent through the appropriate communication channel. Specifically, data including location information and driver ID is automatically transmitted.

[0092] Step 6:

[0093] When restarting operation, the user has the terminal read their authentication information before starting the vehicle. The input is either a physical authentication card or digital information, which the terminal reads and sends to the server. The server receives this data and verifies its validity. The output is "Engine ready to start" if valid, or "Authentication failed" if invalid.

[0094] (Application Example 1)

[0095] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0096] In recent years, traffic accidents caused by drunk driving or abnormal behavior by drivers have become a social problem. Conventional technology has often been insufficient to detect abnormal behavior in advance and prevent accidents. Furthermore, verifying driver credentials before starting to drive is not always safe. In addition, there is a lack of means to continuously monitor the driver's condition while driving and issue appropriate warnings in real time. Therefore, further technologies are needed to reduce the risk of traffic accidents.

[0097] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0098] In this invention, the server includes means for analyzing video data acquired in real time by an image acquisition device installed in the vehicle to detect driver intoxication or abnormal behavior; means for shutting off the vehicle's power source and switching to a semi-autonomous driving mode when an abnormality is detected; and means for continuously monitoring the driver's condition while driving using a smart device equipped with voice and visual warning functions that detect the driver's facial expression and eye movements and issue warnings. This makes it possible to instantly detect abnormal driver behavior and prevent accidents.

[0099] An "image acquisition device" is a device installed inside a vehicle that acquires video data such as the driver's facial expression and eye movements in real time.

[0100] "Methods for real-time analysis" refer to methods for immediately processing acquired video data to detect driver intoxication or abnormal behavior.

[0101] "Methods for detecting abnormal behavior" refer to methods that use generative AI to analyze the driver's biometric information and detect signs such as alcohol consumption or fatigue.

[0102] "Means for stopping the power source" refers to means that, when an abnormality is detected in the driver, the vehicle's power mechanism can be controlled to stop the engine.

[0103] "Means of switching to semi-autonomous driving mode" refers to means of activating driver assistance functions that automatically and safely control the vehicle.

[0104] "Means of verifying qualifications" refers to the means of checking the driver's license or other identification before driving begins and evaluating its validity.

[0105] A "smart device" is a device equipped with a camera and microphone that provides information to the driver by issuing voice and visual warnings.

[0106] To implement this invention, first, an image acquisition device installed inside the vehicle is used to capture biometric information such as the driver's facial color and eye movements in real time. The image acquisition device is installed as a smart device and collects the driver's biometric information. The terminal transmits this video data to a server, where it is analyzed using a generated AI model.

[0107] The server uses machine learning libraries such as TENSORFLOW® to analyze this video data and detect abnormal driver behavior (e.g., facial redness, unnatural eye movements). If an abnormality is detected, the server sends an instruction to the terminal to shut off the vehicle's power source, and the vehicle automatically switches to semi-autonomous driving mode.

[0108] Subsequently, the driver receives audio and visual warnings. For example, warnings such as "Please stop driving" or "Your face is red. Please check for alcohol intoxication" are transmitted through the smart device's speaker and display.

[0109] Through this process, the device continuously monitors the driver's facial expression and eye movements, and takes appropriate safety measures as needed based on instructions from the server. This significantly reduces the risk of traffic accidents.

[0110] An example of a prompt statement is, "Generate a risk prediction algorithm to detect abnormal driving behavior." This prompt statement is used to prompt the generated AI model to perform appropriate analysis.

[0111] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0112] Step 1:

[0113] The terminal captures biometric information, such as the driver's face and eye movements, in real time from an image acquisition device installed inside the vehicle. The input is video data, and the output is the captured video data itself. Specifically, the terminal periodically activates the image acquisition device and takes pictures of the driver.

[0114] Step 2:

[0115] The terminal transfers the acquired video data to the server. The input is the video data captured in step 1, and the output is the transmission of data to the server. Specifically, the terminal uploads the data to the server via the network.

[0116] Step 3:

[0117] The server analyzes the received video data using a generation AI model. The input is the transmitted video data, and the output is the result of the driver's abnormal behavior detection. Specifically, the server uses machine learning libraries such as TensorFlow to detect facial expression, eye movements, and unnatural behavior based on prompt messages.

[0118] Step 4:

[0119] If abnormal behavior is detected, the server sends an instruction to the terminal to issue a warning. The input is the abnormal behavior detection result obtained in step 3, and the output is the warning instruction. Specifically, the server sets up audio and visual warnings and instructs the terminal to execute them.

[0120] Step 5:

[0121] The terminal, based on instructions from the server, issues warnings to the driver via a smart device. The input is a warning instruction from the server, and the output is an audio and visual warning to the driver. Specifically, the terminal plays messages such as "Stop driving" or "Your face is red. Please check for alcohol intoxication."

[0122] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0123] The system of the present invention improves driver safety by combining an image acquisition device, a generating AI, and an emotion engine installed in a vehicle. The system is implemented in the following manner.

[0124] First, the device acquires video data in real time using cameras inside the vehicle. The acquired data is sent to a server and analyzed by a generative AI. The generative AI analyzes the driver's facial expression, eye movements, and behavior to quickly determine if there is any alcohol consumption or other abnormal behavior. It also uses an emotion engine to analyze the driver's emotional state—for example, stress or distraction.

[0125] If signs of abnormality or emotional instability are detected, the server sends an engine shutdown command to the terminal and switches to semi-autonomous driving mode, bringing the vehicle to a safe location. During this time, the server notifies the appropriate relevant authorities of the situation.

[0126] Furthermore, the system prevents inappropriate driving by requiring the user to present their driver's license before starting to drive, and verifying its validity on the server. Once driving begins, the server uses an emotion engine to provide the user with relaxation methods and warnings tailored to the driver's emotional state, via the terminal.

[0127] For example, when the emotional engine detects a high stress level while driving, the server sends a command to the terminal to change the in-car music to something calming or adjust the air conditioning to a comfortable setting. This adjustment helps to alleviate the driver's tension and provide a safer driving environment.

[0128] Thus, by comprehensively assessing the driver's condition, this system can significantly contribute to preventing traffic accidents.

[0129] The following describes the processing flow.

[0130] Step 1:

[0131] The terminal acquires video data of the driver in real time via the vehicle's camera and sends it to a server for processing.

[0132] Step 2:

[0133] To analyze the video data received by the server, a generating AI is used to analyze the driver's facial expression, eye movements, and other behaviors to determine whether or not they are intoxicated or exhibiting abnormal behavior.

[0134] Step 3:

[0135] Simultaneously, the server uses an emotion engine to analyze the driver's emotional state and detect emotional changes such as stress levels and attention deficits.

[0136] Step 4:

[0137] If abnormal behavior or unstable emotions are detected, the server sends a command to the terminal to shut off the engine and switch to semi-autonomous driving mode. Upon receiving this command, the terminal shuts off the vehicle's power source and moves the vehicle to a safe location.

[0138] Step 5:

[0139] After the server detects an anomaly, it automatically notifies the relevant organizations about the vehicle's current location and the driver's status.

[0140] Step 6:

[0141] Before starting the vehicle, the user presents their driver's license to the terminal, and the server verifies the validity of the license by checking the data. If the license is invalid, the terminal prevents the engine from starting.

[0142] Step 7:

[0143] The server uses an emotion engine to monitor the driver's emotions while the vehicle is in operation and provides optimal driving support based on their emotional state. This includes allowing the driver to change music selections and adjust air conditioning settings via their terminal.

[0144] Step 8:

[0145] The server generates music to help the driver relax and warning messages when necessary, and provides them to the user via the terminal.

[0146] This series of processes ensures a safe and smooth operating environment.

[0147] (Example 2)

[0148] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0149] One of the main causes of traffic accidents is driver inattention and unstable emotional states while driving. The challenge lies in preventing these dangers in advance by detecting these factors in real time. Furthermore, it is necessary to prevent unqualified drivers from operating vehicles and ensure driver safety.

[0150] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0151] This invention includes a server that analyzes video data acquired in real time by an image acquisition device installed in the vehicle to detect driver intoxication or abnormal behavior; an emotion analysis device that evaluates the driver's emotional state and provides relaxation methods or warnings according to the emotional state, such as stress; and a means that verifies the driver's credentials before driving begins and disables the vehicle if they are inappropriate or invalid. This makes it possible to monitor the driver's condition from multiple angles, avoid dangerous situations in advance, and ensure safety.

[0152] An "image acquisition device installed inside a vehicle" refers to a device installed inside a vehicle that includes cameras and sensors for acquiring video data of the driver.

[0153] "Generative AI" is an artificial intelligence technology used to analyze acquired video data and identify the driver's biometric information and behavior.

[0154] An "emotion analysis device" is a device that evaluates a driver's emotional state based on their facial expressions and biometric data, and determines their stress level, distractibility, and other factors.

[0155] "Semi-autonomous driving mode" is a driving mode in which the vehicle automatically controls itself to support safe driving.

[0156] "Qualification information" refers to the driver's license and other authentication information necessary for the driver to operate the vehicle.

[0157] "Drive system" refers to the collective term for equipment such as engines and motors necessary to propel a vehicle.

[0158] "Biometric information" refers to information about an individual's physical condition, such as the driver's complexion, eye condition, and behavior.

[0159] "Relaxation techniques" include methods such as changing the music or adjusting the air conditioning to promote safe driving in accordance with the driver's emotional state.

[0160] This invention is a system that improves driver safety by coordinating an image acquisition device, a generating AI, and an emotion analysis device installed inside a vehicle.

[0161] The server receives video data in real time from an image acquisition device installed inside the vehicle. This image acquisition device includes sensors such as cameras and has the function of acquiring the driver's facial color, eye movements, and behavior. The transmitted data is analyzed on the server, and the driver's biometric information is determined by a generating AI.

[0162] The generating AI analyzes video data to recognize the driver's drunk driving and other abnormal behavior. Furthermore, an emotion analysis device assesses the driver's emotional state, particularly detecting stress and distraction. This analysis utilizes advanced facial recognition technology and emotion analysis algorithms.

[0163] If an anomaly or danger is detected, the server sends an engine stop command to the terminal and switches to semi-autonomous driving mode. This allows the vehicle to automatically stop in a safe location. Details of the anomaly are then notified to the relevant organizations.

[0164] Before starting to drive, the user presents their driver's license as credentials, and its validity is verified by the server. This procedure makes it possible to prevent driving with inappropriate or invalid credentials.

[0165] While driving, the server provides relaxation methods tailored to the driver's emotional state based on data obtained from an emotion analysis device. This includes actions such as changing the music in the car or optimizing the air conditioning.

[0166] As a concrete example, if the emotion analysis device detects a high stress level in the driver, the server sends a command to the terminal to change the in-car background music to calming music. An example of the prompt message in this case is: "Analyze the driver's facial expression, eye movements, and behavior in real time, and notify if there are any abnormalities. In addition, use the emotion analysis device to evaluate the stress level and suggest adjustments."

[0167] This system aims to provide a safer driving environment and eliminate potential accident-causing factors early on by monitoring the driver's condition from multiple perspectives.

[0168] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0169] Step 1:

[0170] The terminal acquires real-time video data of the driver using cameras installed inside the vehicle. The input is video data from the in-vehicle cameras, which includes the driver's face, eye movements, and body movements. The output is the acquired raw video data. At this time, a portion of the acquired data is recorded as metadata and tagged for subsequent analysis processing.

[0171] Step 2:

[0172] The device sends the acquired video data to the server. The input is the video data acquired and tagged in step 1, and the output is the transmission of that data to the server. During the transmission process, the data is encrypted to protect privacy.

[0173] Step 3:

[0174] The server analyzes the received video data. The input is encrypted video data transmitted from the terminal. The generating AI uses facial recognition and motion analysis technology to analyze the driver's complexion, eye movements, and behavior. The output is the detection result of signs of intoxication or abnormal behavior in the driver. The analysis results are stored in a database and used for subsequent processing.

[0175] Step 4:

[0176] The server uses an emotion analysis device to evaluate the driver's emotional state. The input is the analysis results obtained in step 3. The emotion analysis device evaluates stress levels and distractibility based on the driver's facial expressions and biometric data. The output is the evaluation result of the emotional state, providing information indicating the need for relaxation or warning.

[0177] Step 5:

[0178] If an abnormality or emotional instability is detected, the server sends an engine stop command to the terminal. The input is the analysis and evaluation results from steps 3 and 4. The output is a command to switch to semi-autonomous driving mode. Specifically, a program to guide the vehicle to the optimal stopping point is activated.

[0179] Step 6:

[0180] Before starting the vehicle, the user presents their credentials to the terminal. The input is the credentials, such as a driver's license, presented by the user. The output is the result of the credentials being verified by the server. In this verification process, if no valid credentials are found, the server locks the vehicle from starting.

[0181] Step 7:

[0182] While driving, the server provides relaxation methods tailored to the driver's state based on information from an emotion analysis device. The input is the real-time evaluation of the driver's emotional state. The output is a command, such as changing music selection or adjusting the air conditioning settings. Specific actions include accessing a music library based on the driver's preferences and automatically adjusting the air conditioning system. Through this process, the system aims to alleviate driver stress and ensure a safe driving environment.

[0183] (Application Example 2)

[0184] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0185] Conventional vehicle safety systems focus on detecting driver intoxication and abnormal behavior, but are insufficient in addressing potential hazards based on the driver's emotional state. Furthermore, if a driver's emotions become unstable while driving, there is no means to appropriately adjust the environment, leaving the risk of accidents unresolved. Moreover, even if a driver is stressed or distracted, there is no system to quickly detect and appropriately address these conditions, resulting in a situation that cannot adequately support safe driving.

[0186] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0187] In this invention, the server includes means for analyzing video data acquired in real time by an image acquisition device installed in the vehicle to detect driver intoxication or abnormal behavior; means for shutting off the vehicle's power source and switching to a semi-autonomous driving mode when an abnormality is detected by the means; and means for analyzing the driver's emotional state, using an emotion engine to detect stress or distraction, and adjusting the driving environment based on this. This makes it possible to immediately detect potential dangers based on the driver's emotional state and support safe and comfortable driving.

[0188] An "image acquisition device" is a device installed inside a vehicle that acquires video data of the driver in real time.

[0189] "Generative AI" is an artificial intelligence technology that analyzes a driver's facial expression, eye movements, and behavior based on the driver's biometric information.

[0190] The "emotional engine" is a program that analyzes the driver's emotional state to detect stress and distraction.

[0191] "Semi-autonomous driving mode" is a driving mode that assists the driver's operation when an abnormality is detected and helps the vehicle stop in a safe location.

[0192] "Qualification information" refers to the licenses and other authorizations required for a driver to operate a vehicle.

[0193] "Dangerous driving" refers to a situation where a driver's actions pose a level of risk far beyond what is considered normal driving, and detecting such situations is crucial.

[0194] "Adjusting the driving environment" refers to the act of changing the in-car sound system and air conditioning according to the driver's emotional state, thereby optimizing the driver's condition.

[0195] To implement this invention, an image acquisition device is installed inside the vehicle, and a system is configured to capture video data of the driver in real time. The server analyzes the acquired video data using various software and evaluates the driver's condition. Face recognition is performed using OpenCV for the analysis, and a generated AI using TensorFlow analyzes biometric information such as the driver's facial color, eye movements, and behavior.

[0196] The server further utilizes an emotion engine to analyze the driver's emotional state, such as stress and distraction. If abnormal behavior or an unstable emotional state is detected, it switches to a semi-autonomous driving mode in conjunction with the vehicle's control system and brings the vehicle to a safe stop. In addition, the server can adjust the in-car sound and air conditioning settings according to the user's emotional state. At this time, it also provides the user with suggestions for relaxation methods and behavioral improvements.

[0197] For example, if the emotion engine determines that the user's stress level is rising during a long drive, the server will change the in-car music to relaxing classical music and automatically adjust the air conditioning to a comfortable temperature. It will also send a notification to the user's device suggesting they take a short break. The prompt sent to the AI ​​model in this case would be something like, "The driver's eyes are closed for longer than usual. Are you experiencing increased tension or stress?"

[0198] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0199] Step 1:

[0200] The terminal captures video data of the driver in real time using an image acquisition device inside the vehicle. The input is video of the driver inside the vehicle, which is obtained as output data in image format.

[0201] Step 2:

[0202] The server receives video data transmitted from the terminal and performs face recognition using OpenCV. The input data is video data of the driver, which is analyzed to identify the driver's face region. The output is the recognized face image and its coordinate information.

[0203] Step 3:

[0204] The server inputs the facial region data recognized by face recognition into a generative AI model using TensorFlow, which analyzes the driver's complexion, eye movements, and behavior. The input is facial region data, and the output is an evaluation of the driver's state based on the analyzed biometric information.

[0205] Step 4:

[0206] The server uses an emotion engine to assess the driver's emotional state, specifically their stress level and degree of distraction. The input is the driver's state assessment result from a generative AI model, and the output is an index indicating the emotional state.

[0207] Step 5:

[0208] If an emotional state or abnormality is detected, the server sends a signal to the vehicle's control system and switches to semi-autonomous driving mode. The input is the result of the abnormality detection, and the output is the control signal to the vehicle's control system.

[0209] Step 6:

[0210] The server sends signals to the terminal to adjust the in-car sound and air conditioning based on the user's emotional state. The input is an indicator of the emotion engine, and the output is a signal instructing adjustments.

[0211] Step 7:

[0212] The terminal changes the sound and air conditioning settings inside the vehicle based on instructions received from the server. The input here is the adjustment instructions from the server, and the output is the specific action to be performed, such as switching music or adjusting the temperature settings.

[0213] Step 8:

[0214] The server sends notifications to the user regarding detected anomalies and emotional states. It alerts the user by sending prompts such as "We recommend you take a short break." The input is the result of the emotion engine's analysis, and the output is the notification content sent to the user.

[0215] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0216] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0217] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0218] [Second Embodiment]

[0219] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0220] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0221] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0222] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0223] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0224] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0225] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0226] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0227] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0228] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0229] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0230] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0231] This invention provides a detailed system for detecting and responding to drunk driving and other abnormal behavior by drivers. The system mainly consists of various devices and a server installed inside the vehicle. An example of its operation is described below.

[0232] In an embodiment of the present invention, the terminal first captures video data in real time from a camera installed inside the vehicle. Then, it transmits this video data to a server for real-time analysis. The server uses generated AI to analyze the driver's facial expression, eye movements, and other behavioral characteristics through this video data to detect signs of drinking or abnormal behavior.

[0233] If an anomaly is detected, the server sends a command to the terminal to shut down the engine and activates semi-autonomous driving mode to safely operate the vehicle. At the same time, the server automatically notifies relevant organizations of the anomaly to facilitate a rapid response.

[0234] For license verification, the user scans their driver's license at the terminal before starting to drive, and its validity is verified by the server. If the license is found to be expired or invalid, the terminal will not allow the engine to start.

[0235] The server also continuously monitors the driver's behavior while driving, and if dangerous driving is detected, the terminal provides real-time warnings and corrective instructions. For example, in a real-world case, if lane departure is detected, the terminal will provide an audio warning to the user saying, "Please stay in your lane." This warning encourages the driver to immediately return to safe driving.

[0236] By implementing this system, we can provide a new vehicle driving support environment that can suppress dangerous behaviors such as drunk driving and prevent traffic accidents.

[0237] The following describes the processing flow.

[0238] Step 1:

[0239] The device acquires video data from the in-car camera in real time and sends that data to the server.

[0240] Step 2:

[0241] To analyze the video data received by the server, a generating AI is called in to analyze the driver's facial expression, eye condition, and other behavioral patterns.

[0242] Step 3:

[0243] Based on the data analyzed by the generated AI, the server determines whether the driver is showing signs of intoxication or exhibiting abnormal behavior.

[0244] Step 4:

[0245] If an abnormality is detected, the server sends an engine stop command and a command to switch to semi-autonomous driving mode to the terminal.

[0246] Step 5:

[0247] The terminal, following the received commands, shuts off the vehicle's power source, activates semi-autonomous driving mode, and stops in a safe location.

[0248] Step 6:

[0249] The server automatically notifies the relevant authorities when it detects an anomaly and sends a message containing details such as the vehicle's location.

[0250] Step 7:

[0251] Before the user starts driving, they scan their driver's license at a terminal, the server receives the information, and verifies the validity of the license.

[0252] Step 8:

[0253] If the server detects that the license has expired or is invalid, it sends an engine start prohibition command to the terminal. The terminal executes this command and prevents the engine from starting.

[0254] Step 9:

[0255] While driving, the server continuously monitors the driver's behavior and, if it determines that dangerous driving is occurring, provides the user with warnings and instructions for improvement via the terminal.

[0256] (Example 1)

[0257] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0258] The challenge lies in preventing traffic accidents caused by drunk driving or abnormal behavior by drivers, and ensuring the safety of drivers and those around them. It is also necessary to prevent accidents and problems caused by drivers operating vehicles without verifying their own driving qualifications. Furthermore, it is crucial to monitor dangerous driving in real time and provide appropriate guidance as needed.

[0259] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0260] In this invention, the server includes means for analyzing video data acquired in real time by a camera installed in the vehicle to detect drunk driving or abnormal behavior by the driver; means for analyzing the driver's biological characteristics by analyzing image data using a generative AI model; and means for transmitting signals to a terminal to gradually decelerate and safely stop the vehicle. This makes it possible to quickly detect drunk driving or abnormal behavior and provide a safe driving environment.

[0261] A "camera" is a device installed inside a vehicle to acquire video data of the interior, including the driver.

[0262] "Real-time analysis" is a process that processes acquired data immediately to derive some kind of judgment or result.

[0263] A "propulsion system" is a system that supplies and controls power, including various devices necessary to move a vehicle forward.

[0264] "Semi-autonomous driving mode" is a driving mode that automatically controls the basic driving operations of the vehicle while minimizing driver intervention.

[0265] "Related facilities" refers to transportation and safety organizations that receive and respond to information related to traffic management and safety management.

[0266] "Authentication information" refers to information that can be used to verify a driver's qualifications and licenses.

[0267] A "generative AI model" is a model that uses artificial intelligence algorithms to analyze data and recognize its features.

[0268] "Biological characteristics" refer to information that describes individual physical features of the driver, such as their facial complexion and eye movements.

[0269] This invention provides a system to prevent drunk driving and abnormal behavior by drivers, thereby supporting safe driving. The system mainly consists of a terminal installed inside the vehicle and a server that performs advanced analysis.

[0270] The terminal uses a camera installed inside the vehicle to capture video data of the driver in real time. Specifically, high-resolution camera equipment is used, and this video data is continuously transmitted to a server. Wireless technologies such as Bluetooth and Wi-Fi are used for communication.

[0271] The server analyzes the received video data and uses a generated AI model to evaluate the driver's biometric characteristics. The server utilizes high-performance hardware such as NVIDIA GPUs and Intel processors. This analysis uses the prompt message "Detect signs of alcohol intoxication from the driver's facial expression and eye movements" to detect signs of alcohol intoxication or abnormal behavior. Based on the analysis results, if an abnormality is detected, the system instructs the driver to stop the propulsion system and activates semi-autonomous driving mode.

[0272] As an example, consider a situation where a driver feels drowsy while driving and their eyes begin to close. The system instantly detects the eye movement and warns the driver in real time via the device, "Please keep your eyes open." This warning allows the driver to quickly regain their focus and return their attention to the road.

[0273] Furthermore, before starting the vehicle, the user enters their authentication information into the terminal, and the server verifies the validity of that information. If the authentication information is valid, the terminal immediately allows the engine to start; however, if it is invalid, it notifies the user that "authentication could not be verified" and restricts the vehicle's operation.

[0274] Thus, the system of the present invention can comprehensively support driver safety through real-time data analysis utilizing a generative AI model.

[0275] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0276] Step 1:

[0277] The terminal activates a camera installed inside the vehicle and captures video data of the driver. The input at this time is a real-time video signal, which is converted into digital data frame by frame. As output, the acquired video data is sent to the server. Specifically, the terminal transfers the compressed video data to the server via Bluetooth or Wi-Fi.

[0278] Step 2:

[0279] The server takes the received video data as input and performs analysis using a generating AI model. During this process, it measures biometric information such as eye movements and facial color, and applies the prompt message, "Detect signs of intoxication from the driver's facial color and eye movements." For data processing, specific features are extracted from the video frames and classified and evaluated by the AI ​​model. The output is the analysis results, such as signs of intoxication or abnormal behavior.

[0280] Step 3:

[0281] When abnormal behavior is detected, the server generates an instruction to stop the propulsion system and outputs it to the terminal. Specifically, considering the current speed and position information of the vehicle, a signal for instructing an optimal deceleration and stop process is created. When this signal is received by the terminal, a gradual deceleration procedure is initiated.

[0282] Step 4:

[0283] Based on the instruction from the server, the terminal directly controls the propulsion system to switch the vehicle to the semi - automatic driving mode. Its input is the control instruction from the server, and as an output, a vehicle speed control signal is generated. Specifically, the control unit is activated to adjust the output of the engine and gradually stop the vehicle at a safe location.

[0284] Step 5:

[0285] After the vehicle stops, the server notifies the relevant facilities of the abnormal situation. The input at this time is the detected abnormal behavior and its related information, and as an output, an abnormal situation notification message is generated and transmitted through an appropriate communication channel. As a specific operation, data including position information and driver ID is automatically transmitted.

[0286] Step 6:

[0287] When restarting driving, the user causes the terminal to read the authentication information before starting driving. The input is a physical authentication card or digital information, which the terminal reads and transmits to the server. The server receives this data and verifies its validity. As an output, if it is valid, "Engine can be started" is notified, and if it is invalid, "Authentication failed" is notified.

[0288] (Application Example 1)

[0289] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0290] In recent years, traffic accidents caused by drunk driving or abnormal behavior by drivers have become a social problem. Conventional technology has often been insufficient to detect abnormal behavior in advance and prevent accidents. Furthermore, verifying driver credentials before starting to drive is not always safe. In addition, there is a lack of means to continuously monitor the driver's condition while driving and issue appropriate warnings in real time. Therefore, further technologies are needed to reduce the risk of traffic accidents.

[0291] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0292] In this invention, the server includes means for analyzing video data acquired in real time by an image acquisition device installed in the vehicle to detect driver intoxication or abnormal behavior; means for shutting off the vehicle's power source and switching to a semi-autonomous driving mode when an abnormality is detected; and means for continuously monitoring the driver's condition while driving using a smart device equipped with voice and visual warning functions that detect the driver's facial expression and eye movements and issue warnings. This makes it possible to instantly detect abnormal driver behavior and prevent accidents.

[0293] An "image acquisition device" is a device installed inside a vehicle that acquires video data such as the driver's facial expression and eye movements in real time.

[0294] "Methods for real-time analysis" refer to methods for immediately processing acquired video data to detect driver intoxication or abnormal behavior.

[0295] "Methods for detecting abnormal behavior" refer to methods that use generative AI to analyze the driver's biometric information and detect signs such as alcohol consumption or fatigue.

[0296] "Means for stopping the power source" refers to means that, when an abnormality is detected in the driver, the vehicle's power mechanism can be controlled to stop the engine.

[0297] "Means of switching to semi-autonomous driving mode" refers to means of activating driver assistance functions that automatically and safely control the vehicle.

[0298] "Means of verifying qualifications" refers to the means of checking the driver's license or other identification before driving begins and evaluating its validity.

[0299] A "smart device" is a device equipped with a camera and microphone that provides information to the driver by issuing voice and visual warnings.

[0300] To implement this invention, first, an image acquisition device installed inside the vehicle is used to capture biometric information such as the driver's facial color and eye movements in real time. The image acquisition device is installed as a smart device and collects the driver's biometric information. The terminal transmits this video data to a server, where it is analyzed using a generated AI model.

[0301] The server uses machine learning libraries such as TensorFlow to analyze this video data and detect abnormal driver behavior (e.g., facial redness, unnatural eye movements). If an abnormality is detected, the server sends an instruction to the terminal to shut off the vehicle's power source, and the vehicle automatically switches to semi-autonomous driving mode.

[0302] Subsequently, the driver receives audio and visual warnings. For example, warnings such as "Please stop driving" or "Your face is red. Please check for alcohol intoxication" are transmitted through the smart device's speaker and display.

[0303] Through this process, the device continuously monitors the driver's facial expression and eye movements, and takes appropriate safety measures as needed based on instructions from the server. This significantly reduces the risk of traffic accidents.

[0304] Examples of prompt sentences include "Generate an algorithm for predicting risks to detect abnormal behavior during driving." This prompt sentence is used to cause the generative AI model to perform appropriate analysis.

[0305] The flow of the specific process in Application Example 1 will be described using FIG. 12.

[0306] Step 1:

[0307] The terminal captures biometric information such as the driver's facial and eye movements in real time from an image acquisition device installed in the vehicle. The input at this time is video data, and the output is the captured video data itself. As a specific operation, the terminal periodically activates the image acquisition device and takes pictures of the driver.

[0308] Step 2:

[0309] The terminal transfers the acquired video data to the server. The input is the video data captured in Step 1, and the output is the data transmission to the server. As a specific operation, the terminal uploads the data to the server via the network.

[0310] Step 3:

[0311] The server analyzes the received video data using the generative AI model. The input is the transferred video data, and the output is the determination result of the driver's abnormal behavior. As a specific operation, the server utilizes a machine learning library such as TensorFlow to detect facial color, eye movements, and unnatural behaviors based on the prompt sentence.

[0312] Step 4:

[0313] If abnormal behavior is detected, the server sends an instruction to the terminal to issue a warning. The input is the abnormal behavior detection result obtained in step 3, and the output is the warning instruction. Specifically, the server sets up audio and visual warnings and instructs the terminal to execute them.

[0314] Step 5:

[0315] The terminal, based on instructions from the server, issues warnings to the driver via a smart device. The input is a warning instruction from the server, and the output is an audio and visual warning to the driver. Specifically, the terminal plays messages such as "Stop driving" or "Your face is red. Please check for alcohol intoxication."

[0316] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0317] The system of the present invention improves driver safety by combining an image acquisition device, a generating AI, and an emotion engine installed in a vehicle. The system is implemented in the following manner.

[0318] First, the device acquires video data in real time using cameras inside the vehicle. The acquired data is sent to a server and analyzed by a generative AI. The generative AI analyzes the driver's facial expression, eye movements, and behavior to quickly determine if there is any alcohol consumption or other abnormal behavior. It also uses an emotion engine to analyze the driver's emotional state—for example, stress or distraction.

[0319] If signs of abnormality or emotional instability are detected, the server sends an engine shutdown command to the terminal and switches to semi-autonomous driving mode, bringing the vehicle to a safe location. During this time, the server notifies the appropriate relevant authorities of the situation.

[0320] Furthermore, the system prevents inappropriate driving by requiring the user to present their driver's license before starting to drive, and verifying its validity on the server. Once driving begins, the server uses an emotion engine to provide the user with relaxation methods and warnings tailored to the driver's emotional state, via the terminal.

[0321] For example, when the emotional engine detects a high stress level while driving, the server sends a command to the terminal to change the in-car music to something calming or adjust the air conditioning to a comfortable setting. This adjustment helps to alleviate the driver's tension and provide a safer driving environment.

[0322] Thus, by comprehensively assessing the driver's condition, this system can significantly contribute to preventing traffic accidents.

[0323] The following describes the processing flow.

[0324] Step 1:

[0325] The terminal acquires video data of the driver in real time via the vehicle's camera and sends it to a server for processing.

[0326] Step 2:

[0327] To analyze the video data received by the server, a generating AI is used to analyze the driver's facial expression, eye movements, and other behaviors to determine whether or not they are intoxicated or exhibiting abnormal behavior.

[0328] Step 3:

[0329] Simultaneously, the server uses an emotion engine to analyze the driver's emotional state and detect emotional changes such as stress levels and attention deficits.

[0330] Step 4:

[0331] If abnormal behavior or unstable emotions are detected, the server sends a command to the terminal to shut off the engine and switch to semi-autonomous driving mode. Upon receiving this command, the terminal shuts off the vehicle's power source and moves the vehicle to a safe location.

[0332] Step 5:

[0333] After the server detects an anomaly, it automatically notifies the relevant organizations about the vehicle's current location and the driver's status.

[0334] Step 6:

[0335] Before starting the vehicle, the user presents their driver's license to the terminal, and the server verifies the validity of the license by checking the data. If the license is invalid, the terminal prevents the engine from starting.

[0336] Step 7:

[0337] The server uses an emotion engine to monitor the driver's emotions while the vehicle is in operation and provides optimal driving support based on their emotional state. This includes allowing the driver to change music selections and adjust air conditioning settings via their terminal.

[0338] Step 8:

[0339] The server generates music to help the driver relax and warning messages when necessary, and provides them to the user via the terminal.

[0340] This series of processes ensures a safe and smooth operating environment.

[0341] (Example 2)

[0342] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0343] One of the main causes of traffic accidents is driver inattention and unstable emotional states while driving. The challenge lies in preventing these dangers in advance by detecting these factors in real time. Furthermore, it is necessary to prevent unqualified drivers from operating vehicles and ensure driver safety.

[0344] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0345] This invention includes a server that analyzes video data acquired in real time by an image acquisition device installed in the vehicle to detect driver intoxication or abnormal behavior; an emotion analysis device that evaluates the driver's emotional state and provides relaxation methods or warnings according to the emotional state, such as stress; and a means that verifies the driver's credentials before driving begins and disables the vehicle if they are inappropriate or invalid. This makes it possible to monitor the driver's condition from multiple angles, avoid dangerous situations in advance, and ensure safety.

[0346] An "image acquisition device installed inside a vehicle" refers to a device installed inside a vehicle that includes cameras and sensors for acquiring video data of the driver.

[0347] "Generative AI" is an artificial intelligence technology used to analyze acquired video data and identify the driver's biometric information and behavior.

[0348] An "emotion analysis device" is a device that evaluates a driver's emotional state based on their facial expressions and biometric data, and determines their stress level, distractibility, and other factors.

[0349] "Semi-autonomous driving mode" is a driving mode in which the vehicle automatically controls itself to support safe driving.

[0350] "Qualification information" refers to the driver's license and other authentication information necessary for the driver to operate the vehicle.

[0351] "Drive system" refers to the collective term for equipment such as engines and motors necessary to propel a vehicle.

[0352] "Biometric information" refers to information about an individual's physical condition, such as the driver's complexion, eye condition, and behavior.

[0353] "Relaxation techniques" include methods such as changing the music or adjusting the air conditioning to promote safe driving in accordance with the driver's emotional state.

[0354] This invention is a system that improves driver safety by coordinating an image acquisition device, a generating AI, and an emotion analysis device installed inside a vehicle.

[0355] The server receives video data in real time from an image acquisition device installed inside the vehicle. This image acquisition device includes sensors such as cameras and has the function of acquiring the driver's facial color, eye movements, and behavior. The transmitted data is analyzed on the server, and the driver's biometric information is determined by a generating AI.

[0356] The generating AI analyzes video data to recognize the driver's drunk driving and other abnormal behavior. Furthermore, an emotion analysis device assesses the driver's emotional state, particularly detecting stress and distraction. This analysis utilizes advanced facial recognition technology and emotion analysis algorithms.

[0357] If an anomaly or danger is detected, the server sends an engine stop command to the terminal and switches to semi-autonomous driving mode. This allows the vehicle to automatically stop in a safe location. Details of the anomaly are then notified to the relevant organizations.

[0358] Before starting to drive, the user presents their driver's license as credentials, and its validity is verified by the server. This procedure makes it possible to prevent driving with inappropriate or invalid credentials.

[0359] While driving, the server provides relaxation methods tailored to the driver's emotional state based on data obtained from an emotion analysis device. This includes actions such as changing the music in the car or optimizing the air conditioning.

[0360] As a concrete example, if the emotion analysis device detects a high stress level in the driver, the server sends a command to the terminal to change the in-car background music to calming music. An example of the prompt message in this case is: "Analyze the driver's facial expression, eye movements, and behavior in real time, and notify if there are any abnormalities. In addition, use the emotion analysis device to evaluate the stress level and suggest adjustments."

[0361] This system aims to provide a safer driving environment and eliminate potential accident-causing factors early on by monitoring the driver's condition from multiple perspectives.

[0362] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0363] Step 1:

[0364] The terminal acquires real-time video data of the driver using cameras installed inside the vehicle. The input is video data from the in-vehicle cameras, which includes the driver's face, eye movements, and body movements. The output is the acquired raw video data. At this time, a portion of the acquired data is recorded as metadata and tagged for subsequent analysis processing.

[0365] Step 2:

[0366] The device sends the acquired video data to the server. The input is the video data acquired and tagged in step 1, and the output is the transmission of that data to the server. During the transmission process, the data is encrypted to protect privacy.

[0367] Step 3:

[0368] The server analyzes the received video data. The input is encrypted video data transmitted from the terminal. The generating AI uses facial recognition and motion analysis technology to analyze the driver's complexion, eye movements, and behavior. The output is the detection result of signs of intoxication or abnormal behavior in the driver. The analysis results are stored in a database and used for subsequent processing.

[0369] Step 4:

[0370] The server uses an emotion analysis device to evaluate the driver's emotional state. The input is the analysis results obtained in step 3. The emotion analysis device evaluates stress levels and distractibility based on the driver's facial expressions and biometric data. The output is the evaluation result of the emotional state, providing information indicating the need for relaxation or warning.

[0371] Step 5:

[0372] If an abnormality or emotional instability is detected, the server sends an engine stop command to the terminal. The input is the analysis and evaluation results from steps 3 and 4. The output is a command to switch to semi-autonomous driving mode. Specifically, a program to guide the vehicle to the optimal stopping point is activated.

[0373] Step 6:

[0374] Before starting the vehicle, the user presents their credentials to the terminal. The input is the credentials, such as a driver's license, presented by the user. The output is the result of the credentials being verified by the server. In this verification process, if no valid credentials are found, the server locks the vehicle from starting.

[0375] Step 7:

[0376] While driving, the server provides relaxation methods tailored to the driver's state based on information from an emotion analysis device. The input is the real-time evaluation of the driver's emotional state. The output is a command, such as changing music selection or adjusting the air conditioning settings. Specific actions include accessing a music library based on the driver's preferences and automatically adjusting the air conditioning system. Through this process, the system aims to alleviate driver stress and ensure a safe driving environment.

[0377] (Application Example 2)

[0378] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0379] Conventional vehicle safety systems focus on detecting driver intoxication and abnormal behavior, but are insufficient in addressing potential hazards based on the driver's emotional state. Furthermore, if a driver's emotions become unstable while driving, there is no means to appropriately adjust the environment, leaving the risk of accidents unresolved. Moreover, even if a driver is stressed or distracted, there is no system to quickly detect and appropriately address these conditions, resulting in a situation that cannot adequately support safe driving.

[0380] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0381] In this invention, the server includes means for analyzing video data acquired in real time by an image acquisition device installed in the vehicle to detect driver intoxication or abnormal behavior; means for shutting off the vehicle's power source and switching to a semi-autonomous driving mode when an abnormality is detected by the means; and means for analyzing the driver's emotional state, using an emotion engine to detect stress or distraction, and adjusting the driving environment based on this. This makes it possible to immediately detect potential dangers based on the driver's emotional state and support safe and comfortable driving.

[0382] An "image acquisition device" is a device installed inside a vehicle that acquires video data of the driver in real time.

[0383] "Generative AI" is an artificial intelligence technology that analyzes a driver's facial expression, eye movements, and behavior based on the driver's biometric information.

[0384] The "emotional engine" is a program that analyzes the driver's emotional state to detect stress and distraction.

[0385] "Semi-autonomous driving mode" is a driving mode that assists the driver's operation when an abnormality is detected and helps the vehicle stop in a safe location.

[0386] "Qualification information" refers to the licenses and other authorizations required for a driver to operate a vehicle.

[0387] "Dangerous driving" refers to a situation where a driver's actions pose a level of risk far beyond what is considered normal driving, and detecting such situations is crucial.

[0388] "Adjusting the driving environment" refers to the act of changing the in-car sound system and air conditioning according to the driver's emotional state, thereby optimizing the driver's condition.

[0389] To implement this invention, an image acquisition device is installed inside the vehicle, and a system is configured to capture video data of the driver in real time. The server analyzes the acquired video data using various software and evaluates the driver's condition. Face recognition is performed using OpenCV for the analysis, and a generated AI using TensorFlow analyzes biometric information such as the driver's facial color, eye movements, and behavior.

[0390] The server further utilizes an emotion engine to analyze the driver's emotional state, such as stress and distraction. If abnormal behavior or an unstable emotional state is detected, it switches to a semi-autonomous driving mode in conjunction with the vehicle's control system and brings the vehicle to a safe stop. In addition, the server can adjust the in-car sound and air conditioning settings according to the user's emotional state. At this time, it also provides the user with suggestions for relaxation methods and behavioral improvements.

[0391] For example, if the emotion engine determines that the user's stress level is rising during a long drive, the server will change the in-car music to relaxing classical music and automatically adjust the air conditioning to a comfortable temperature. It will also send a notification to the user's device suggesting they take a short break. The prompt sent to the AI ​​model in this case would be something like, "The driver's eyes are closed for longer than usual. Are you experiencing increased tension or stress?"

[0392] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0393] Step 1:

[0394] The terminal captures video data of the driver in real time using an image acquisition device inside the vehicle. The input is video of the driver inside the vehicle, which is obtained as output data in image format.

[0395] Step 2:

[0396] The server receives video data transmitted from the terminal and performs face recognition using OpenCV. The input data is video data of the driver, which is analyzed to identify the driver's face region. The output is the recognized face image and its coordinate information.

[0397] Step 3:

[0398] The server inputs the facial region data recognized by face recognition into a generative AI model using TensorFlow, which analyzes the driver's complexion, eye movements, and behavior. The input is facial region data, and the output is an evaluation of the driver's state based on the analyzed biometric information.

[0399] Step 4:

[0400] The server uses an emotion engine to assess the driver's emotional state, specifically their stress level and degree of distraction. The input is the driver's state assessment result from a generative AI model, and the output is an index indicating the emotional state.

[0401] Step 5:

[0402] If an emotional state or abnormality is detected, the server sends a signal to the vehicle's control system and switches to semi-autonomous driving mode. The input is the result of the abnormality detection, and the output is the control signal to the vehicle's control system.

[0403] Step 6:

[0404] The server sends signals to the terminal to adjust the in-car sound and air conditioning based on the user's emotional state. The input is an indicator of the emotion engine, and the output is a signal instructing adjustments.

[0405] Step 7:

[0406] The terminal changes the sound and air conditioning settings inside the vehicle based on instructions received from the server. The input here is the adjustment instructions from the server, and the output is the specific action to be performed, such as switching music or adjusting the temperature settings.

[0407] Step 8:

[0408] The server sends notifications to the user regarding detected anomalies and emotional states. It alerts the user by sending prompts such as "We recommend you take a short break." The input is the result of the emotion engine's analysis, and the output is the notification content sent to the user.

[0409] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0410] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0411] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0412] [Third Embodiment]

[0413] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0414] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0415] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0416] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0417] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0418] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0419] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0420] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0421] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0422] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0423] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0424] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0425] This invention provides a detailed system for detecting and responding to drunk driving and other abnormal behavior by drivers. The system mainly consists of various devices and a server installed inside the vehicle. An example of its operation is described below.

[0426] In an embodiment of the present invention, the terminal first captures video data in real time from a camera installed inside the vehicle. Then, it transmits this video data to a server for real-time analysis. The server uses generated AI to analyze the driver's facial expression, eye movements, and other behavioral characteristics through this video data to detect signs of drinking or abnormal behavior.

[0427] If an anomaly is detected, the server sends a command to the terminal to shut down the engine and activates semi-autonomous driving mode to safely operate the vehicle. At the same time, the server automatically notifies relevant organizations of the anomaly to facilitate a rapid response.

[0428] For license verification, the user scans their driver's license at the terminal before starting to drive, and its validity is verified by the server. If the license is found to be expired or invalid, the terminal will not allow the engine to start.

[0429] The server also continuously monitors the driver's behavior while driving, and if dangerous driving is detected, the terminal provides real-time warnings and corrective instructions. For example, in a real-world case, if lane departure is detected, the terminal will provide an audio warning to the user saying, "Please stay in your lane." This warning encourages the driver to immediately return to safe driving.

[0430] By implementing this system, we can provide a new vehicle driving support environment that can suppress dangerous behaviors such as drunk driving and prevent traffic accidents.

[0431] The following describes the processing flow.

[0432] Step 1:

[0433] The device acquires video data from the in-car camera in real time and sends that data to the server.

[0434] Step 2:

[0435] To analyze the video data received by the server, a generating AI is called in to analyze the driver's facial expression, eye condition, and other behavioral patterns.

[0436] Step 3:

[0437] Based on the data analyzed by the generated AI, the server determines whether the driver is showing signs of intoxication or exhibiting abnormal behavior.

[0438] Step 4:

[0439] If an abnormality is detected, the server sends an engine stop command and a command to switch to semi-autonomous driving mode to the terminal.

[0440] Step 5:

[0441] The terminal, following the received commands, shuts off the vehicle's power source, activates semi-autonomous driving mode, and stops in a safe location.

[0442] Step 6:

[0443] The server automatically notifies the relevant authorities when it detects an anomaly and sends a message containing details such as the vehicle's location.

[0444] Step 7:

[0445] Before the user starts driving, they scan their driver's license at a terminal, the server receives the information, and verifies the validity of the license.

[0446] Step 8:

[0447] If the server detects that the license has expired or is invalid, it sends an engine start prohibition command to the terminal. The terminal executes this command and prevents the engine from starting.

[0448] Step 9:

[0449] While driving, the server continuously monitors the driver's behavior and, if it determines that dangerous driving is occurring, provides the user with warnings and instructions for improvement via the terminal.

[0450] (Example 1)

[0451] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0452] The challenge lies in preventing traffic accidents caused by drunk driving or abnormal behavior by drivers, and ensuring the safety of drivers and those around them. It is also necessary to prevent accidents and problems caused by drivers operating vehicles without verifying their own driving qualifications. Furthermore, it is crucial to monitor dangerous driving in real time and provide appropriate guidance as needed.

[0453] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0454] In this invention, the server includes means for analyzing video data acquired in real time by a camera installed in the vehicle to detect drunk driving or abnormal behavior by the driver; means for analyzing the driver's biological characteristics by analyzing image data using a generative AI model; and means for transmitting signals to a terminal to gradually decelerate and safely stop the vehicle. This makes it possible to quickly detect drunk driving or abnormal behavior and provide a safe driving environment.

[0455] A "camera" is a device installed inside a vehicle to acquire video data of the interior, including the driver.

[0456] "Real-time analysis" is a process that processes acquired data immediately to derive some kind of judgment or result.

[0457] A "propulsion system" is a system that supplies and controls power, including various devices necessary to move a vehicle forward.

[0458] "Semi-autonomous driving mode" is a driving mode that automatically controls the basic driving operations of the vehicle while minimizing driver intervention.

[0459] "Related facilities" refers to transportation and safety organizations that receive and respond to information related to traffic management and safety management.

[0460] "Authentication information" refers to information that can be used to verify a driver's qualifications and licenses.

[0461] A "generative AI model" is a model that uses artificial intelligence algorithms to analyze data and recognize its features.

[0462] "Biological characteristics" refer to information that describes individual physical features of the driver, such as their facial complexion and eye movements.

[0463] This invention provides a system to prevent drunk driving and abnormal behavior by drivers, thereby supporting safe driving. The system mainly consists of a terminal installed inside the vehicle and a server that performs advanced analysis.

[0464] The terminal uses a camera installed inside the vehicle to capture video data of the driver in real time. Specifically, high-resolution camera equipment is used, and this video data is continuously transmitted to a server. Wireless technologies such as Bluetooth and Wi-Fi are used for communication.

[0465] The server analyzes the received video data and uses a generated AI model to evaluate the driver's biometric characteristics. The server utilizes high-performance hardware such as NVIDIA GPUs and Intel processors. This analysis uses the prompt message "Detect signs of alcohol intoxication from the driver's facial expression and eye movements" to detect signs of alcohol intoxication or abnormal behavior. Based on the analysis results, if an abnormality is detected, the system instructs the driver to stop the propulsion system and activates semi-autonomous driving mode.

[0466] As an example, consider a situation where a driver feels drowsy while driving and their eyes begin to close. The system instantly detects the eye movement and warns the driver in real time via the device, "Please keep your eyes open." This warning allows the driver to quickly regain their focus and return their attention to the road.

[0467] Furthermore, before starting the vehicle, the user enters their authentication information into the terminal, and the server verifies the validity of that information. If the authentication information is valid, the terminal immediately allows the engine to start; however, if it is invalid, it notifies the user that "authentication could not be verified" and restricts the vehicle's operation.

[0468] Thus, the system of the present invention can comprehensively support driver safety through real-time data analysis utilizing a generative AI model.

[0469] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0470] Step 1:

[0471] The terminal activates a camera installed inside the vehicle and captures video data of the driver. The input at this time is a real-time video signal, which is converted into digital data frame by frame. As output, the acquired video data is sent to the server. Specifically, the terminal transfers the compressed video data to the server via Bluetooth or Wi-Fi.

[0472] Step 2:

[0473] The server takes the received video data as input and performs analysis using a generating AI model. During this process, it measures biometric information such as eye movements and facial color, and applies the prompt message, "Detect signs of intoxication from the driver's facial color and eye movements." For data processing, specific features are extracted from the video frames and classified and evaluated by the AI ​​model. The output is the analysis results, such as signs of intoxication or abnormal behavior.

[0474] Step 3:

[0475] If abnormal behavior is detected, the server generates a command to shut down the propulsion system and outputs it to the terminal. Specifically, it creates a signal that instructs the optimal deceleration and stopping process, taking into account the vehicle's current speed and location information. When this signal is received by the terminal, the stepwise deceleration procedure begins.

[0476] Step 4:

[0477] Based on instructions from the server, the terminal directly controls the propulsion system to switch the vehicle into semi-autonomous driving mode. Its input is the server's control instructions, and its output is a vehicle speed control signal. Specifically, the control unit is activated, adjusting the engine output and gradually bringing the vehicle to a safe stop.

[0478] Step 5:

[0479] After the vehicle stops, the server notifies the relevant facilities of the abnormal situation. The input at this time is the detected abnormal behavior and related information, and the output is an abnormal situation notification message which is sent through the appropriate communication channel. Specifically, data including location information and driver ID is automatically transmitted.

[0480] Step 6:

[0481] When restarting operation, the user has the terminal read their authentication information before starting the vehicle. The input is either a physical authentication card or digital information, which the terminal reads and sends to the server. The server receives this data and verifies its validity. The output is "Engine ready to start" if valid, or "Authentication failed" if invalid.

[0482] (Application Example 1)

[0483] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0484] In recent years, traffic accidents caused by drunk driving or abnormal behavior by drivers have become a social problem. Conventional technology has often been insufficient to detect abnormal behavior in advance and prevent accidents. Furthermore, verifying driver credentials before starting to drive is not always safe. In addition, there is a lack of means to continuously monitor the driver's condition while driving and issue appropriate warnings in real time. Therefore, further technologies are needed to reduce the risk of traffic accidents.

[0485] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0486] In this invention, the server includes means for analyzing video data acquired in real time by an image acquisition device installed in the vehicle to detect driver intoxication or abnormal behavior; means for shutting off the vehicle's power source and switching to a semi-autonomous driving mode when an abnormality is detected; and means for continuously monitoring the driver's condition while driving using a smart device equipped with voice and visual warning functions that detect the driver's facial expression and eye movements and issue warnings. This makes it possible to instantly detect abnormal driver behavior and prevent accidents.

[0487] An "image acquisition device" is a device installed inside a vehicle that acquires video data such as the driver's facial expression and eye movements in real time.

[0488] "Methods for real-time analysis" refer to methods for immediately processing acquired video data to detect driver intoxication or abnormal behavior.

[0489] "Methods for detecting abnormal behavior" refer to methods that use generative AI to analyze the driver's biometric information and detect signs such as alcohol consumption or fatigue.

[0490] "Means for stopping the power source" refers to means that, when an abnormality is detected in the driver, the vehicle's power mechanism can be controlled to stop the engine.

[0491] "Means of switching to semi-autonomous driving mode" refers to means of activating driver assistance functions that automatically and safely control the vehicle.

[0492] "Means of verifying qualifications" refers to the means of checking the driver's license or other identification before driving begins and evaluating its validity.

[0493] A "smart device" is a device equipped with a camera and microphone that provides information to the driver by issuing voice and visual warnings.

[0494] To implement this invention, first, an image acquisition device installed inside the vehicle is used to capture biometric information such as the driver's facial color and eye movements in real time. The image acquisition device is installed as a smart device and collects the driver's biometric information. The terminal transmits this video data to a server, where it is analyzed using a generated AI model.

[0495] The server uses machine learning libraries such as TensorFlow to analyze this video data and detect abnormal driver behavior (e.g., facial redness, unnatural eye movements). If an abnormality is detected, the server sends an instruction to the terminal to shut off the vehicle's power source, and the vehicle automatically switches to semi-autonomous driving mode.

[0496] Subsequently, the driver receives audio and visual warnings. For example, warnings such as "Please stop driving" or "Your face is red. Please check for alcohol intoxication" are transmitted through the smart device's speaker and display.

[0497] Through this process, the device continuously monitors the driver's facial expression and eye movements, and takes appropriate safety measures as needed based on instructions from the server. This significantly reduces the risk of traffic accidents.

[0498] An example of a prompt statement is, "Generate a risk prediction algorithm to detect abnormal driving behavior." This prompt statement is used to prompt the generated AI model to perform appropriate analysis.

[0499] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0500] Step 1:

[0501] The terminal captures biometric information, such as the driver's face and eye movements, in real time from an image acquisition device installed inside the vehicle. The input is video data, and the output is the captured video data itself. Specifically, the terminal periodically activates the image acquisition device and takes pictures of the driver.

[0502] Step 2:

[0503] The terminal transfers the acquired video data to the server. The input is the video data captured in step 1, and the output is the transmission of data to the server. Specifically, the terminal uploads the data to the server via the network.

[0504] Step 3:

[0505] The server analyzes the received video data using a generation AI model. The input is the transmitted video data, and the output is the result of the driver's abnormal behavior detection. Specifically, the server uses machine learning libraries such as TensorFlow to detect facial expression, eye movements, and unnatural behavior based on prompt messages.

[0506] Step 4:

[0507] If abnormal behavior is detected, the server sends an instruction to the terminal to issue a warning. The input is the abnormal behavior detection result obtained in step 3, and the output is the warning instruction. Specifically, the server sets up audio and visual warnings and instructs the terminal to execute them.

[0508] Step 5:

[0509] The terminal, based on instructions from the server, issues warnings to the driver via a smart device. The input is a warning instruction from the server, and the output is an audio and visual warning to the driver. Specifically, the terminal plays messages such as "Stop driving" or "Your face is red. Please check for alcohol intoxication."

[0510] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0511] The system of the present invention improves driver safety by combining an image acquisition device, a generating AI, and an emotion engine installed in a vehicle. The system is implemented in the following manner.

[0512] First, the device acquires video data in real time using cameras inside the vehicle. The acquired data is sent to a server and analyzed by a generative AI. The generative AI analyzes the driver's facial expression, eye movements, and behavior to quickly determine if there is any alcohol consumption or other abnormal behavior. It also uses an emotion engine to analyze the driver's emotional state—for example, stress or distraction.

[0513] If signs of abnormality or emotional instability are detected, the server sends an engine shutdown command to the terminal and switches to semi-autonomous driving mode, bringing the vehicle to a safe location. During this time, the server notifies the appropriate relevant authorities of the situation.

[0514] Furthermore, the system prevents inappropriate driving by requiring the user to present their driver's license before starting to drive, and verifying its validity on the server. Once driving begins, the server uses an emotion engine to provide the user with relaxation methods and warnings tailored to the driver's emotional state, via the terminal.

[0515] For example, when the emotional engine detects a high stress level while driving, the server sends a command to the terminal to change the in-car music to something calming or adjust the air conditioning to a comfortable setting. This adjustment helps to alleviate the driver's tension and provide a safer driving environment.

[0516] Thus, by comprehensively assessing the driver's condition, this system can significantly contribute to preventing traffic accidents.

[0517] The following describes the processing flow.

[0518] Step 1:

[0519] The terminal acquires video data of the driver in real time via the vehicle's camera and sends it to a server for processing.

[0520] Step 2:

[0521] To analyze the video data received by the server, a generating AI is used to analyze the driver's facial expression, eye movements, and other behaviors to determine whether or not they are intoxicated or exhibiting abnormal behavior.

[0522] Step 3:

[0523] Simultaneously, the server uses an emotion engine to analyze the driver's emotional state and detect emotional changes such as stress levels and attention deficits.

[0524] Step 4:

[0525] If abnormal behavior or unstable emotions are detected, the server sends a command to the terminal to shut off the engine and switch to semi-autonomous driving mode. Upon receiving this command, the terminal shuts off the vehicle's power source and moves the vehicle to a safe location.

[0526] Step 5:

[0527] After the server detects an anomaly, it automatically notifies the relevant organizations about the vehicle's current location and the driver's status.

[0528] Step 6:

[0529] Before starting the vehicle, the user presents their driver's license to the terminal, and the server verifies the validity of the license by checking the data. If the license is invalid, the terminal prevents the engine from starting.

[0530] Step 7:

[0531] The server uses an emotion engine to monitor the driver's emotions while the vehicle is in operation and provides optimal driving support based on their emotional state. This includes allowing the driver to change music selections and adjust air conditioning settings via their terminal.

[0532] Step 8:

[0533] The server generates music to help the driver relax and warning messages when necessary, and provides them to the user via the terminal.

[0534] This series of processes ensures a safe and smooth operating environment.

[0535] (Example 2)

[0536] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0537] One of the main causes of traffic accidents is driver inattention and unstable emotional states while driving. The challenge lies in preventing these dangers in advance by detecting these factors in real time. Furthermore, it is necessary to prevent unqualified drivers from operating vehicles and ensure driver safety.

[0538] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0539] This invention includes a server that analyzes video data acquired in real time by an image acquisition device installed in the vehicle to detect driver intoxication or abnormal behavior; an emotion analysis device that evaluates the driver's emotional state and provides relaxation methods or warnings according to the emotional state, such as stress; and a means that verifies the driver's credentials before driving begins and disables the vehicle if they are inappropriate or invalid. This makes it possible to monitor the driver's condition from multiple angles, avoid dangerous situations in advance, and ensure safety.

[0540] An "image acquisition device installed inside a vehicle" refers to a device installed inside a vehicle that includes cameras and sensors for acquiring video data of the driver.

[0541] "Generative AI" is an artificial intelligence technology used to analyze acquired video data and identify the driver's biometric information and behavior.

[0542] An "emotion analysis device" is a device that evaluates a driver's emotional state based on their facial expressions and biometric data, and determines their stress level, distractibility, and other factors.

[0543] "Semi-autonomous driving mode" is a driving mode in which the vehicle automatically controls itself to support safe driving.

[0544] "Qualification information" refers to the driver's license and other authentication information necessary for the driver to operate the vehicle.

[0545] "Drive system" refers to the collective term for equipment such as engines and motors necessary to propel a vehicle.

[0546] "Biometric information" refers to information about an individual's physical condition, such as the driver's complexion, eye condition, and behavior.

[0547] "Relaxation techniques" include methods such as changing the music or adjusting the air conditioning to promote safe driving in accordance with the driver's emotional state.

[0548] This invention is a system that improves driver safety by coordinating an image acquisition device, a generating AI, and an emotion analysis device installed inside a vehicle.

[0549] The server receives video data in real time from an image acquisition device installed inside the vehicle. This image acquisition device includes sensors such as cameras and has the function of acquiring the driver's facial color, eye movements, and behavior. The transmitted data is analyzed on the server, and the driver's biometric information is determined by a generating AI.

[0550] The generating AI analyzes video data to recognize the driver's drunk driving and other abnormal behavior. Furthermore, an emotion analysis device assesses the driver's emotional state, particularly detecting stress and distraction. This analysis utilizes advanced facial recognition technology and emotion analysis algorithms.

[0551] If an anomaly or danger is detected, the server sends an engine stop command to the terminal and switches to semi-autonomous driving mode. This allows the vehicle to automatically stop in a safe location. Details of the anomaly are then notified to the relevant organizations.

[0552] Before starting to drive, the user presents their driver's license as credentials, and its validity is verified by the server. This procedure makes it possible to prevent driving with inappropriate or invalid credentials.

[0553] While driving, the server provides relaxation methods tailored to the driver's emotional state based on data obtained from an emotion analysis device. This includes actions such as changing the music in the car or optimizing the air conditioning.

[0554] As a concrete example, if the emotion analysis device detects a high stress level in the driver, the server sends a command to the terminal to change the in-car background music to calming music. An example of the prompt message in this case is: "Analyze the driver's facial expression, eye movements, and behavior in real time, and notify if there are any abnormalities. In addition, use the emotion analysis device to evaluate the stress level and suggest adjustments."

[0555] This system aims to provide a safer driving environment and eliminate potential accident-causing factors early on by monitoring the driver's condition from multiple perspectives.

[0556] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0557] Step 1:

[0558] The terminal acquires real-time video data of the driver using cameras installed inside the vehicle. The input is video data from the in-vehicle cameras, which includes the driver's face, eye movements, and body movements. The output is the acquired raw video data. At this time, a portion of the acquired data is recorded as metadata and tagged for subsequent analysis processing.

[0559] Step 2:

[0560] The device sends the acquired video data to the server. The input is the video data acquired and tagged in step 1, and the output is the transmission of that data to the server. During the transmission process, the data is encrypted to protect privacy.

[0561] Step 3:

[0562] The server analyzes the received video data. The input is encrypted video data transmitted from the terminal. The generating AI uses facial recognition and motion analysis technology to analyze the driver's complexion, eye movements, and behavior. The output is the detection result of signs of intoxication or abnormal behavior in the driver. The analysis results are stored in a database and used for subsequent processing.

[0563] Step 4:

[0564] The server uses an emotion analysis device to evaluate the driver's emotional state. The input is the analysis results obtained in step 3. The emotion analysis device evaluates stress levels and distractibility based on the driver's facial expressions and biometric data. The output is the evaluation result of the emotional state, providing information indicating the need for relaxation or warning.

[0565] Step 5:

[0566] If an abnormality or emotional instability is detected, the server sends an engine stop command to the terminal. The input is the analysis and evaluation results from steps 3 and 4. The output is a command to switch to semi-autonomous driving mode. Specifically, a program to guide the vehicle to the optimal stopping point is activated.

[0567] Step 6:

[0568] Before starting the vehicle, the user presents their credentials to the terminal. The input is the credentials, such as a driver's license, presented by the user. The output is the result of the credentials being verified by the server. In this verification process, if no valid credentials are found, the server locks the vehicle from starting.

[0569] Step 7:

[0570] While driving, the server provides relaxation methods tailored to the driver's state based on information from an emotion analysis device. The input is the real-time evaluation of the driver's emotional state. The output is a command, such as changing music selection or adjusting the air conditioning settings. Specific actions include accessing a music library based on the driver's preferences and automatically adjusting the air conditioning system. Through this process, the system aims to alleviate driver stress and ensure a safe driving environment.

[0571] (Application Example 2)

[0572] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0573] Conventional vehicle safety systems focus on detecting driver intoxication and abnormal behavior, but are insufficient in addressing potential hazards based on the driver's emotional state. Furthermore, if a driver's emotions become unstable while driving, there is no means to appropriately adjust the environment, leaving the risk of accidents unresolved. Moreover, even if a driver is stressed or distracted, there is no system to quickly detect and appropriately address these conditions, resulting in a situation that cannot adequately support safe driving.

[0574] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0575] In this invention, the server includes means for analyzing video data acquired in real time by an image acquisition device installed in the vehicle to detect driver intoxication or abnormal behavior; means for shutting off the vehicle's power source and switching to a semi-autonomous driving mode when an abnormality is detected by the means; and means for analyzing the driver's emotional state, using an emotion engine to detect stress or distraction, and adjusting the driving environment based on this. This makes it possible to immediately detect potential dangers based on the driver's emotional state and support safe and comfortable driving.

[0576] An "image acquisition device" is a device installed inside a vehicle that acquires video data of the driver in real time.

[0577] "Generative AI" is an artificial intelligence technology that analyzes a driver's facial expression, eye movements, and behavior based on the driver's biometric information.

[0578] The "emotional engine" is a program that analyzes the driver's emotional state to detect stress and distraction.

[0579] "Semi-autonomous driving mode" is a driving mode that assists the driver's operation when an abnormality is detected and helps the vehicle stop in a safe location.

[0580] "Qualification information" refers to the licenses and other authorizations required for a driver to operate a vehicle.

[0581] "Dangerous driving" refers to a situation where a driver's actions pose a level of risk far beyond what is considered normal driving, and detecting such situations is crucial.

[0582] "Adjusting the driving environment" refers to the act of changing the in-car sound system and air conditioning according to the driver's emotional state, thereby optimizing the driver's condition.

[0583] To implement this invention, an image acquisition device is installed inside the vehicle, and a system is configured to capture video data of the driver in real time. The server analyzes the acquired video data using various software and evaluates the driver's condition. Face recognition is performed using OpenCV for the analysis, and a generated AI using TensorFlow analyzes biometric information such as the driver's facial color, eye movements, and behavior.

[0584] The server further utilizes an emotion engine to analyze the driver's emotional state, such as stress and distraction. If abnormal behavior or an unstable emotional state is detected, it switches to a semi-autonomous driving mode in conjunction with the vehicle's control system and brings the vehicle to a safe stop. In addition, the server can adjust the in-car sound and air conditioning settings according to the user's emotional state. At this time, it also provides the user with suggestions for relaxation methods and behavioral improvements.

[0585] For example, if the emotion engine determines that the user's stress level is rising during a long drive, the server will change the in-car music to relaxing classical music and automatically adjust the air conditioning to a comfortable temperature. It will also send a notification to the user's device suggesting they take a short break. The prompt sent to the AI ​​model in this case would be something like, "The driver's eyes are closed for longer than usual. Are you experiencing increased tension or stress?"

[0586] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0587] Step 1:

[0588] The terminal captures video data of the driver in real time using an image acquisition device inside the vehicle. The input is video of the driver inside the vehicle, which is obtained as output data in image format.

[0589] Step 2:

[0590] The server receives video data transmitted from the terminal and performs face recognition using OpenCV. The input data is video data of the driver, which is analyzed to identify the driver's face region. The output is the recognized face image and its coordinate information.

[0591] Step 3:

[0592] The server inputs the facial region data recognized by face recognition into a generative AI model using TensorFlow, which analyzes the driver's complexion, eye movements, and behavior. The input is facial region data, and the output is an evaluation of the driver's state based on the analyzed biometric information.

[0593] Step 4:

[0594] The server uses an emotion engine to assess the driver's emotional state, specifically their stress level and degree of distraction. The input is the driver's state assessment result from a generative AI model, and the output is an index indicating the emotional state.

[0595] Step 5:

[0596] If an emotional state or abnormality is detected, the server sends a signal to the vehicle's control system and switches to semi-autonomous driving mode. The input is the result of the abnormality detection, and the output is the control signal to the vehicle's control system.

[0597] Step 6:

[0598] The server sends signals to the terminal to adjust the in-car sound and air conditioning based on the user's emotional state. The input is an indicator of the emotion engine, and the output is a signal instructing adjustments.

[0599] Step 7:

[0600] The terminal changes the sound and air conditioning settings inside the vehicle based on instructions received from the server. The input here is the adjustment instructions from the server, and the output is the specific action to be performed, such as switching music or adjusting the temperature settings.

[0601] Step 8:

[0602] The server sends notifications to the user regarding detected anomalies and emotional states. It alerts the user by sending prompts such as "We recommend you take a short break." The input is the result of the emotion engine's analysis, and the output is the notification content sent to the user.

[0603] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0604] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0605] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0606] [Fourth Embodiment]

[0607] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0608] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0609] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0610] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0611] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0612] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0613] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0614] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0615] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0616] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0617] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0618] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0619] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0620] This invention provides a detailed system for detecting and responding to drunk driving and other abnormal behavior by drivers. The system mainly consists of various devices and a server installed inside the vehicle. An example of its operation is described below.

[0621] In an embodiment of the present invention, the terminal first captures video data in real time from a camera installed inside the vehicle. Then, it transmits this video data to a server for real-time analysis. The server uses generated AI to analyze the driver's facial expression, eye movements, and other behavioral characteristics through this video data to detect signs of drinking or abnormal behavior.

[0622] If an anomaly is detected, the server sends a command to the terminal to shut down the engine and activates semi-autonomous driving mode to safely operate the vehicle. At the same time, the server automatically notifies relevant organizations of the anomaly to facilitate a rapid response.

[0623] For license verification, the user scans their driver's license at the terminal before starting to drive, and its validity is verified by the server. If the license is found to be expired or invalid, the terminal will not allow the engine to start.

[0624] The server also continuously monitors the driver's behavior while driving, and if dangerous driving is detected, the terminal provides real-time warnings and corrective instructions. For example, in a real-world case, if lane departure is detected, the terminal will provide an audio warning to the user saying, "Please stay in your lane." This warning encourages the driver to immediately return to safe driving.

[0625] By implementing this system, we can provide a new vehicle driving support environment that can suppress dangerous behaviors such as drunk driving and prevent traffic accidents.

[0626] The following describes the processing flow.

[0627] Step 1:

[0628] The device acquires video data from the in-car camera in real time and sends that data to the server.

[0629] Step 2:

[0630] To analyze the video data received by the server, a generating AI is called in to analyze the driver's facial expression, eye condition, and other behavioral patterns.

[0631] Step 3:

[0632] Based on the data analyzed by the generated AI, the server determines whether the driver is showing signs of intoxication or exhibiting abnormal behavior.

[0633] Step 4:

[0634] If an abnormality is detected, the server sends an engine stop command and a command to switch to semi-autonomous driving mode to the terminal.

[0635] Step 5:

[0636] The terminal, following the received commands, shuts off the vehicle's power source, activates semi-autonomous driving mode, and stops in a safe location.

[0637] Step 6:

[0638] The server automatically notifies the relevant authorities when it detects an anomaly and sends a message containing details such as the vehicle's location.

[0639] Step 7:

[0640] Before the user starts driving, they scan their driver's license at a terminal, the server receives the information, and verifies the validity of the license.

[0641] Step 8:

[0642] If the server detects that the license has expired or is invalid, it sends an engine start prohibition command to the terminal. The terminal executes this command and prevents the engine from starting.

[0643] Step 9:

[0644] While driving, the server continuously monitors the driver's behavior and, if it determines that dangerous driving is occurring, provides the user with warnings and instructions for improvement via the terminal.

[0645] (Example 1)

[0646] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0647] The challenge lies in preventing traffic accidents caused by drunk driving or abnormal behavior by drivers, and ensuring the safety of drivers and those around them. It is also necessary to prevent accidents and problems caused by drivers operating vehicles without verifying their own driving qualifications. Furthermore, it is crucial to monitor dangerous driving in real time and provide appropriate guidance as needed.

[0648] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0649] In this invention, the server includes means for analyzing video data acquired in real time by a camera installed in the vehicle to detect drunk driving or abnormal behavior by the driver; means for analyzing the driver's biological characteristics by analyzing image data using a generative AI model; and means for transmitting signals to a terminal to gradually decelerate and safely stop the vehicle. This makes it possible to quickly detect drunk driving or abnormal behavior and provide a safe driving environment.

[0650] A "camera" is a device installed inside a vehicle to acquire video data of the interior, including the driver.

[0651] "Real-time analysis" is a process that processes acquired data immediately to derive some kind of judgment or result.

[0652] A "propulsion system" is a system that supplies and controls power, including various devices necessary to move a vehicle forward.

[0653] "Semi-autonomous driving mode" is a driving mode that automatically controls the basic driving operations of the vehicle while minimizing driver intervention.

[0654] "Related facilities" refers to transportation and safety organizations that receive and respond to information related to traffic management and safety management.

[0655] "Authentication information" refers to information that can be used to verify a driver's qualifications and licenses.

[0656] A "generative AI model" is a model that uses artificial intelligence algorithms to analyze data and recognize its features.

[0657] "Biological characteristics" refer to information that describes individual physical features of the driver, such as their facial complexion and eye movements.

[0658] This invention provides a system to prevent drunk driving and abnormal behavior by drivers, thereby supporting safe driving. The system mainly consists of a terminal installed inside the vehicle and a server that performs advanced analysis.

[0659] The terminal uses a camera installed inside the vehicle to capture video data of the driver in real time. Specifically, high-resolution camera equipment is used, and this video data is continuously transmitted to a server. Wireless technologies such as Bluetooth and Wi-Fi are used for communication.

[0660] The server analyzes the received video data and uses a generated AI model to evaluate the driver's biometric characteristics. The server utilizes high-performance hardware such as NVIDIA GPUs and Intel processors. This analysis uses the prompt message "Detect signs of alcohol intoxication from the driver's facial expression and eye movements" to detect signs of alcohol intoxication or abnormal behavior. Based on the analysis results, if an abnormality is detected, the system instructs the driver to stop the propulsion system and activates semi-autonomous driving mode.

[0661] As an example, consider a situation where a driver feels drowsy while driving and their eyes begin to close. The system instantly detects the eye movement and warns the driver in real time via the device, "Please keep your eyes open." This warning allows the driver to quickly regain their focus and return their attention to the road.

[0662] Furthermore, before starting the vehicle, the user enters their authentication information into the terminal, and the server verifies the validity of that information. If the authentication information is valid, the terminal immediately allows the engine to start; however, if it is invalid, it notifies the user that "authentication could not be verified" and restricts the vehicle's operation.

[0663] Thus, the system of the present invention can comprehensively support driver safety through real-time data analysis utilizing a generative AI model.

[0664] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0665] Step 1:

[0666] The terminal activates a camera installed inside the vehicle and captures video data of the driver. The input at this time is a real-time video signal, which is converted into digital data frame by frame. As output, the acquired video data is sent to the server. Specifically, the terminal transfers the compressed video data to the server via Bluetooth or Wi-Fi.

[0667] Step 2:

[0668] The server takes the received video data as input and performs analysis using a generating AI model. During this process, it measures biometric information such as eye movements and facial color, and applies the prompt message, "Detect signs of intoxication from the driver's facial color and eye movements." For data processing, specific features are extracted from the video frames and classified and evaluated by the AI ​​model. The output is the analysis results, such as signs of intoxication or abnormal behavior.

[0669] Step 3:

[0670] If abnormal behavior is detected, the server generates a command to shut down the propulsion system and outputs it to the terminal. Specifically, it creates a signal that instructs the optimal deceleration and stopping process, taking into account the vehicle's current speed and location information. When this signal is received by the terminal, the stepwise deceleration procedure begins.

[0671] Step 4:

[0672] Based on instructions from the server, the terminal directly controls the propulsion system to switch the vehicle into semi-autonomous driving mode. Its input is the server's control instructions, and its output is a vehicle speed control signal. Specifically, the control unit is activated, adjusting the engine output and gradually bringing the vehicle to a safe stop.

[0673] Step 5:

[0674] After the vehicle stops, the server notifies the relevant facilities of the abnormal situation. The input at this time is the detected abnormal behavior and related information, and the output is an abnormal situation notification message which is sent through the appropriate communication channel. Specifically, data including location information and driver ID is automatically transmitted.

[0675] Step 6:

[0676] When restarting operation, the user has the terminal read their authentication information before starting the vehicle. The input is either a physical authentication card or digital information, which the terminal reads and sends to the server. The server receives this data and verifies its validity. The output is "Engine ready to start" if valid, or "Authentication failed" if invalid.

[0677] (Application Example 1)

[0678] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0679] In recent years, traffic accidents caused by drunk driving or abnormal behavior by drivers have become a social problem. Conventional technology has often been insufficient to detect abnormal behavior in advance and prevent accidents. Furthermore, verifying driver credentials before starting to drive is not always safe. In addition, there is a lack of means to continuously monitor the driver's condition while driving and issue appropriate warnings in real time. Therefore, further technologies are needed to reduce the risk of traffic accidents.

[0680] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0681] In this invention, the server includes means for analyzing video data acquired in real time by an image acquisition device installed in the vehicle to detect driver intoxication or abnormal behavior; means for shutting off the vehicle's power source and switching to a semi-autonomous driving mode when an abnormality is detected; and means for continuously monitoring the driver's condition while driving using a smart device equipped with voice and visual warning functions that detect the driver's facial expression and eye movements and issue warnings. This makes it possible to instantly detect abnormal driver behavior and prevent accidents.

[0682] An "image acquisition device" is a device installed inside a vehicle that acquires video data such as the driver's facial expression and eye movements in real time.

[0683] "Methods for real-time analysis" refer to methods for immediately processing acquired video data to detect driver intoxication or abnormal behavior.

[0684] "Methods for detecting abnormal behavior" refer to methods that use generative AI to analyze the driver's biometric information and detect signs such as alcohol consumption or fatigue.

[0685] "Means for stopping the power source" refers to means that, when an abnormality is detected in the driver, the vehicle's power mechanism can be controlled to stop the engine.

[0686] "Means of switching to semi-autonomous driving mode" refers to means of activating driver assistance functions that automatically and safely control the vehicle.

[0687] "Means of verifying qualifications" refers to the means of checking the driver's license or other identification before driving begins and evaluating its validity.

[0688] A "smart device" is a device equipped with a camera and microphone that provides information to the driver by issuing voice and visual warnings.

[0689] To implement this invention, first, an image acquisition device installed inside the vehicle is used to capture biometric information such as the driver's facial color and eye movements in real time. The image acquisition device is installed as a smart device and collects the driver's biometric information. The terminal transmits this video data to a server, where it is analyzed using a generated AI model.

[0690] The server uses machine learning libraries such as TensorFlow to analyze this video data and detect abnormal driver behavior (e.g., facial redness, unnatural eye movements). If an abnormality is detected, the server sends an instruction to the terminal to shut off the vehicle's power source, and the vehicle automatically switches to semi-autonomous driving mode.

[0691] Subsequently, the driver receives audio and visual warnings. For example, warnings such as "Please stop driving" or "Your face is red. Please check for alcohol intoxication" are transmitted through the smart device's speaker and display.

[0692] Through this process, the device continuously monitors the driver's facial expression and eye movements, and takes appropriate safety measures as needed based on instructions from the server. This significantly reduces the risk of traffic accidents.

[0693] An example of a prompt statement is, "Generate a risk prediction algorithm to detect abnormal driving behavior." This prompt statement is used to prompt the generated AI model to perform appropriate analysis.

[0694] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0695] Step 1:

[0696] The terminal captures biometric information, such as the driver's face and eye movements, in real time from an image acquisition device installed inside the vehicle. The input is video data, and the output is the captured video data itself. Specifically, the terminal periodically activates the image acquisition device and takes pictures of the driver.

[0697] Step 2:

[0698] The terminal transfers the acquired video data to the server. The input is the video data captured in step 1, and the output is the transmission of data to the server. Specifically, the terminal uploads the data to the server via the network.

[0699] Step 3:

[0700] The server analyzes the received video data using a generation AI model. The input is the transmitted video data, and the output is the result of the driver's abnormal behavior detection. Specifically, the server uses machine learning libraries such as TensorFlow to detect facial expression, eye movements, and unnatural behavior based on prompt messages.

[0701] Step 4:

[0702] If abnormal behavior is detected, the server sends an instruction to the terminal to issue a warning. The input is the abnormal behavior detection result obtained in step 3, and the output is the warning instruction. Specifically, the server sets up audio and visual warnings and instructs the terminal to execute them.

[0703] Step 5:

[0704] The terminal, based on instructions from the server, issues warnings to the driver via a smart device. The input is a warning instruction from the server, and the output is an audio and visual warning to the driver. Specifically, the terminal plays messages such as "Stop driving" or "Your face is red. Please check for alcohol intoxication."

[0705] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0706] The system of the present invention improves driver safety by combining an image acquisition device, a generating AI, and an emotion engine installed in a vehicle. The system is implemented in the following manner.

[0707] First, the device acquires video data in real time using cameras inside the vehicle. The acquired data is sent to a server and analyzed by a generative AI. The generative AI analyzes the driver's facial expression, eye movements, and behavior to quickly determine if there is any alcohol consumption or other abnormal behavior. It also uses an emotion engine to analyze the driver's emotional state—for example, stress or distraction.

[0708] If signs of abnormality or emotional instability are detected, the server sends an engine shutdown command to the terminal and switches to semi-autonomous driving mode, bringing the vehicle to a safe location. During this time, the server notifies the appropriate relevant authorities of the situation.

[0709] Furthermore, the system prevents inappropriate driving by requiring the user to present their driver's license before starting to drive, and verifying its validity on the server. Once driving begins, the server uses an emotion engine to provide the user with relaxation methods and warnings tailored to the driver's emotional state, via the terminal.

[0710] For example, when the emotional engine detects a high stress level while driving, the server sends a command to the terminal to change the in-car music to something calming or adjust the air conditioning to a comfortable setting. This adjustment helps to alleviate the driver's tension and provide a safer driving environment.

[0711] Thus, by comprehensively assessing the driver's condition, this system can significantly contribute to preventing traffic accidents.

[0712] The following describes the processing flow.

[0713] Step 1:

[0714] The terminal acquires video data of the driver in real time via the vehicle's camera and sends it to a server for processing.

[0715] Step 2:

[0716] To analyze the video data received by the server, a generating AI is used to analyze the driver's facial expression, eye movements, and other behaviors to determine whether or not they are intoxicated or exhibiting abnormal behavior.

[0717] Step 3:

[0718] Simultaneously, the server uses an emotion engine to analyze the driver's emotional state and detect emotional changes such as stress levels and attention deficits.

[0719] Step 4:

[0720] If abnormal behavior or unstable emotions are detected, the server sends a command to the terminal to shut off the engine and switch to semi-autonomous driving mode. Upon receiving this command, the terminal shuts off the vehicle's power source and moves the vehicle to a safe location.

[0721] Step 5:

[0722] After the server detects an anomaly, it automatically notifies the relevant organizations about the vehicle's current location and the driver's status.

[0723] Step 6:

[0724] Before starting the vehicle, the user presents their driver's license to the terminal, and the server verifies the validity of the license by checking the data. If the license is invalid, the terminal prevents the engine from starting.

[0725] Step 7:

[0726] The server uses an emotion engine to monitor the driver's emotions while the vehicle is in operation and provides optimal driving support based on their emotional state. This includes allowing the driver to change music selections and adjust air conditioning settings via their terminal.

[0727] Step 8:

[0728] The server generates music to help the driver relax and warning messages when necessary, and provides them to the user via the terminal.

[0729] This series of processes ensures a safe and smooth operating environment.

[0730] (Example 2)

[0731] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0732] One of the main causes of traffic accidents is driver inattention and unstable emotional states while driving. The challenge lies in preventing these dangers in advance by detecting these factors in real time. Furthermore, it is necessary to prevent unqualified drivers from operating vehicles and ensure driver safety.

[0733] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0734] This invention includes a server that analyzes video data acquired in real time by an image acquisition device installed in the vehicle to detect driver intoxication or abnormal behavior; an emotion analysis device that evaluates the driver's emotional state and provides relaxation methods or warnings according to the emotional state, such as stress; and a means that verifies the driver's credentials before driving begins and disables the vehicle if they are inappropriate or invalid. This makes it possible to monitor the driver's condition from multiple angles, avoid dangerous situations in advance, and ensure safety.

[0735] An "image acquisition device installed inside a vehicle" refers to a device installed inside a vehicle that includes cameras and sensors for acquiring video data of the driver.

[0736] "Generative AI" is an artificial intelligence technology used to analyze acquired video data and identify the driver's biometric information and behavior.

[0737] An "emotion analysis device" is a device that evaluates a driver's emotional state based on their facial expressions and biometric data, and determines their stress level, distractibility, and other factors.

[0738] "Semi-autonomous driving mode" is a driving mode in which the vehicle automatically controls itself to support safe driving.

[0739] "Qualification information" refers to the driver's license and other authentication information necessary for the driver to operate the vehicle.

[0740] "Drive system" refers to the collective term for equipment such as engines and motors necessary to propel a vehicle.

[0741] "Biometric information" refers to information about an individual's physical condition, such as the driver's complexion, eye condition, and behavior.

[0742] "Relaxation techniques" include methods such as changing the music or adjusting the air conditioning to promote safe driving in accordance with the driver's emotional state.

[0743] This invention is a system that improves driver safety by coordinating an image acquisition device, a generating AI, and an emotion analysis device installed inside a vehicle.

[0744] The server receives video data in real time from an image acquisition device installed inside the vehicle. This image acquisition device includes sensors such as cameras and has the function of acquiring the driver's facial color, eye movements, and behavior. The transmitted data is analyzed on the server, and the driver's biometric information is determined by a generating AI.

[0745] The generating AI analyzes video data to recognize the driver's drunk driving and other abnormal behavior. Furthermore, an emotion analysis device assesses the driver's emotional state, particularly detecting stress and distraction. This analysis utilizes advanced facial recognition technology and emotion analysis algorithms.

[0746] If an anomaly or danger is detected, the server sends an engine stop command to the terminal and switches to semi-autonomous driving mode. This allows the vehicle to automatically stop in a safe location. Details of the anomaly are then notified to the relevant organizations.

[0747] Before starting to drive, the user presents their driver's license as credentials, and its validity is verified by the server. This procedure makes it possible to prevent driving with inappropriate or invalid credentials.

[0748] While driving, the server provides relaxation methods tailored to the driver's emotional state based on data obtained from an emotion analysis device. This includes actions such as changing the music in the car or optimizing the air conditioning.

[0749] As a concrete example, if the emotion analysis device detects a high stress level in the driver, the server sends a command to the terminal to change the in-car background music to calming music. An example of the prompt message in this case is: "Analyze the driver's facial expression, eye movements, and behavior in real time, and notify if there are any abnormalities. In addition, use the emotion analysis device to evaluate the stress level and suggest adjustments."

[0750] This system aims to provide a safer driving environment and eliminate potential accident-causing factors early on by monitoring the driver's condition from multiple perspectives.

[0751] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0752] Step 1:

[0753] The terminal acquires real-time video data of the driver using cameras installed inside the vehicle. The input is video data from the in-vehicle cameras, which includes the driver's face, eye movements, and body movements. The output is the acquired raw video data. At this time, a portion of the acquired data is recorded as metadata and tagged for subsequent analysis processing.

[0754] Step 2:

[0755] The device sends the acquired video data to the server. The input is the video data acquired and tagged in step 1, and the output is the transmission of that data to the server. During the transmission process, the data is encrypted to protect privacy.

[0756] Step 3:

[0757] The server analyzes the received video data. The input is encrypted video data transmitted from the terminal. The generating AI uses facial recognition and motion analysis technology to analyze the driver's complexion, eye movements, and behavior. The output is the detection result of signs of intoxication or abnormal behavior in the driver. The analysis results are stored in a database and used for subsequent processing.

[0758] Step 4:

[0759] The server uses an emotion analysis device to evaluate the driver's emotional state. The input is the analysis results obtained in step 3. The emotion analysis device evaluates stress levels and distractibility based on the driver's facial expressions and biometric data. The output is the evaluation result of the emotional state, providing information indicating the need for relaxation or warning.

[0760] Step 5:

[0761] If an abnormality or emotional instability is detected, the server sends an engine stop command to the terminal. The input is the analysis and evaluation results from steps 3 and 4. The output is a command to switch to semi-autonomous driving mode. Specifically, a program to guide the vehicle to the optimal stopping point is activated.

[0762] Step 6:

[0763] Before starting the vehicle, the user presents their credentials to the terminal. The input is the credentials, such as a driver's license, presented by the user. The output is the result of the credentials being verified by the server. In this verification process, if no valid credentials are found, the server locks the vehicle from starting.

[0764] Step 7:

[0765] While driving, the server provides relaxation methods tailored to the driver's state based on information from an emotion analysis device. The input is the real-time evaluation of the driver's emotional state. The output is a command, such as changing music selection or adjusting the air conditioning settings. Specific actions include accessing a music library based on the driver's preferences and automatically adjusting the air conditioning system. Through this process, the system aims to alleviate driver stress and ensure a safe driving environment.

[0766] (Application Example 2)

[0767] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0768] Conventional vehicle safety systems focus on detecting driver intoxication and abnormal behavior, but are insufficient in addressing potential hazards based on the driver's emotional state. Furthermore, if a driver's emotions become unstable while driving, there is no means to appropriately adjust the environment, leaving the risk of accidents unresolved. Moreover, even if a driver is stressed or distracted, there is no system to quickly detect and appropriately address these conditions, resulting in a situation that cannot adequately support safe driving.

[0769] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0770] In this invention, the server includes means for analyzing video data acquired in real time by an image acquisition device installed in the vehicle to detect driver intoxication or abnormal behavior; means for shutting off the vehicle's power source and switching to a semi-autonomous driving mode when an abnormality is detected by the means; and means for analyzing the driver's emotional state, using an emotion engine to detect stress or distraction, and adjusting the driving environment based on this. This makes it possible to immediately detect potential dangers based on the driver's emotional state and support safe and comfortable driving.

[0771] An "image acquisition device" is a device installed inside a vehicle that acquires video data of the driver in real time.

[0772] "Generative AI" is an artificial intelligence technology that analyzes a driver's facial expression, eye movements, and behavior based on the driver's biometric information.

[0773] The "emotional engine" is a program that analyzes the driver's emotional state to detect stress and distraction.

[0774] "Semi-autonomous driving mode" is a driving mode that assists the driver's operation when an abnormality is detected and helps the vehicle stop in a safe location.

[0775] "Qualification information" refers to the licenses and other authorizations required for a driver to operate a vehicle.

[0776] "Dangerous driving" refers to a situation where a driver's actions pose a level of risk far beyond what is considered normal driving, and detecting such situations is crucial.

[0777] "Adjusting the driving environment" refers to the act of changing the in-car sound system and air conditioning according to the driver's emotional state, thereby optimizing the driver's condition.

[0778] To implement this invention, an image acquisition device is installed inside the vehicle, and a system is configured to capture video data of the driver in real time. The server analyzes the acquired video data using various software and evaluates the driver's condition. Face recognition is performed using OpenCV for the analysis, and a generated AI using TensorFlow analyzes biometric information such as the driver's facial color, eye movements, and behavior.

[0779] The server further utilizes an emotion engine to analyze the driver's emotional state, such as stress and distraction. If abnormal behavior or an unstable emotional state is detected, it switches to a semi-autonomous driving mode in conjunction with the vehicle's control system and brings the vehicle to a safe stop. In addition, the server can adjust the in-car sound and air conditioning settings according to the user's emotional state. At this time, it also provides the user with suggestions for relaxation methods and behavioral improvements.

[0780] For example, if the emotion engine determines that the user's stress level is rising during a long drive, the server will change the in-car music to relaxing classical music and automatically adjust the air conditioning to a comfortable temperature. It will also send a notification to the user's device suggesting they take a short break. The prompt sent to the AI ​​model in this case would be something like, "The driver's eyes are closed for longer than usual. Are you experiencing increased tension or stress?"

[0781] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0782] Step 1:

[0783] The terminal captures video data of the driver in real time using an image acquisition device inside the vehicle. The input is video of the driver inside the vehicle, which is obtained as output data in image format.

[0784] Step 2:

[0785] The server receives video data transmitted from the terminal and performs face recognition using OpenCV. The input data is video data of the driver, which is analyzed to identify the driver's face region. The output is the recognized face image and its coordinate information.

[0786] Step 3:

[0787] The server inputs the facial region data recognized by face recognition into a generative AI model using TensorFlow, which analyzes the driver's complexion, eye movements, and behavior. The input is facial region data, and the output is an evaluation of the driver's state based on the analyzed biometric information.

[0788] Step 4:

[0789] The server uses an emotion engine to assess the driver's emotional state, specifically their stress level and degree of distraction. The input is the driver's state assessment result from a generative AI model, and the output is an index indicating the emotional state.

[0790] Step 5:

[0791] If an emotional state or abnormality is detected, the server sends a signal to the vehicle's control system and switches to semi-autonomous driving mode. The input is the result of the abnormality detection, and the output is the control signal to the vehicle's control system.

[0792] Step 6:

[0793] The server sends signals to the terminal to adjust the in-car sound and air conditioning based on the user's emotional state. The input is an indicator of the emotion engine, and the output is a signal instructing adjustments.

[0794] Step 7:

[0795] The terminal changes the sound and air conditioning settings inside the vehicle based on instructions received from the server. The input here is the adjustment instructions from the server, and the output is the specific action to be performed, such as switching music or adjusting the temperature settings.

[0796] Step 8:

[0797] The server sends notifications to the user regarding detected anomalies and emotional states. It alerts the user by sending prompts such as "We recommend you take a short break." The input is the result of the emotion engine's analysis, and the output is the notification content sent to the user.

[0798] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0799] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0800] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0801] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0802] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0803] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0804] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0805] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0806] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0807] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0808] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0809] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0810] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0811] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0812] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0813] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0814] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0815] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0816] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0817] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0818] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0819] The following is further disclosed regarding the embodiments described above.

[0820] (Claim 1)

[0821] A means for detecting driver intoxication or abnormal behavior by analyzing video data acquired in real time by an image acquisition device installed inside the vehicle,

[0822] The means for stopping the vehicle's power source and switching to a semi-autonomous driving mode when an abnormality is detected by the means described above,

[0823] After detecting the aforementioned anomaly, the vehicle is stopped in a safe location, and the anomaly is automatically notified to the relevant authorities.

[0824] Before starting the vehicle, the system verifies the driver's qualifications and provides a means to prevent the vehicle from starting if the information is inappropriate or invalid.

[0825] A means of monitoring the driver's behavior while driving and providing warnings or instructions for improvement if dangerous driving is detected,

[0826] A system that includes this.

[0827] (Claim 2)

[0828] The system according to claim 1, wherein the image acquisition device is equipped with a generating AI for analyzing biometric information including the driver's facial color, eye condition, and behavior.

[0829] (Claim 3)

[0830] The system according to claim 1, wherein the means for stopping the power source of the vehicle is to perform a stepwise deceleration operation before stopping, taking into consideration the safety of the driver.

[0831] "Example 1"

[0832] (Claim 1)

[0833] A means for detecting driver intoxication or abnormal behavior by analyzing video data acquired in real time by a camera installed inside the vehicle,

[0834] A means for stopping the vehicle's propulsion system and switching to a semi-autonomous driving mode when an abnormality is detected by the aforementioned means,

[0835] After detecting the aforementioned anomaly, the system includes means for stopping the vehicle in a safe area and automatically notifying the relevant facilities of the anomaly,

[0836] Before starting the vehicle, the system verifies the driver's authentication information and has a mechanism to prevent the vehicle from starting if it is inappropriate or invalid.

[0837] A means of monitoring the driver's behavior while driving and providing warnings or instructions for improvement if dangerous driving is detected,

[0838] A means of analyzing the driver's biological characteristics by analyzing image data using a generative AI model,

[0839] A means for transmitting signals to a terminal to gradually decelerate and safely stop the vehicle,

[0840] A system that includes this.

[0841] (Claim 2)

[0842] The system according to claim 1, which uses video data to monitor the driver's behavior and provides real-time warnings against dangerous driving.

[0843] (Claim 3)

[0844] The system according to claim 1, wherein the generating AI model uses prompt statements to analyze the driver's eye movements, facial color, and other biometric information.

[0845] "Application Example 1"

[0846] (Claim 1)

[0847] A means for detecting driver intoxication or abnormal behavior by analyzing video data acquired in real time by an image acquisition device installed inside the vehicle,

[0848] The means for stopping the vehicle's power source and switching to a semi-autonomous driving mode when an abnormality is detected by the means described above,

[0849] After detecting the aforementioned anomaly, the vehicle is stopped in a safe location, and the anomaly is automatically notified to the relevant authorities.

[0850] Before starting the vehicle, the system verifies the driver's qualifications and provides a means to prevent the vehicle from starting if the information is inappropriate or invalid.

[0851] A means for continuously monitoring the driver's condition while driving, using a smart device equipped with voice and visual warning functions that detect and warn the driver's facial expression and eye movements,

[0852] A system that includes this.

[0853] (Claim 2)

[0854] The system according to claim 1, wherein the image acquisition device is equipped with a generating AI for analyzing biometric information including the driver's facial color, eye condition, and behavior, and provides voice and visual warnings.

[0855] (Claim 3)

[0856] The system according to claim 1, wherein the means for stopping the power source of the vehicle performs a stepwise deceleration operation before stopping, taking into consideration the safety of the driver, and simultaneously prompts the driver to prepare to stop with an audible voice.

[0857] "Example 2 of combining an emotion engine"

[0858] (Claim 1)

[0859] A means for detecting driver intoxication or abnormal behavior by analyzing video data acquired in real time by an image acquisition device installed inside the vehicle,

[0860] The means for stopping the vehicle's drive system and switching to a semi-autonomous driving mode when an abnormality is detected by the means described above,

[0861] After detecting the aforementioned anomaly, the system includes means for stopping the vehicle in a safe location and automatically notifying the relevant organization of the anomaly,

[0862] Before starting the vehicle, the system verifies the driver's qualifications and provides a means to prevent the vehicle from starting if the information is inappropriate or invalid.

[0863] A means of monitoring the driver's behavior while driving and providing warnings or instructions for improvement if dangerous driving is detected,

[0864] A means of evaluating the driver's emotional state using an emotion analysis device and providing relaxation methods and warnings according to the emotional state, such as stress,

[0865] A system that includes this.

[0866] (Claim 2)

[0867] The system according to claim 1, wherein the image acquisition device is equipped with a generative AI for analyzing biometric information including the driver's facial color, eye condition, and behavior, and the emotion analysis device evaluates the driver's stress level.

[0868] (Claim 3)

[0869] The system according to claim 1, wherein the means for stopping the vehicle's drive system performs a stepwise deceleration operation before stopping, taking into consideration the driver's safety, and adaptively adjusts the driver's state using an emotion analysis device.

[0870] "Application example 2 when combining with an emotional engine"

[0871] (Claim 1)

[0872] A means for detecting driver intoxication or abnormal behavior by analyzing video data acquired in real time by an image acquisition device installed inside the vehicle,

[0873] The means for stopping the vehicle's power source and switching to a semi-autonomous driving mode when an abnormality is detected by the means described above,

[0874] After detecting the aforementioned anomaly, the vehicle is stopped in a safe location, and the anomaly is automatically notified to the relevant authorities.

[0875] Before starting the vehicle, the system verifies the driver's qualifications and provides a means to prevent the vehicle from starting if the information is inappropriate or invalid.

[0876] A means of monitoring the driver's behavior while driving and providing warnings or instructions for improvement if dangerous driving is detected,

[0877] A means for analyzing the driver's emotional state, using an emotion engine to detect stress and distraction, and adjusting the driving environment based on this,

[0878] A system that includes this.

[0879] (Claim 2)

[0880] The system according to claim 1, wherein the image acquisition device is equipped with a generative AI for analyzing biometric information including the driver's facial color, eye condition, and behavior, and has a function to adjust the in-car sound and air conditioning when an unstable emotional state is detected.

[0881] (Claim 3)

[0882] The system according to claim 1, wherein the means for stopping the power source of the vehicle performs a gradual deceleration operation before stopping, taking into consideration the driver's safety, and also notifies the driver of how to relax. [Explanation of Symbols]

[0883] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for detecting driver intoxication or abnormal behavior by analyzing video data acquired in real time by an image acquisition device installed inside the vehicle, The means for stopping the vehicle's power source and switching to a semi-autonomous driving mode when an abnormality is detected by the means described above, After detecting the aforementioned anomaly, the vehicle is stopped in a safe location, and the anomaly is automatically notified to the relevant authorities. Before starting the vehicle, the system verifies the driver's qualifications and provides a means to prevent the vehicle from starting if the information is inappropriate or invalid. A means of monitoring the driver's behavior while driving and providing warnings or instructions for improvement if dangerous driving is detected, A system that includes this.

2. The system according to claim 1, wherein the image acquisition device is equipped with a generating AI for analyzing biological information including the driver's facial color, eye condition, and behavior.

3. The system according to claim 1, wherein the means for stopping the power source of the vehicle performs a stepwise deceleration operation before stopping, taking into consideration the safety of the driver.

Images