System

Abstract

A system is provided.SOLUTION: A system including a terminal mounted in a vehicle, the terminal including means for collecting video data and audio data in real time, means for preprocessing the video data and the audio data collected by the terminal, a server that receives the preprocessed data and uses an artificial intelligence model for analyzing a state of a driver, means for notifying a driver, a surrounding vehicle, and a base of danger information based on an analysis result, means for storing the analysis result and driving data, means for generating an optimal driving pattern based on the stored driving data, and means for applying the generated driving pattern to a vehicle system.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In the logistics industry, labor shortages and long working hours have become the norm, making ensuring safe driving a major issue. Driver fatigue and dangerous driving pose a particularly high risk of serious accidents. To solve this problem and ensure driver safety, a system is needed that monitors the driver's condition in real time and issues warnings or instructions to take breaks at the appropriate time. However, conventional drive recorder systems often fail to accurately grasp the driver's condition, preventing appropriate measures from being taken. The present invention aims to solve these issues and improve driver safety. [Means for solving the problem]

[0005] The present invention provides a system including a terminal mounted on a vehicle, which includes: means for collecting video data and audio data in real time; means for preprocessing the video data and audio data collected by the terminal; a server that receives the preprocessed data and uses an artificial intelligence model to analyze the driver's condition; means for notifying the driver, surrounding vehicles, and base stations of hazard information based on the analysis results; means for storing the analysis results and driving data; means for generating an optimal driving pattern based on the stored driving data; and means for applying the generated driving pattern to the vehicle system. This system enables real-time monitoring of the driver's condition and the implementation of appropriate countermeasures. For example, if signs of fatigue are detected by analyzing the driver's facial features, a warning can be issued to the driver, surrounding vehicles, and base stations. Furthermore, by analyzing past driving data and generating an optimal driving pattern, it is possible to reduce the driver's burden and support safe driving.

[0006] "Terminal" refers to a device installed in a vehicle that collects and pre-processes video and audio data in real time.

[0007] "Video data" refers to data that includes visual information acquired using an optical device such as a camera.

[0008] "Audio data" refers to data containing auditory information obtained using an acoustic device such as a microphone.

[0009] "Preprocessing" refers to converting collected data into a format that is easier for an AI model to analyze, such as converting video data to grayscale or audio data into a suitable buffer.

[0010] The "artificial intelligence model" is a model that analyzes the driver's state using machine learning and deep learning technologies.

[0011] A "server" is a computer system located at a remote location that receives the pre-processed data and analyzes the driver's condition using artificial intelligence models.

[0012] "Analysis results" refers to information obtained as a result of analysis by the artificial intelligence model, and includes the driver's level of fatigue and signs of dangerous driving.

[0013] "Notification" refers to the act of sending information to the driver, surrounding vehicles, and base stations in real time based on the analysis results.

[0014] "Driving data" refers to information that includes all data collected while driving (video data, audio data, analysis results, etc.).

[0015] "Optimal driving pattern" refers to the most efficient and safe driving method for a driver, obtained by analyzing collected driving data.

[0016] A "vehicle system" is a system that includes various devices and functions for controlling driving within a vehicle. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] This invention is a system in which a terminal installed in a vehicle collects video and audio data in real time and preprocesses the data. The preprocessed data is sent to a server, where the driver's condition is analyzed using an artificial intelligence model. Based on the analysis results, danger information is then sent to the driver, surrounding vehicles, and base stations. The server also stores the driving data, generates optimal driving patterns, and reflects them in the vehicle system. This system makes it possible to monitor the driver's safety in real time and support safe driving.

[0039] Data collection and preprocessing

[0040] The terminal collects video and audio data in real time using the camera and microphone installed in the vehicle. The collected data is first pre-processed on the terminal. Specifically, video data is converted to grayscale and audio data is converted to a buffer format. This pre-processing makes the data suitable for analysis by artificial intelligence models.

[0041] Analysis by AI model

[0042] The server receives the preprocessed data and inputs it into an AI model, which uses deep learning technology to analyze the driver's facial features and voice patterns to detect signs of fatigue or dangerous driving, such as the frequency with which the driver opens and closes their eyes, facial expressions, and the tone and rhythm of their voice.

[0043] Risk information notification

[0044] Based on the analysis results of the AI ​​model, the server issues a real-time warning if the driver is fatigued or if dangerous driving behavior is detected. Specifically, a warning message is displayed on the driver's device and an audio alert is given. The server also warns surrounding vehicles via wireless communication and sends a situation report to a base (e.g., a logistics center).

[0045] Data storage and generation of optimal driving patterns

[0046] The server stores all collected and analyzed driving data. This stored data is used for later analysis. Specifically, it statistically analyzes past driving data to generate an optimal driving pattern for the driver. This optimal driving pattern is then applied to the vehicle system to reduce driver fatigue and promote safe driving.

[0047] Specific examples

[0048] For example, if a driver has been driving continuously for a long period of time, the device will detect signs of fatigue from their facial expressions and voice. The collected data is sent to a server, where an artificial intelligence model analyzes it. If fatigue is confirmed, the server will issue a warning to the driver saying, "Fatigue has been detected. Please take a break." At the same time, a message is sent to surrounding vehicles saying, "There is a possibility of dangerous driving. Please be careful," and a report is sent to the base saying, "Driver ID 123 is in a state of fatigue, so please take appropriate action." The server will also use the collected driving data to generate optimal driving patterns for the driver in the future and apply them to the vehicle system, further improving driver safety.

[0049] The processing flow will be explained below.

[0050] Step 1:

[0051] The device collects video and audio data inside the vehicle. Specifically, a camera installed inside the vehicle captures the driver's video and a microphone records the driver's voice. This data is acquired in real time.

[0052] Step 2:

[0053] The terminal preprocesses the collected video and audio data. Specifically, it converts video data to grayscale and audio data to an appropriate format (e.g., integer buffer). This completes the preprocessing required for subsequent analysis.

[0054] Step 3:

[0055] The device sends the pre-processed data to the server, where it is encrypted for secure transmission and uploaded to the server, ready for real-time driver status monitoring.

[0056] Step 4:

[0057] The server inputs the received pre-processed data into an AI model for analysis. The AI ​​model uses deep learning technology to analyze the driver's facial features and voice patterns, thereby detecting signs of driver fatigue and dangerous driving.

[0058] Step 5:

[0059] The server determines risk information based on the analysis results of the AI ​​model. Specifically, if the fatigue score or dangerous driving score indicated by the model exceeds a certain threshold, it determines that the driver is fatigued.

[0060] Step 6:

[0061] The server issues a warning to the driver based on the results of the assessment. Specifically, it sends a notification message to the device saying, "Fatigue has been detected. Please take a break," and issues a voice alert. It also sends wireless communications to nearby vehicles to warn them.

[0062] Step 7:

[0063] The server reports the driver's condition to the base. Specifically, it sends a notification saying, "Driver ID 123 is in a fatigued state, so please take appropriate action." This allows the base to understand the driver's condition in real time.

[0064] Step 8:

[0065] The server stores the collected and analyzed driving data, which is then used for later analysis and optimization. Specifically, the driving data is stored in a database and later statistically analyzed.

[0066] Step 9:

[0067] The server generates optimal driving patterns based on the stored driving data. Specifically, it analyzes past driving data and derives efficient and safe driving methods for the driver.

[0068] Step 10:

[0069] The server applies the generated optimal driving pattern to the vehicle system. Specifically, the pattern is reflected in the vehicle's various control systems, reducing the burden on the driver and supporting safe driving.

[0070] Example 1

[0071] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0072] Conventional vehicle driving monitoring systems have had difficulty accurately grasping the driver's condition in real time and issuing appropriate warnings or notifications. In particular, there is a lack of technology that can quickly detect signs of driver fatigue or dangerous driving and reliably issue warnings, making it difficult to prevent serious traffic accidents.

[0073] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0074] In this invention, the server includes means for receiving the preprocessed data and using a generative AI model to analyze the driver's condition, means for notifying the driver, surrounding vehicles, and bases of danger information based on the analysis results, means for saving the analysis results and driving data, means for analyzing the driver's eye opening and closing frequency and facial expressions from the video data, and means for analyzing the tone and rhythm of the voice from the audio data. This makes it possible to accurately grasp the driver's condition in real time and issue warnings and notifications quickly and appropriately.

[0075] A "terminal" is a device that is installed in a vehicle and collects video data and audio data in real time.

[0076] The "server" is a central computing device that receives the pre-processed data, analyzes the driver's state using a generative AI model, and issues warnings or notifications based on the results.

[0077] A "generative AI model" is a type of artificial intelligence used to analyze video and audio data of a driver and evaluate the driver's condition.

[0078] The "pre-processing means" is a process that converts the collected video data to grayscale and converts the audio data into a suitable buffer format.

[0079] "Means of analysis" refers to the process of inputting the collected pre-processed data into an AI model and evaluating the driver's condition by analyzing the frequency of eye opening and closing, facial expressions, and tone and rhythm of voice.

[0080] "Means for notifying danger information" refers to a means of sending warning messages and alerts to drivers, surrounding vehicles, and bases such as logistics centers based on the analysis results of the AI ​​model.

[0081] "Means for storing data" refers to the process of storing the analyzed driving data in a database or the like and using it for later analysis or for generating optimal driving patterns.

[0082] The "means for generating optimal driving patterns" is a process of statistically analyzing past driving data to derive driving patterns that reduce driver fatigue and promote safe driving.

[0083] The "means for applying the driving pattern" is a process for reflecting the generated optimal driving pattern in the vehicle system.

[0084] "Collecting in real time" refers to obtaining video and audio data instantly without any specific time delay during driving.

[0085] "Video data" refers to visual information that records the inside of a vehicle and the driver's face and expressions.

[0086] "Voice data" refers to auditory information that records the acoustic environment inside the vehicle and the driver's voice.

[0087] "Converting to grayscale" is the process of converting color video data to black and white brightness only.

[0088] "Converting to a buffer format" is the process of temporarily storing audio data and later converting it into a format that is easy to analyze and transmit.

[0089] "Driver status" is an assessment of the driver's current state, based on an analysis of their fatigue, attention, facial expressions, and voice patterns, and is necessary for driving behavior.

[0090] This invention is a system in which a terminal installed in a vehicle collects video and audio data in real time and preprocesses that data. The preprocessed data is sent to a server, where the driver's condition is analyzed using a generative AI model. Based on the analysis results, danger information is then sent to the driver, surrounding vehicles, and base stations. The server also stores the driving data, generates optimal driving patterns, and reflects them in the vehicle system. This system makes it possible to monitor the driver's safety in real time and support safe driving.

[0091] Data collection and preprocessing

[0092] The device uses a camera and microphone installed in the vehicle to collect video and audio data in real time. The video data is captured mainly of the driver's face, while the audio data collects the driver's speech and surrounding sounds. The collected video data is converted to grayscale, and the audio data is converted to a buffer format, making the data suitable for analysis by the generative AI model.

[0093] Examples:

[0094] A camera records the driver's facial expressions in real time, and a microphone picks up the driver's tone of voice.

[0095] The collected color data is quickly converted to grayscale data, and the audio data is stored in a buffer every second.

[0096] Analysis by AI model

[0097] The server receives the preprocessed data and inputs it into a generative AI model. The generative AI model uses deep learning techniques to analyze the driver's facial features and voice patterns to detect signs of driver fatigue or unsafe driving. For example, it analyzes the frequency of the driver's eye opening and closing, facial expressions, and voice tone and rhythm.

[0098] Examples:

[0099] The server analyzes the frequency with which the driver opens and closes their eyelids from the video data to detect signs of fatigue.

[0100] Changes in the tone of the driver's voice are analyzed from audio data to detect signs of lack of attention.

[0101] Example prompts for generative AI models:

[0102] "Analyze pre-processed video and audio data to detect driver fatigue from facial expressions."

[0103] Risk information notification

[0104] Based on the analysis results of the generative AI model, the server issues a real-time warning if the driver is fatigued or if dangerous driving behavior is detected. Specifically, a warning message is displayed on the driver's device and an audio alert is given. The server also warns surrounding vehicles via wireless communication and sends situation reports to bases such as logistics centers.

[0105] Examples:

[0106] The server warns the driver, "Signs of dangerous driving have been detected. Please take a break."

[0107] Sends a message to surrounding vehicles saying, "Possible dangerous driving. Please be careful."

[0108] Example prompts for generative AI models:

[0109] "Generate a warning message to encourage drivers to take a break."

[0110] Data storage and generation of optimal driving patterns

[0111] The server stores all collected and analyzed driving data. This stored data is used for later analysis. Specifically, it statistically analyzes past driving data to generate an optimal driving pattern for the driver. This optimal driving pattern is then applied to the vehicle system to reduce driver fatigue and promote safe driving.

[0112] Examples:

[0113] The server stores each driver's analytical data in a driving database and identifies them with a unique ID.

[0114] Past driving data is analyzed to generate optimal driving patterns, such as "increasing rest time," and these are reflected in the vehicle's navigation system.

[0115] Example prompts for generative AI models:

[0116] "Analyze past driving data and generate optimal driving patterns."

[0117] The above is an embodiment of the present invention. This system can accurately grasp the driver's condition in real time and provide necessary warnings and optimizations to support safe driving.

[0118] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0119] (Program processing flow)

[0120] Step 1:

[0121] The device activates the camera and microphone installed in the vehicle and collects video and audio data in real time.

[0122] Input: In-car video and audio

[0123] Specific behavior:

[0124] The camera captures footage centered on the driver's face.

[0125] A microphone collects audio, including the driver's voice.

[0126] Output: Collected video and audio data

[0127] Step 2:

[0128] The terminal converts the collected video data into grayscale and converts the audio data into a buffer format.

[0129] Input: Collected video and audio data

[0130] Specific behavior:

[0131] Converts video data to grayscale.

[0132] The audio data is divided into fixed time intervals and stored in a buffer.

[0133] Output: Grayscale video data and buffered audio data

[0134] Step 3:

[0135] The terminal transmits the pre-processed data to the server.

[0136] Input: Grayscale video data and buffered audio data

[0137] Specific behavior:

[0138] Upload data to the server using Wi-Fi or mobile data.

[0139] You will receive a confirmation message of the submission.

[0140] Output: Preprocessed data sent to the server

[0141] Step 4:

[0142] The server inputs the received pre-processed data into the generative AI model and analyzes the driver's condition.

[0143] Input: Preprocessed data (grayscaled video data and buffered audio data)

[0144] Specific behavior:

[0145] Send the preprocessed data to the generative AI model.

[0146] The generative AI model analyzes the driver's eye opening and closing frequency, facial expressions, and voice tone and rhythm.

[0147] Output: Driver status analysis results

[0148] Step 5:

[0149] Based on the analysis results of the generative AI model, the server issues a warning in real time if the driver is fatigued or if dangerous driving behavior is detected.

[0150] Input: Driver condition analysis results

[0151] Specific behavior:

[0152] A warning message is sent to the driver's device.

[0153] Issues an audio alert.

[0154] Alerts are sent to surrounding vehicles and bases via radio communication.

[0155] Outputs: Warning messages, audio alerts, and wireless communication notifications

[0156] Step 6:

[0157] The server stores all collected and analyzed driving data.

[0158] Input: Driver status analysis results and other driving data

[0159] Specific behavior:

[0160] Save the analyzed data in a database.

[0161] A unique ID is assigned to the stored data so that it can be searched for later.

[0162] Output: Saved operating data

[0163] Step 7:

[0164] The server statistically analyzes past driving data based on the stored data and generates an optimal driving pattern for the driver.

[0165] Input: Saved historical driving data

[0166] Specific behavior:

[0167] Analyze historical data using statistical analysis algorithms.

[0168] Generate optimal driving patterns and provide feedback to vehicle systems.

[0169] Output: Generated optimal driving pattern

[0170] This allows the system to accurately grasp the driver's condition in real time and provide prompt and appropriate warnings and optimizations.

[0171] (Application example 1)

[0172] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0173] Conventional in-vehicle driver monitoring systems have difficulty accurately assessing the driver's condition in real time and issuing appropriate warnings. Furthermore, the lack of a system that monitors the driver's condition and sends appropriate alerts is a problem, particularly in autonomous vehicles. The present invention aims to solve these problems and provide a system that ensures the safety of the driver and surrounding vehicles.

[0174] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0175] In this invention, the server includes a means for receiving the preprocessed data and using an artificial intelligence model to analyze the driver's condition, a means for notifying the driver, surrounding vehicles, and bases of danger information based on the analysis results, and a means for storing the analysis results and driving data. This allows the server to monitor the driver's condition in real time and issue appropriate warnings to ensure the safety of the driver and those around them.

[0176] A "terminal" is a device that is installed in a vehicle and collects video and audio data in real time and performs preprocessing.

[0177] "Video data" refers to video information captured using a camera showing the conditions inside and outside the vehicle.

[0178] "Audio data" refers to audio information obtained using a microphone of the audio environment inside the vehicle, particularly the driver's voice and surrounding noise.

[0179] "Pre-processing" refers to the process of converting the collected video data to grayscale and converting the audio data into a suitable buffer format.

[0180] The "server" is a central control computer that receives the pre-processed data and analyzes the driver's condition using artificial intelligence models.

[0181] An "artificial intelligence model" is a software algorithm that uses deep learning techniques to analyze data and analyse the driver's facial features and voice patterns.

[0182] "Analysis results" are information derived from artificial intelligence models about driver fatigue and signs of dangerous driving.

[0183] "Notification" refers to the act of the server transmitting danger information to the driver, surrounding vehicles, and base stations based on the analysis results.

[0184] "Storage" refers to the act of recording the analysis results and operational data in a database or the like for use in later analysis.

[0185] A "driving pattern" is a guideline that indicates the optimal driving method for a driver, generated based on past driving data.

[0186] "Smart glasses" are wearable devices that are installed in the driver's seat and collect video and audio data.

[0187] A "deep learning model" is an algorithm based on a neural network with multiple layers that runs on a server.

[0188] An "alert" is a warning or caution message to the driver that is conveyed through audio or visual means.

[0189] The embodiments of the present invention will be described in detail below.

[0190] System configuration

[0191] The system of the present invention mainly comprises the following elements:

[0192] Terminals installed in vehicles

[0193] server

[0194] Smart Glasses

[0195] Deep Learning Model

[0196] Data collection and preprocessing

[0197] The terminal collects video and audio data in real time using the vehicle's onboard camera and microphone. The smart glasses acquire and preprocess the data, converting video data to grayscale and audio data to a suitable buffer format.

[0198] Data transmission and analysis

[0199] The preprocessed data is sent to a server via a computer installed inside the vehicle. The server then analyzes the received data using a deep learning model. This model analyzes the driver's facial features and voice patterns to detect signs of fatigue or dangerous driving. The artificial intelligence model uses frameworks such as TENSORFLOW (registered trademark) and PyTorch.

[0200] Risk information notification

[0201] Based on the analysis results of the AI ​​model, the server issues a real-time warning if the driver is fatigued or if dangerous driving behavior is detected. Specifically, a warning message is displayed on the driver's device and an audio alert is given. The server also warns surrounding vehicles via wireless communication and sends a situation report to the base station.

[0202] Data storage and driving pattern generation

[0203] The server stores all collected and analyzed driving data, using cloud storage and databases. This stored data is then used for further analysis. Specifically, it statistically analyzes past driving data and generates an optimal driving pattern for the driver. This driving pattern is then applied to the vehicle systems to reduce driver fatigue and promote safe driving.

[0204] Specific examples

[0205] For example, if a driver has been driving continuously for a long period of time, the smart glasses will detect changes in the driver's facial expression and voice and send the data to a server. The server will then use an artificial intelligence model to analyze the data and, if it determines that the driver is fatigued, will issue a warning to the driver saying, "Fatigue has been detected. Please take a break." At the same time, a message will be sent to surrounding vehicles saying, "Possible dangerous driving. Please be careful," and a report will be sent to the base saying, "Driver ID 123 is fatigued, so please take appropriate action." The server will also use the collected driving data to generate optimal driving patterns for the driver in the future and apply them to the vehicle system, further improving safety.

[0206] Prompt Sentence Examples

[0207] An example prompt is:

[0208] "Please build an AI model system that uses the driver's facial features and voice patterns to analyze their fatigue level and risky driving behavior in real time and issue a warning to the vehicle system. Based on the invention, data will be collected from the camera and microphone and preprocessed into grayscale images and buffered audio data. Then, the data will be sent to the server and analyzed by deep learning. Please also provide specific instructions and examples of warning messages."

[0209] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0210] Step 1:

[0211] The device uses a camera and microphone to collect video and audio data from inside the vehicle in real time.

[0212] Input: In-vehicle video and audio

[0213] Output: Raw collected data (video and audio)

[0214] Specifically, the device activates the camera and microphone to continuously capture the driver's facial expressions and voice.

[0215] Step 2:

[0216] The terminal converts the collected video data to grayscale and converts the audio data to an appropriate buffer format.

[0217] Input: Raw collected data (video and audio)

[0218] Output: Preprocessed data (grayscale image and audio in buffer format)

[0219] Specifically, the video data is converted to grayscale using the OpenCV library, and the audio data is converted using Python's buffer function.

[0220] Step 3:

[0221] The pre-processed data is sent to a server via a computer inside the vehicle.

[0222] Input: Preprocessed data (grayscale images and audio in buffer format)

[0223] Output: Data sent to the server

[0224] Specifically, the preprocessed data is uploaded to the server using an HTTP POST request.

[0225] Step 4:

[0226] The data received by the server is input into an artificial intelligence model (deep learning model) to analyze the driver's condition.

[0227] Input: Received data (grayscale image and buffered audio)

[0228] Output: Analysis results (status information based on the driver's facial features and voice patterns)

[0229] Specifically, it analyzes data using deep learning frameworks such as TensorFlow and PyTorch to detect signs of fatigue and dangerous driving.

[0230] Step 5:

[0231] Based on the analysis results, the server displays a warning message on the driver's device and issues an audio alert, while also wirelessly alerting surrounding vehicles and sending a situation report to the base.

[0232] Input: Analysis results

[0233] Outputs: Warning messages, audio alerts, radio communication messages, situation reports

[0234] Specifically, it alerts the driver using SMS or application notifications, alerts surrounding vehicles using wireless communication protocols, and sends reports to a base station via email or dashboard.

[0235] Step 6:

[0236] The server stores the analysis results and operational data in a database.

[0237] Input: Analysis results and operating data

[0238] Output: Saved data

[0239] Specifically, the data is recorded in cloud storage or a relational database.

[0240] Step 7:

[0241] The server statistically analyzes the stored driving data, generates an optimal driving pattern for the driver, and applies it to the vehicle system.

[0242] Input: Saved data

[0243] Output: Optimal driving pattern, applied driving pattern

[0244] Specifically, it uses machine learning algorithms to analyze data, form driving patterns, and push them to the vehicle's systems for application.

[0245] The above are the specific processing steps of the system that realizes the application example.

[0246] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0247] This invention is a system in which a terminal installed in a vehicle collects video and audio data in real time and preprocesses that data. The preprocessed data is sent to a server, where the driver's condition is analyzed using an artificial intelligence model and an emotion engine. Based on the analysis results, danger information is then sent to the driver, surrounding vehicles, and base stations. The server also stores the driving data, generates optimal driving patterns, and reflects them in the vehicle system. This system makes it possible to monitor the driver's condition in real time and support safe driving.

[0248] Data collection and preprocessing

[0249] The terminal collects video and audio data in real time using the camera and microphone installed in the vehicle. The collected data is pre-processed on the terminal. Specifically, the video data is converted to grayscale and the audio data is converted to a buffer format. This completes the pre-processing required for subsequent analysis.

[0250] Analysis by AI models and sentiment engines

[0251] The server receives the preprocessed data and inputs it into the AI ​​model and emotion engine. The AI ​​model uses deep learning technology to analyze the driver's facial features and voice patterns to detect signs of fatigue or dangerous driving. Meanwhile, the emotion engine analyzes the driver's emotions from video and audio data. For example, it analyzes the driver's facial expressions and tone of voice to recognize emotions such as anger, joy, and sadness.

[0252] Risk information notification

[0253] Based on the analysis results of the AI ​​model and emotion engine, the server issues a real-time warning if the driver is fatigued or if dangerous driving behavior is detected. Specifically, it sends a notification message to the device, such as "Fatigue has been detected. Please take a break," and issues a voice alert. If the driver appears emotional (angry, sad, etc.), it provides appropriate feedback, such as "Please stay calm." Surrounding vehicles are alerted via wireless communication, and a situation report is sent to the base (e.g., logistics center).

[0254] Data storage and generation of optimal driving patterns

[0255] The server stores all collected and analyzed driving data. This stored data is used for later analysis. Specifically, it statistically analyzes past driving data to generate an optimal driving pattern for the driver. This optimal driving pattern is then applied to the vehicle system to reduce driver fatigue and promote safe driving.

[0256] Specific examples

[0257] For example, if a driver has been driving continuously for a long period of time, the device will detect signs of fatigue or anger from their facial expressions and voice. The collected data is sent to a server, where it is analyzed by an artificial intelligence model and emotion engine. If fatigue or anger is confirmed, the server will issue a warning to the driver, saying, "Fatigue has been detected. Please take a break and remain calm." At the same time, a message is sent to surrounding vehicles saying, "There is a possibility of dangerous driving. Please be careful," and a report is sent to the base saying, "Driver ID 123 is in a state of fatigue and agitation. Please take appropriate action." The server will also use the collected driving data to generate optimal driving patterns for the driver in the future and apply them to the vehicle system, further improving driver safety.

[0258] The processing flow will be explained below.

[0259] Step 1:

[0260] The device is installed inside the vehicle and uses a camera and microphone to collect video and audio data. Specifically, the camera captures the driver's face and the microphone records the driver's voice. This data is acquired in real time.

[0261] Step 2:

[0262] The device converts the video data collected by the device into grayscale and converts the audio data into an appropriate format. Specifically, the video data is converted into grayscale for easier processing, and the audio data is converted into a buffer format.

[0263] Step 3:

[0264] The device sends the pre-processed data to the server, where it is encrypted and then uploaded to the server, ensuring secure data transmission.

[0265] Step 4:

[0266] The server inputs the received data into an AI model and emotion engine. The AI ​​model uses deep learning technology to analyze the driver's facial features and voice patterns to detect signs of fatigue or dangerous driving. Meanwhile, the emotion engine recognizes the driver's emotions (joy, anger, sadness, etc.) from video and audio data.

[0267] Step 5:

[0268] The server receives the analysis results of the AI ​​model and emotion engine and determines the driver's state. Specifically, if the fatigue score or emotion score exceeds a certain threshold, it determines that the driver is in a fatigued or emotionally aroused state.

[0269] Step 6:

[0270] The server issues a warning to the driver based on the results of the assessment. Specifically, it sends a notification message to the device saying, "Fatigue has been detected. Please take a break. Please stay calm." It also issues a voice alert. It also sends a warning message to surrounding vehicles via wireless communication.

[0271] Step 7:

[0272] The server reports the driver's condition to the base. Specifically, it sends a message saying, "Driver ID 123 is in a state of fatigue and emotional arousal, so please take appropriate action." This allows the base to understand the driver's condition in real time.

[0273] Step 8:

[0274] The server stores the collected and analyzed driving data. The stored data is used for later analysis and optimization. Specifically, the driving data is stored in a database and later statistically analyzed.

[0275] Step 9:

[0276] The server generates optimal driving patterns based on the stored driving data. Specifically, it analyzes past driving data and suggests efficient and safe driving methods for the driver.

[0277] Step 10:

[0278] The server applies the generated optimal driving pattern to the vehicle system. Specifically, it reflects the optimal driving pattern in the vehicle's various control systems, reducing the burden on the driver and supporting safe driving.

[0279] Example 2

[0280] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0281] In recent years, there has been an increasing demand for systems that support safe driving. However, existing systems face challenges in accurately monitoring the driver's fatigue and emotional state in real time and providing appropriate feedback. Furthermore, existing systems lack the ability to effectively utilize driving data and generate optimal driving patterns. As a result, they are unable to respond quickly to changes in the driver's fatigue and emotions, which can increase the risk of traffic accidents.

[0282] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a preprocessing means for converting video data collected by the terminal into grayscale and audio data into a buffer format, a means for receiving the preprocessed data and using an artificial intelligence model to analyze the driver's condition, a means for analyzing the driver's emotions and fatigue state using the artificial intelligence model and an emotion engine, a means for notifying the driver, surrounding vehicles, and base stations of danger information based on the analysis results, a means for storing the analysis results and driving data, a means for generating an optimal driving pattern based on the stored driving data, and a means for applying the generated driving pattern to the vehicle system. This makes it possible to accurately monitor the driver's condition in real time and provide appropriate feedback. Furthermore, by effectively utilizing the stored driving data and generating and applying an optimal driving pattern for the driver, it is possible to promote safe driving and reduce the risk of traffic accidents.

[0283] A "terminal" is a device that is installed in a vehicle and collects video data and audio data in real time.

[0284] "Video data" refers to visual information that captures the driver's face and the interior of the vehicle, collected by a camera mounted on the device.

[0285] "Voice data" refers to the driver's voice and acoustic information inside the vehicle collected by a microphone installed in the terminal.

[0286] The "pre-processing means" is a processing function that converts collected video data into grayscale and audio data into a buffer format.

[0287] The "server" is a central management device that receives pre-processed data, analyzes it using an artificial intelligence model and emotion engine, and notifies and stores various information based on the analysis results.

[0288] The "artificial intelligence model" is an algorithm that uses deep learning technology to analyze a driver's facial features and voice patterns to detect signs of fatigue or dangerous driving.

[0289] The "emotion engine" is an algorithm that analyzes the driver's emotions from video and audio data and recognizes emotional states such as anger, joy, and sadness.

[0290] "Analysis results" are the assessments and diagnostic results of the driver's condition produced by the artificial intelligence model and emotion engine.

[0291] The "notification means" is a function that sends warnings and report messages to the driver, surrounding vehicles, and base stations based on the analysis results.

[0292] "Storage means" refers to a function for recording and storing analysis results and operating data.

[0293] The "optimal driving pattern" is a guideline for driving behavior that is generated through statistical analysis of stored driving data to improve driver safety and efficiency.

[0294] "Vehicle system" means a set of electronic devices and functions installed within a vehicle for monitoring and controlling driver behavior.

[0295] This invention is a system in which a terminal installed in a vehicle collects video and audio data in real time and preprocesses the data. The preprocessed data is sent to a server, where the driver's state is analyzed using an artificial intelligence model and an emotion engine. Based on the analysis results, danger information is then sent to the driver, surrounding vehicles, and base stations. The server then stores the driving data, generates optimal driving patterns, and applies them to the vehicle system.

[0296] Data collection and preprocessing

[0297] The terminal collects video and audio data in real time using a camera and microphone installed in the vehicle. Specific hardware includes a high-resolution camera and a highly sensitive microphone. The collected data is preprocessed on the terminal. Specifically, the video data is converted to grayscale, and the audio data is converted to a buffer format. This completes the preprocessing required for subsequent analysis.

[0298] Sending data

[0299] The terminal transmits the pre-processed data to the server. Specifically, the data is transmitted using wireless communication technology. Examples of specific technologies used here include LTE and WiFi. The collected data is appropriately compressed and transmitted efficiently.

[0300] Analysis by AI models and sentiment engines

[0301] The server receives the preprocessed data and inputs it into the AI ​​model and emotion engine. The AI ​​model uses deep learning technology to analyze the driver's facial features and voice patterns to detect signs of fatigue or dangerous driving. Specific software used is TensorFlow and PyTorch. Meanwhile, the emotion engine analyzes the driver's emotions from video and audio data. For example, it analyzes the driver's facial expressions and tone of voice to recognize emotions such as anger, joy, and sadness.

[0302] Risk information notification

[0303] Based on the analysis results of the AI ​​model and emotion engine, the server issues a real-time warning if the driver is fatigued or if dangerous driving behavior is detected. Specifically, it sends a notification message to the device, such as "Fatigue has been detected. Please take a break," and issues a voice alert. If the driver appears emotional, it also provides appropriate feedback, such as "Please stay calm." Surrounding vehicles are alerted via wireless communication, and a situation report is sent to the base (e.g., logistics center).

[0304] Data storage and generation of optimal driving patterns

[0305] The server stores all collected and analyzed driving data. This stored data is used for later analysis. Specifically, it statistically analyzes past driving data to generate an optimal driving pattern for the driver. This optimal driving pattern is then applied to the vehicle system to reduce driver fatigue and promote safe driving.

[0306] Specific examples

[0307] For example, if a driver has been driving continuously for a long period of time, the device will detect signs of fatigue or anger from their facial expressions and voice. The collected data is sent to a server, where it is analyzed by an artificial intelligence model and emotion engine. If fatigue or anger is confirmed, the server will issue a warning to the driver, saying, "Fatigue has been detected. Please take a break and remain calm." At the same time, a message is sent to surrounding vehicles saying, "There is a possibility of dangerous driving. Please be careful," and a report is sent to the base saying, "Driver ID 123 is in a state of fatigue and agitation. Please take appropriate action." The server will also use the collected driving data to generate optimal driving patterns for the driver in the future and apply them to the vehicle system, further improving driver safety.

[0308] Prompt Sentence Examples

[0309] "Your role is to understand the operation of the driver support system and analyze the following driving data: Detect emotions and signs of fatigue from the driver's facial expressions and voice data. Based on the results, generate warning messages for the driver and notifications to surrounding vehicles and bases."

[0310] Inquiry Data:

[0311] Video data: Grayscale image of the driver's face

[0312] Audio data: Driver's audio buffer

[0313] The above is a specific embodiment of this system.

[0314] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0315] Step 1: Collect data

[0316] The device uses a camera and microphone installed in the vehicle to collect video and audio data in real time. Specifically, the camera captures the driver's face and obtains high-resolution video data. The microphone captures the driver's voice and any sounds from inside the vehicle and stores them as audio data. This data is input into the device as soon as it is collected.

[0317] input:

[0318] Video data: Driver's face and interior of the vehicle

[0319] Voice data: Driver's voice and acoustic information inside the vehicle

[0320] output:

[0321] Raw video and audio data collected

[0322] Step 2: Preprocessing the data

[0323] The terminal converts the collected video data into grayscale format and audio data into buffer format. Specifically, color information is removed from the video data, and only brightness information is retained. Audio data is divided into segments of a certain length and stored in a buffer. This reduces the amount of data and makes subsequent analysis more efficient.

[0324] input:

[0325] Raw video data

[0326] Raw audio data

[0327] output:

[0328] Video data converted to grayscale

[0329] Audio data converted into buffer format

[0330] Step 3: Sending data

[0331] The device then sends the preprocessed data to the server. Specifically, the data is divided into packets and transmitted to the server using wireless communication technology (e.g., LTE or WiFi). The data is also compressed during transmission, improving communication efficiency.

[0332] input:

[0333] Video data converted to grayscale

[0334] Audio data converted into buffer format

[0335] output:

[0336] Pre-processed data sent to the server

[0337] Step 4: Analysis by AI models and sentiment engines

[0338] The server receives the preprocessed data and inputs it into an artificial intelligence model and emotion engine. Specifically, it uses deep learning algorithms to analyze the driver's facial features and voice patterns to detect signs of driver fatigue or dangerous driving. It then uses the emotion engine to analyze the driver's emotions.

[0339] input:

[0340] Grayscale video data

[0341] Buffered audio data

[0342] output:

[0343] Analysis of driver fatigue and emotional state

[0344] Step 5: Risk notification

[0345] Based on the analysis results, the server notifies the driver, surrounding vehicles, and the base station of any danger information. Specifically, it sends a real-time warning to the driver's device, such as "Fatigue detected. Please take a break." At the same time, it sends a warning message to surrounding vehicles via wireless communication and sends a detailed situation report to the base station.

[0346] input:

[0347] Analysis of driver fatigue and emotional state

[0348] output:

[0349] Driver warning message

[0350] Warning message to surrounding vehicles

[0351] Report the situation to the base

[0352] Step 6: Save your data

[0353] The server stores the analysis results and collected data, specifically by recording this information in a database for long-term data storage and subsequent analysis.

[0354] input:

[0355] Operational data and analysis results

[0356] output:

[0357] Stored database records

[0358] Step 7: Generate optimal driving patterns

[0359] The server statistically analyzes the stored data and generates optimal driving patterns. Specifically, it analyzes past driving data to determine driving behaviors suitable for a specific driver and generates patterns for reducing fatigue and driving safely.

[0360] input:

[0361] Saved driving data

[0362] output:

[0363] Optimal driving pattern

[0364] Step 8: Application to vehicle systems

[0365] The server applies the generated optimal driving pattern to the vehicle system, sending instructions to the vehicle's control system and navigating the vehicle in the optimal driving method for the driver in real time.

[0366] input:

[0367] Optimal driving pattern

[0368] output:

[0369] Driving assistance based on applied driving patterns

[0370] (Application example 2)

[0371] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0372] Many current vehicles lack systems that effectively detect driver fatigue and emotional changes in real time and support safe driving. As a result, there is a high risk of accidents occurring due to long driving times or emotional arousal. Furthermore, conventional systems lack a means to effectively communicate warnings to drivers and do not contribute sufficiently to accident prevention. Furthermore, systems that generate and apply optimal driving patterns for individual drivers are also rare. Effective methods to solve these problems are needed.

[0373] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving preprocessed data and using an artificial intelligence model to analyze the driver's state, means for notifying the driver, surrounding vehicles, and bases of danger information based on the analysis results, and means for using a wearable device to notify the driver of visual and audio warnings in real time. This makes it possible to support the driver in safe driving.

[0374] The "terminal mounted in a vehicle" is a device that is installed inside a vehicle and collects video data and audio data in real time.

[0375] "Video data" refers to visual information obtained from a visual sensor such as a camera.

[0376] "Audio data" refers to acoustic information obtained from an audio sensor such as a microphone.

[0377] "Preprocessing" refers to the process of converting collected data into a form suitable for subsequent analysis.

[0378] An "artificial intelligence model" is a model that includes deep learning algorithms used to analyze the driver's state.

[0379] A "server" is a central processing unit that receives, analyzes, and stores pre-processed data.

[0380] "Danger information" refers to information that warns or alerts drivers about their fatigue level or dangerous driving behavior.

[0381] "Storage" is the act of recording collected and analyzed data in a retrievable form.

[0382] An "optimal driving pattern" is a guideline for driving behavior that is generated to maximize driver safety and efficiency.

[0383] "Applying to vehicle systems" refers to reflecting the generated optimal driving patterns in actual driving operations and control systems.

[0384] A "wearable device" is a device worn by a driver that provides visual and audio notifications.

[0385] This invention provides a system that uses a terminal installed in a vehicle to collect video and audio data in real time, preprocess the data, and transmit it to a server. The server analyzes the driver's condition based on the preprocessed data using an artificial intelligence model and notifies the driver of hazardous situations. The server also stores the driving data, generates optimal driving patterns, and applies them to the vehicle system, supporting safe driving.

[0386] Data collection and preprocessing

[0387] The device uses a camera and microphone to collect in-car video and audio data in real time. The collected video data is converted to grayscale using software libraries such as OpenCV, and the audio data is converted to a buffer format using libraries such as SoundDevice. This preprocesses the collected data into a format optimal for subsequent analysis.

[0388] Analysis by AI models and sentiment engines

[0389] The server receives the preprocessed video and audio data and analyzes the driver's state using an AI model and emotion engine. Specifically, the AI ​​model uses deep learning technology to analyze the driver's facial features and voice patterns to detect signs of driver fatigue or dangerous driving. The emotion engine also analyzes the driver's facial expressions and voice tone to detect changes in emotion (anger, joy, sadness, etc.).

[0390] Risk information notification

[0391] The server notifies the driver of danger information in real time based on the analysis results. Specifically, it provides visual and audio warnings using a wearable device (e.g., smart glasses) worn by the driver. For example, if the driver is fatigued, a message saying "Fatigue detected. Please take a break" is displayed and an audio notification is given. At the same time, a warning message is sent via wireless communication to surrounding vehicles and bases such as logistics centers.

[0392] Data storage and generation of optimal driving patterns

[0393] The server stores all collected and analyzed driving data. This stored data is used for later analysis. Specifically, past driving data is statistically analyzed to generate an optimal driving pattern for each individual driver. This optimal driving pattern is then applied to the vehicle system to reduce driver fatigue and promote safe driving.

[0394] Specific examples

[0395] For example, if a driver has been driving continuously for a long period of time, the device will detect signs of fatigue from their facial expressions and voice. The collected data is sent to a server, where it is analyzed by an artificial intelligence model and emotion engine. If fatigue or agitation is confirmed, the server will issue a warning to the driver, saying, "Fatigue has been detected. Please take a break and stay calm." At the same time, a warning message is sent to surrounding vehicles, and a situation report is sent to the base station. Furthermore, the server can use the collected driving data to generate optimal driving patterns for the driver in the future and apply them to the vehicle system, further improving driver safety.

[0396] Prompt Sentence Examples

[0397] Create an AI model that can detect driver fatigue from video and audio data and provide real-time warnings during long driving sessions. Consider the following steps:

[0398] 1. Convert the video data to grayscale and perform preprocessing.

[0399] 2. Convert the audio data into a buffer format and perform preprocessing.

[0400] 3. The pre-processed data is sent to the server and analyzed by the AI ​​model and emotion engine.

[0401] 4. If fatigue or stress is detected, the driver will be notified with appropriate warning messages via audio and visual means.

[0402] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0403] Step 1:

[0404] The device uses a camera and microphone inside the vehicle to collect video and audio data in real time. Specifically, the camera captures the driver's face and the surrounding environment, while the microphone records the driver's voice and surrounding sounds. The input is video data from the camera and audio data from the microphone, which are then imported into the system in real time.

[0405] Step 2:

[0406] The video data collected by the device is converted to grayscale using OpenCV, and the audio data is converted to buffer format using SoundDevice. Specifically, color data is converted to grayscale, and audio signals are converted to digital data format. The input is color data and raw audio data, and the output is grayscale video data and buffer format audio data.

[0407] Step 3:

[0408] The preprocessed video and audio data are sent to the server. Specifically, the preprocessed data is packaged in JSON format and sent to the server via an HTTP request. The input is grayscale video data and buffered audio data, and the output is the data packet sent to the server.

[0409] Step 4:

[0410] The server inputs the received data into an artificial intelligence model and emotion engine to analyze the driver's condition. Specifically, it uses facial feature recognition algorithms and voice pattern analysis to detect fatigue and changes in emotion. The input is preprocessed video and audio data, and the output is the analysis result regarding the driver's condition.

[0411] Step 5:

[0412] The server generates a message to warn the driver in real time based on the analysis results and sends it to the wearable device. Specifically, if fatigue or dangerous driving is detected, an appropriate warning message is sent visually and audibly. The input is the analysis results, and the output is the warning message.

[0413] Step 6:

[0414] The server saves the analysis results and operational data. Specifically, it stores the results in a database and keeps them as records for later analysis. The input is the analysis results and operational data, and the output is the data saved in the database.

[0415] Step 7:

[0416] The server analyzes the stored driving data and generates optimal driving patterns. Specifically, it uses statistical methods and machine learning algorithms to optimize driving patterns and generate individual patterns for each driver. The input is the stored driving data, and the output is the optimal driving pattern.

[0417] Step 8:

[0418] The generated optimal driving pattern is applied to the vehicle system. Specifically, it is reflected in the vehicle's control system and appropriate feedback is provided to the driver. The input is the optimal driving pattern, and the output is the operation of the driving assistance system according to the applied driving pattern.

[0419] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0420] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0421] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0422] [Second embodiment]

[0423] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0424] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0425] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0426] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0427] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0428] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0429] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0430] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0431] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0432] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0433] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0434] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0435] This invention is a system in which a terminal installed in a vehicle collects video and audio data in real time and preprocesses the data. The preprocessed data is sent to a server, where the driver's condition is analyzed using an artificial intelligence model. Based on the analysis results, danger information is then sent to the driver, surrounding vehicles, and base stations. The server also stores the driving data, generates optimal driving patterns, and reflects them in the vehicle system. This system makes it possible to monitor the driver's safety in real time and support safe driving.

[0436] Data collection and preprocessing

[0437] The terminal collects video and audio data in real time using the camera and microphone installed in the vehicle. The collected data is first pre-processed on the terminal. Specifically, video data is converted to grayscale and audio data is converted to a buffer format. This pre-processing makes the data suitable for analysis by artificial intelligence models.

[0438] Analysis by AI model

[0439] The server receives the preprocessed data and inputs it into an AI model, which uses deep learning technology to analyze the driver's facial features and voice patterns to detect signs of fatigue or dangerous driving, such as the frequency with which the driver opens and closes their eyes, facial expressions, and the tone and rhythm of their voice.

[0440] Risk information notification

[0441] Based on the analysis results of the AI ​​model, the server issues a real-time warning if the driver is fatigued or if dangerous driving behavior is detected. Specifically, a warning message is displayed on the driver's device and an audio alert is given. The server also warns surrounding vehicles via wireless communication and sends a situation report to a base (e.g., a logistics center).

[0442] Data storage and generation of optimal driving patterns

[0443] The server stores all collected and analyzed driving data. This stored data is used for later analysis. Specifically, it statistically analyzes past driving data to generate an optimal driving pattern for the driver. This optimal driving pattern is then applied to the vehicle system to reduce driver fatigue and promote safe driving.

[0444] Specific examples

[0445] For example, if a driver has been driving continuously for a long period of time, the device will detect signs of fatigue from their facial expressions and voice. The collected data is sent to a server, where an artificial intelligence model analyzes it. If fatigue is confirmed, the server will issue a warning to the driver saying, "Fatigue has been detected. Please take a break." At the same time, a message is sent to surrounding vehicles saying, "There is a possibility of dangerous driving. Please be careful," and a report is sent to the base saying, "Driver ID 123 is in a state of fatigue, so please take appropriate action." The server will also use the collected driving data to generate optimal driving patterns for the driver in the future and apply them to the vehicle system, further improving driver safety.

[0446] The processing flow will be explained below.

[0447] Step 1:

[0448] The device collects video and audio data inside the vehicle. Specifically, a camera installed inside the vehicle captures the driver's video and a microphone records the driver's voice. This data is acquired in real time.

[0449] Step 2:

[0450] The terminal preprocesses the collected video and audio data. Specifically, it converts video data to grayscale and audio data to an appropriate format (e.g., integer buffer). This completes the preprocessing required for subsequent analysis.

[0451] Step 3:

[0452] The device sends the pre-processed data to the server, where it is encrypted for secure transmission and uploaded to the server, ready for real-time driver status monitoring.

[0453] Step 4:

[0454] The server inputs the received pre-processed data into an AI model for analysis. The AI ​​model uses deep learning technology to analyze the driver's facial features and voice patterns, thereby detecting signs of driver fatigue and dangerous driving.

[0455] Step 5:

[0456] The server determines risk information based on the analysis results of the AI ​​model. Specifically, if the fatigue score or dangerous driving score indicated by the model exceeds a certain threshold, it determines that the driver is fatigued.

[0457] Step 6:

[0458] The server issues a warning to the driver based on the results of the assessment. Specifically, it sends a notification message to the device saying, "Fatigue has been detected. Please take a break," and issues a voice alert. It also sends wireless communications to nearby vehicles to warn them.

[0459] Step 7:

[0460] The server reports the driver's condition to the base. Specifically, it sends a notification saying, "Driver ID 123 is in a fatigued state, so please take appropriate action." This allows the base to understand the driver's condition in real time.

[0461] Step 8:

[0462] The server stores the collected and analyzed driving data, which is then used for later analysis and optimization. Specifically, the driving data is stored in a database and later statistically analyzed.

[0463] Step 9:

[0464] The server generates optimal driving patterns based on the stored driving data. Specifically, it analyzes past driving data and derives efficient and safe driving methods for the driver.

[0465] Step 10:

[0466] The server applies the generated optimal driving pattern to the vehicle system. Specifically, the pattern is reflected in the vehicle's various control systems, reducing the burden on the driver and supporting safe driving.

[0467] Example 1

[0468] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0469] Conventional vehicle driving monitoring systems have had difficulty accurately grasping the driver's condition in real time and issuing appropriate warnings or notifications. In particular, there is a lack of technology that can quickly detect signs of driver fatigue or dangerous driving and reliably issue warnings, making it difficult to prevent serious traffic accidents.

[0470] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0471] In this invention, the server includes means for receiving the preprocessed data and using a generative AI model to analyze the driver's condition, means for notifying the driver, surrounding vehicles, and bases of danger information based on the analysis results, means for saving the analysis results and driving data, means for analyzing the driver's eye opening and closing frequency and facial expressions from the video data, and means for analyzing the tone and rhythm of the voice from the audio data. This makes it possible to accurately grasp the driver's condition in real time and issue warnings and notifications quickly and appropriately.

[0472] A "terminal" is a device that is installed in a vehicle and collects video data and audio data in real time.

[0473] The "server" is a central computing device that receives the pre-processed data, analyzes the driver's state using a generative AI model, and issues warnings or notifications based on the results.

[0474] A "generative AI model" is a type of artificial intelligence used to analyze video and audio data of a driver and evaluate the driver's condition.

[0475] The "pre-processing means" is a process that converts the collected video data to grayscale and converts the audio data into a suitable buffer format.

[0476] "Means of analysis" refers to the process of inputting the collected pre-processed data into an AI model and evaluating the driver's condition by analyzing the frequency of eye opening and closing, facial expressions, and tone and rhythm of voice.

[0477] "Means for notifying danger information" refers to a means of sending warning messages and alerts to drivers, surrounding vehicles, and bases such as logistics centers based on the analysis results of the AI ​​model.

[0478] "Means for storing data" refers to the process of storing the analyzed driving data in a database or the like and using it for later analysis or for generating optimal driving patterns.

[0479] The "means for generating optimal driving patterns" is a process of statistically analyzing past driving data to derive driving patterns that reduce driver fatigue and promote safe driving.

[0480] The "means for applying the driving pattern" is a process for reflecting the generated optimal driving pattern in the vehicle system.

[0481] "Collecting in real time" refers to obtaining video and audio data instantly without any specific time delay during driving.

[0482] "Video data" refers to visual information that records the inside of a vehicle and the driver's face and expressions.

[0483] "Voice data" refers to auditory information that records the acoustic environment inside the vehicle and the driver's voice.

[0484] "Converting to grayscale" is the process of converting color video data to black and white brightness only.

[0485] "Converting to a buffer format" is the process of temporarily storing audio data and later converting it into a format that is easy to analyze and transmit.

[0486] "Driver status" is an assessment of the driver's current state, based on an analysis of their fatigue, attention, facial expressions, and voice patterns, and is necessary for driving behavior.

[0487] This invention is a system in which a terminal installed in a vehicle collects video and audio data in real time and preprocesses that data. The preprocessed data is sent to a server, where the driver's condition is analyzed using a generative AI model. Based on the analysis results, danger information is then sent to the driver, surrounding vehicles, and base stations. The server also stores the driving data, generates optimal driving patterns, and reflects them in the vehicle system. This system makes it possible to monitor the driver's safety in real time and support safe driving.

[0488] Data collection and preprocessing

[0489] The device uses a camera and microphone installed in the vehicle to collect video and audio data in real time. The video data is captured mainly of the driver's face, while the audio data collects the driver's speech and surrounding sounds. The collected video data is converted to grayscale, and the audio data is converted to a buffer format, making the data suitable for analysis by the generative AI model.

[0490] Examples:

[0491] A camera records the driver's facial expressions in real time, and a microphone picks up the driver's tone of voice.

[0492] The collected color data is quickly converted to grayscale data, and the audio data is stored in a buffer every second.

[0493] Analysis by AI model

[0494] The server receives the preprocessed data and inputs it into a generative AI model. The generative AI model uses deep learning techniques to analyze the driver's facial features and voice patterns to detect signs of driver fatigue or unsafe driving. For example, it analyzes the frequency of the driver's eye opening and closing, facial expressions, and voice tone and rhythm.

[0495] Examples:

[0496] The server analyzes the frequency with which the driver opens and closes their eyelids from the video data to detect signs of fatigue.

[0497] Changes in the tone of the driver's voice are analyzed from audio data to detect signs of lack of attention.

[0498] Example prompts for generative AI models:

[0499] "Analyze pre-processed video and audio data to detect driver fatigue from facial expressions."

[0500] Risk information notification

[0501] Based on the analysis results of the generative AI model, the server issues a real-time warning if the driver is fatigued or if dangerous driving behavior is detected. Specifically, a warning message is displayed on the driver's device and an audio alert is given. The server also warns surrounding vehicles via wireless communication and sends situation reports to bases such as logistics centers.

[0502] Examples:

[0503] The server warns the driver, "Signs of dangerous driving have been detected. Please take a break."

[0504] Sends a message to surrounding vehicles saying, "Possible dangerous driving. Please be careful."

[0505] Example prompts for generative AI models:

[0506] "Generate a warning message to encourage drivers to take a break."

[0507] Data storage and generation of optimal driving patterns

[0508] The server stores all collected and analyzed driving data. This stored data is used for later analysis. Specifically, it statistically analyzes past driving data to generate an optimal driving pattern for the driver. This optimal driving pattern is then applied to the vehicle system to reduce driver fatigue and promote safe driving.

[0509] Examples:

[0510] The server stores each driver's analytical data in a driving database and identifies them with a unique ID.

[0511] Past driving data is analyzed to generate optimal driving patterns, such as "increasing rest time," and these are reflected in the vehicle's navigation system.

[0512] Example prompts for generative AI models:

[0513] "Analyze past driving data and generate optimal driving patterns."

[0514] The above is an embodiment of the present invention. This system can accurately grasp the driver's condition in real time and provide necessary warnings and optimizations to support safe driving.

[0515] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0516] (Program processing flow)

[0517] Step 1:

[0518] The device activates the camera and microphone installed in the vehicle and collects video and audio data in real time.

[0519] Input: In-car video and audio

[0520] Specific behavior:

[0521] The camera captures footage centered on the driver's face.

[0522] A microphone collects audio, including the driver's voice.

[0523] Output: Collected video and audio data

[0524] Step 2:

[0525] The terminal converts the collected video data into grayscale and converts the audio data into a buffer format.

[0526] Input: Collected video and audio data

[0527] Specific behavior:

[0528] Converts video data to grayscale.

[0529] The audio data is divided into fixed time intervals and stored in a buffer.

[0530] Output: Grayscale video data and buffered audio data

[0531] Step 3:

[0532] The terminal transmits the pre-processed data to the server.

[0533] Input: Grayscale video data and buffered audio data

[0534] Specific behavior:

[0535] Upload data to the server using Wi-Fi or mobile data.

[0536] You will receive a confirmation message of the submission.

[0537] Output: Preprocessed data sent to the server

[0538] Step 4:

[0539] The server inputs the received pre-processed data into the generative AI model and analyzes the driver's condition.

[0540] Input: Preprocessed data (grayscaled video data and buffered audio data)

[0541] Specific behavior:

[0542] Send the preprocessed data to the generative AI model.

[0543] The generative AI model analyzes the driver's eye opening and closing frequency, facial expressions, and voice tone and rhythm.

[0544] Output: Driver status analysis results

[0545] Step 5:

[0546] Based on the analysis results of the generative AI model, the server issues a warning in real time if the driver is fatigued or if dangerous driving behavior is detected.

[0547] Input: Driver condition analysis results

[0548] Specific behavior:

[0549] A warning message is sent to the driver's device.

[0550] Issues an audio alert.

[0551] Alerts are sent to surrounding vehicles and bases via radio communication.

[0552] Outputs: Warning messages, audio alerts, and wireless communication notifications

[0553] Step 6:

[0554] The server stores all collected and analyzed driving data.

[0555] Input: Driver status analysis results and other driving data

[0556] Specific behavior:

[0557] Save the analyzed data in a database.

[0558] A unique ID is assigned to the stored data so that it can be searched for later.

[0559] Output: Saved operating data

[0560] Step 7:

[0561] The server statistically analyzes past driving data based on the stored data and generates an optimal driving pattern for the driver.

[0562] Input: Saved historical driving data

[0563] Specific behavior:

[0564] Analyze historical data using statistical analysis algorithms.

[0565] Generate optimal driving patterns and provide feedback to vehicle systems.

[0566] Output: Generated optimal driving pattern

[0567] This allows the system to accurately grasp the driver's condition in real time and provide prompt and appropriate warnings and optimizations.

[0568] (Application example 1)

[0569] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0570] Conventional in-vehicle driver monitoring systems have difficulty accurately assessing the driver's condition in real time and issuing appropriate warnings. Furthermore, the lack of a system that monitors the driver's condition and sends appropriate alerts is a problem, particularly in autonomous vehicles. The present invention aims to solve these problems and provide a system that ensures the safety of the driver and surrounding vehicles.

[0571] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0572] In this invention, the server includes a means for receiving the preprocessed data and using an artificial intelligence model to analyze the driver's condition, a means for notifying the driver, surrounding vehicles, and bases of danger information based on the analysis results, and a means for storing the analysis results and driving data. This allows the server to monitor the driver's condition in real time and issue appropriate warnings to ensure the safety of the driver and those around them.

[0573] A "terminal" is a device that is installed in a vehicle and collects video and audio data in real time and performs preprocessing.

[0574] "Video data" refers to video information captured using a camera showing the conditions inside and outside the vehicle.

[0575] "Audio data" refers to audio information obtained using a microphone of the audio environment inside the vehicle, particularly the driver's voice and surrounding noise.

[0576] "Pre-processing" refers to the process of converting the collected video data to grayscale and converting the audio data into a suitable buffer format.

[0577] The "server" is a central control computer that receives the pre-processed data and analyzes the driver's condition using artificial intelligence models.

[0578] An "artificial intelligence model" is a software algorithm that uses deep learning techniques to analyze data and analyse the driver's facial features and voice patterns.

[0579] "Analysis results" are information derived from artificial intelligence models about driver fatigue and signs of dangerous driving.

[0580] "Notification" refers to the act of the server transmitting danger information to the driver, surrounding vehicles, and base stations based on the analysis results.

[0581] "Storage" refers to the act of recording the analysis results and operational data in a database or the like for use in later analysis.

[0582] A "driving pattern" is a guideline that indicates the optimal driving method for a driver, generated based on past driving data.

[0583] "Smart glasses" are wearable devices that are installed in the driver's seat and collect video and audio data.

[0584] A "deep learning model" is an algorithm based on a neural network with multiple layers that runs on a server.

[0585] An "alert" is a warning or caution message to the driver that is conveyed through audio or visual means.

[0586] The embodiments of the present invention will be described in detail below.

[0587] System configuration

[0588] The system of the present invention mainly comprises the following elements:

[0589] Terminals installed in vehicles

[0590] server

[0591] Smart Glasses

[0592] Deep Learning Model

[0593] Data collection and preprocessing

[0594] The terminal collects video and audio data in real time using the vehicle's onboard camera and microphone. The smart glasses acquire and preprocess the data, converting video data to grayscale and audio data to a suitable buffer format.

[0595] Data transmission and analysis

[0596] The pre-processed data is sent to a server via a computer installed inside the vehicle. The server then analyzes the received data using a deep learning model. This model analyzes the driver's facial features and voice patterns to detect signs of fatigue or dangerous driving. The artificial intelligence model uses frameworks such as TensorFlow and PyTorch.

[0597] Risk information notification

[0598] Based on the analysis results of the AI ​​model, the server issues a real-time warning if the driver is fatigued or if dangerous driving behavior is detected. Specifically, a warning message is displayed on the driver's device and an audio alert is given. The server also warns surrounding vehicles via wireless communication and sends a situation report to the base station.

[0599] Data storage and driving pattern generation

[0600] The server stores all collected and analyzed driving data, using cloud storage and databases. This stored data is then used for further analysis. Specifically, it statistically analyzes past driving data and generates an optimal driving pattern for the driver. This driving pattern is then applied to the vehicle systems to reduce driver fatigue and promote safe driving.

[0601] Specific examples

[0602] For example, if a driver has been driving continuously for a long period of time, the smart glasses will detect changes in the driver's facial expression and voice and send the data to a server. The server will then use an artificial intelligence model to analyze the data and, if it determines that the driver is fatigued, will issue a warning to the driver saying, "Fatigue has been detected. Please take a break." At the same time, a message will be sent to surrounding vehicles saying, "Possible dangerous driving. Please be careful," and a report will be sent to the base saying, "Driver ID 123 is fatigued, so please take appropriate action." The server will also use the collected driving data to generate optimal driving patterns for the driver in the future and apply them to the vehicle system, further improving safety.

[0603] Prompt Sentence Examples

[0604] An example prompt is:

[0605] "Please build an AI model system that uses the driver's facial features and voice patterns to analyze their fatigue level and risky driving behavior in real time and issue a warning to the vehicle system. Based on the invention, data will be collected from the camera and microphone and preprocessed into grayscale images and buffered audio data. Then, the data will be sent to the server and analyzed by deep learning. Please also provide specific instructions and examples of warning messages."

[0606] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0607] Step 1:

[0608] The device uses a camera and microphone to collect video and audio data from inside the vehicle in real time.

[0609] Input: In-vehicle video and audio

[0610] Output: Raw collected data (video and audio)

[0611] Specifically, the device activates the camera and microphone to continuously capture the driver's facial expressions and voice.

[0612] Step 2:

[0613] The terminal converts the collected video data to grayscale and converts the audio data to an appropriate buffer format.

[0614] Input: Raw collected data (video and audio)

[0615] Output: Preprocessed data (grayscale image and audio in buffer format)

[0616] Specifically, the video data is converted to grayscale using the OpenCV library, and the audio data is converted using Python's buffer function.

[0617] Step 3:

[0618] The pre-processed data is sent to a server via a computer inside the vehicle.

[0619] Input: Preprocessed data (grayscale images and audio in buffer format)

[0620] Output: Data sent to the server

[0621] Specifically, the preprocessed data is uploaded to the server using an HTTP POST request.

[0622] Step 4:

[0623] The data received by the server is input into an artificial intelligence model (deep learning model) to analyze the driver's condition.

[0624] Input: Received data (grayscale image and buffered audio)

[0625] Output: Analysis results (status information based on the driver's facial features and voice patterns)

[0626] Specifically, it analyzes data using deep learning frameworks such as TensorFlow and PyTorch to detect signs of fatigue and dangerous driving.

[0627] Step 5:

[0628] Based on the analysis results, the server displays a warning message on the driver's device and issues an audio alert, while also wirelessly alerting surrounding vehicles and sending a situation report to the base.

[0629] Input: Analysis results

[0630] Outputs: Warning messages, audio alerts, radio communication messages, situation reports

[0631] Specifically, it alerts the driver using SMS or application notifications, alerts surrounding vehicles using wireless communication protocols, and sends reports to a base station via email or dashboard.

[0632] Step 6:

[0633] The server stores the analysis results and operational data in a database.

[0634] Input: Analysis results and operating data

[0635] Output: Saved data

[0636] Specifically, the data is recorded in cloud storage or a relational database.

[0637] Step 7:

[0638] The server statistically analyzes the stored driving data, generates an optimal driving pattern for the driver, and applies it to the vehicle system.

[0639] Input: Saved data

[0640] Output: Optimal driving pattern, applied driving pattern

[0641] Specifically, it uses machine learning algorithms to analyze data, form driving patterns, and push them to the vehicle's systems for application.

[0642] The above are the specific processing steps of the system that realizes the application example.

[0643] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0644] This invention is a system in which a terminal installed in a vehicle collects video and audio data in real time and preprocesses that data. The preprocessed data is sent to a server, where the driver's condition is analyzed using an artificial intelligence model and an emotion engine. Based on the analysis results, danger information is then sent to the driver, surrounding vehicles, and base stations. The server also stores the driving data, generates optimal driving patterns, and reflects them in the vehicle system. This system makes it possible to monitor the driver's condition in real time and support safe driving.

[0645] Data collection and preprocessing

[0646] The terminal collects video and audio data in real time using the camera and microphone installed in the vehicle. The collected data is pre-processed on the terminal. Specifically, the video data is converted to grayscale and the audio data is converted to a buffer format. This completes the pre-processing required for subsequent analysis.

[0647] Analysis by AI models and sentiment engines

[0648] The server receives the preprocessed data and inputs it into the AI ​​model and emotion engine. The AI ​​model uses deep learning technology to analyze the driver's facial features and voice patterns to detect signs of fatigue or dangerous driving. Meanwhile, the emotion engine analyzes the driver's emotions from video and audio data. For example, it analyzes the driver's facial expressions and tone of voice to recognize emotions such as anger, joy, and sadness.

[0649] Risk information notification

[0650] Based on the analysis results of the AI ​​model and emotion engine, the server issues a real-time warning if the driver is fatigued or if dangerous driving behavior is detected. Specifically, it sends a notification message to the device, such as "Fatigue has been detected. Please take a break," and issues a voice alert. If the driver appears emotional (angry, sad, etc.), it provides appropriate feedback, such as "Please stay calm." Surrounding vehicles are alerted via wireless communication, and a situation report is sent to the base (e.g., logistics center).

[0651] Data storage and generation of optimal driving patterns

[0652] The server stores all collected and analyzed driving data. This stored data is used for later analysis. Specifically, it statistically analyzes past driving data to generate an optimal driving pattern for the driver. This optimal driving pattern is then applied to the vehicle system to reduce driver fatigue and promote safe driving.

[0653] Specific examples

[0654] For example, if a driver has been driving continuously for a long period of time, the device will detect signs of fatigue or anger from their facial expressions and voice. The collected data is sent to a server, where it is analyzed by an artificial intelligence model and emotion engine. If fatigue or anger is confirmed, the server will issue a warning to the driver, saying, "Fatigue has been detected. Please take a break and remain calm." At the same time, a message is sent to surrounding vehicles saying, "There is a possibility of dangerous driving. Please be careful," and a report is sent to the base saying, "Driver ID 123 is in a state of fatigue and agitation. Please take appropriate action." The server will also use the collected driving data to generate optimal driving patterns for the driver in the future and apply them to the vehicle system, further improving driver safety.

[0655] The processing flow will be explained below.

[0656] Step 1:

[0657] The device is installed inside the vehicle and uses a camera and microphone to collect video and audio data. Specifically, the camera captures the driver's face and the microphone records the driver's voice. This data is acquired in real time.

[0658] Step 2:

[0659] The device converts the video data collected by the device into grayscale and converts the audio data into an appropriate format. Specifically, the video data is converted into grayscale for easier processing, and the audio data is converted into a buffer format.

[0660] Step 3:

[0661] The device sends the pre-processed data to the server, where it is encrypted and then uploaded to the server, ensuring secure data transmission.

[0662] Step 4:

[0663] The server inputs the received data into an AI model and emotion engine. The AI ​​model uses deep learning technology to analyze the driver's facial features and voice patterns to detect signs of fatigue or dangerous driving. Meanwhile, the emotion engine recognizes the driver's emotions (joy, anger, sadness, etc.) from video and audio data.

[0664] Step 5:

[0665] The server receives the analysis results of the AI ​​model and emotion engine and determines the driver's state. Specifically, if the fatigue score or emotion score exceeds a certain threshold, it determines that the driver is in a fatigued or emotionally aroused state.

[0666] Step 6:

[0667] The server issues a warning to the driver based on the results of the assessment. Specifically, it sends a notification message to the device saying, "Fatigue has been detected. Please take a break. Please stay calm." It also issues a voice alert. It also sends a warning message to surrounding vehicles via wireless communication.

[0668] Step 7:

[0669] The server reports the driver's condition to the base. Specifically, it sends a message saying, "Driver ID 123 is in a state of fatigue and emotional arousal, so please take appropriate action." This allows the base to understand the driver's condition in real time.

[0670] Step 8:

[0671] The server stores the collected and analyzed driving data. The stored data is used for later analysis and optimization. Specifically, the driving data is stored in a database and later statistically analyzed.

[0672] Step 9:

[0673] The server generates optimal driving patterns based on the stored driving data. Specifically, it analyzes past driving data and suggests efficient and safe driving methods for the driver.

[0674] Step 10:

[0675] The server applies the generated optimal driving pattern to the vehicle system. Specifically, it reflects the optimal driving pattern in the vehicle's various control systems, reducing the burden on the driver and supporting safe driving.

[0676] Example 2

[0677] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0678] In recent years, there has been an increasing demand for systems that support safe driving. However, existing systems face challenges in accurately monitoring the driver's fatigue and emotional state in real time and providing appropriate feedback. Furthermore, existing systems lack the ability to effectively utilize driving data and generate optimal driving patterns. As a result, they are unable to respond quickly to changes in the driver's fatigue and emotions, which can increase the risk of traffic accidents.

[0679] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a preprocessing means for converting video data collected by the terminal into grayscale and audio data into a buffer format, a means for receiving the preprocessed data and using an artificial intelligence model to analyze the driver's condition, a means for analyzing the driver's emotions and fatigue state using the artificial intelligence model and an emotion engine, a means for notifying the driver, surrounding vehicles, and base stations of danger information based on the analysis results, a means for storing the analysis results and driving data, a means for generating an optimal driving pattern based on the stored driving data, and a means for applying the generated driving pattern to the vehicle system. This makes it possible to accurately monitor the driver's condition in real time and provide appropriate feedback. Furthermore, by effectively utilizing the stored driving data and generating and applying an optimal driving pattern for the driver, it is possible to promote safe driving and reduce the risk of traffic accidents.

[0680] A "terminal" is a device that is installed in a vehicle and collects video data and audio data in real time.

[0681] "Video data" refers to visual information that captures the driver's face and the interior of the vehicle, collected by a camera mounted on the device.

[0682] "Voice data" refers to the driver's voice and acoustic information inside the vehicle collected by a microphone installed in the terminal.

[0683] The "pre-processing means" is a processing function that converts collected video data into grayscale and audio data into a buffer format.

[0684] The "server" is a central management device that receives pre-processed data, analyzes it using an artificial intelligence model and emotion engine, and notifies and stores various information based on the analysis results.

[0685] The "artificial intelligence model" is an algorithm that uses deep learning technology to analyze a driver's facial features and voice patterns to detect signs of fatigue or dangerous driving.

[0686] The "emotion engine" is an algorithm that analyzes the driver's emotions from video and audio data and recognizes emotional states such as anger, joy, and sadness.

[0687] "Analysis results" are the assessments and diagnostic results of the driver's condition produced by the artificial intelligence model and emotion engine.

[0688] The "notification means" is a function that sends warnings and report messages to the driver, surrounding vehicles, and base stations based on the analysis results.

[0689] "Storage means" refers to a function for recording and storing analysis results and operating data.

[0690] The "optimal driving pattern" is a guideline for driving behavior that is generated through statistical analysis of stored driving data to improve driver safety and efficiency.

[0691] "Vehicle system" means a set of electronic devices and functions installed within a vehicle for monitoring and controlling driver behavior.

[0692] This invention is a system in which a terminal installed in a vehicle collects video and audio data in real time and preprocesses the data. The preprocessed data is sent to a server, where the driver's state is analyzed using an artificial intelligence model and an emotion engine. Based on the analysis results, danger information is then sent to the driver, surrounding vehicles, and base stations. The server then stores the driving data, generates optimal driving patterns, and applies them to the vehicle system.

[0693] Data collection and preprocessing

[0694] The terminal collects video and audio data in real time using a camera and microphone installed in the vehicle. Specific hardware includes a high-resolution camera and a highly sensitive microphone. The collected data is preprocessed on the terminal. Specifically, the video data is converted to grayscale, and the audio data is converted to a buffer format. This completes the preprocessing required for subsequent analysis.

[0695] Sending data

[0696] The terminal transmits the pre-processed data to the server. Specifically, the data is transmitted using wireless communication technology. Examples of specific technologies used here include LTE and WiFi. The collected data is appropriately compressed and transmitted efficiently.

[0697] Analysis by AI models and sentiment engines

[0698] The server receives the preprocessed data and inputs it into the AI ​​model and emotion engine. The AI ​​model uses deep learning technology to analyze the driver's facial features and voice patterns to detect signs of fatigue or dangerous driving. Specific software used is TensorFlow and PyTorch. Meanwhile, the emotion engine analyzes the driver's emotions from video and audio data. For example, it analyzes the driver's facial expressions and tone of voice to recognize emotions such as anger, joy, and sadness.

[0699] Risk information notification

[0700] Based on the analysis results of the AI ​​model and emotion engine, the server issues a real-time warning if the driver is fatigued or if dangerous driving behavior is detected. Specifically, it sends a notification message to the device, such as "Fatigue has been detected. Please take a break," and issues a voice alert. If the driver appears emotional, it also provides appropriate feedback, such as "Please stay calm." Surrounding vehicles are alerted via wireless communication, and a situation report is sent to the base (e.g., logistics center).

[0701] Data storage and generation of optimal driving patterns

[0702] The server stores all collected and analyzed driving data. This stored data is used for later analysis. Specifically, it statistically analyzes past driving data to generate an optimal driving pattern for the driver. This optimal driving pattern is then applied to the vehicle system to reduce driver fatigue and promote safe driving.

[0703] Specific examples

[0704] For example, if a driver has been driving continuously for a long period of time, the device will detect signs of fatigue or anger from their facial expressions and voice. The collected data is sent to a server, where it is analyzed by an artificial intelligence model and emotion engine. If fatigue or anger is confirmed, the server will issue a warning to the driver, saying, "Fatigue has been detected. Please take a break and remain calm." At the same time, a message is sent to surrounding vehicles saying, "There is a possibility of dangerous driving. Please be careful," and a report is sent to the base saying, "Driver ID 123 is in a state of fatigue and agitation. Please take appropriate action." The server will also use the collected driving data to generate optimal driving patterns for the driver in the future and apply them to the vehicle system, further improving driver safety.

[0705] Prompt Sentence Examples

[0706] "Your role is to understand the operation of the driver support system and analyze the following driving data: Detect emotions and signs of fatigue from the driver's facial expressions and voice data. Based on the results, generate warning messages for the driver and notifications to surrounding vehicles and bases."

[0707] Inquiry Data:

[0708] Video data: Grayscale image of the driver's face

[0709] Audio data: Driver's audio buffer

[0710] The above is a specific embodiment of this system.

[0711] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0712] Step 1: Collect data

[0713] The device uses a camera and microphone installed in the vehicle to collect video and audio data in real time. Specifically, the camera captures the driver's face and obtains high-resolution video data. The microphone captures the driver's voice and any sounds from inside the vehicle and stores them as audio data. This data is input into the device as soon as it is collected.

[0714] input:

[0715] Video data: Driver's face and interior of the vehicle

[0716] Voice data: Driver's voice and acoustic information inside the vehicle

[0717] output:

[0718] Raw video and audio data collected

[0719] Step 2: Preprocessing the data

[0720] The terminal converts the collected video data into grayscale format and audio data into buffer format. Specifically, color information is removed from the video data, and only brightness information is retained. Audio data is divided into segments of a certain length and stored in a buffer. This reduces the amount of data and makes subsequent analysis more efficient.

[0721] input:

[0722] Raw video data

[0723] Raw audio data

[0724] output:

[0725] Video data converted to grayscale

[0726] Audio data converted into buffer format

[0727] Step 3: Sending data

[0728] The device then sends the preprocessed data to the server. Specifically, the data is divided into packets and transmitted to the server using wireless communication technology (e.g., LTE or WiFi). The data is also compressed during transmission, improving communication efficiency.

[0729] input:

[0730] Video data converted to grayscale

[0731] Audio data converted into buffer format

[0732] output:

[0733] Pre-processed data sent to the server

[0734] Step 4: Analysis by AI models and sentiment engines

[0735] The server receives the preprocessed data and inputs it into an artificial intelligence model and emotion engine. Specifically, it uses deep learning algorithms to analyze the driver's facial features and voice patterns to detect signs of driver fatigue or dangerous driving. It then uses the emotion engine to analyze the driver's emotions.

[0736] input:

[0737] Grayscale video data

[0738] Buffered audio data

[0739] output:

[0740] Analysis of driver fatigue and emotional state

[0741] Step 5: Risk notification

[0742] Based on the analysis results, the server notifies the driver, surrounding vehicles, and the base station of any danger information. Specifically, it sends a real-time warning to the driver's device, such as "Fatigue detected. Please take a break." At the same time, it sends a warning message to surrounding vehicles via wireless communication and sends a detailed situation report to the base station.

[0743] input:

[0744] Analysis of driver fatigue and emotional state

[0745] output:

[0746] Driver warning message

[0747] Warning message to surrounding vehicles

[0748] Report the situation to the base

[0749] Step 6: Save your data

[0750] The server stores the analysis results and collected data, specifically by recording this information in a database for long-term data storage and subsequent analysis.

[0751] input:

[0752] Operational data and analysis results

[0753] output:

[0754] Stored database records

[0755] Step 7: Generate optimal driving patterns

[0756] The server statistically analyzes the stored data and generates optimal driving patterns. Specifically, it analyzes past driving data to determine driving behaviors suitable for a specific driver and generates patterns for reducing fatigue and driving safely.

[0757] input:

[0758] Saved driving data

[0759] output:

[0760] Optimal driving pattern

[0761] Step 8: Application to vehicle systems

[0762] The server applies the generated optimal driving pattern to the vehicle system, sending instructions to the vehicle's control system and navigating the vehicle in the optimal driving method for the driver in real time.

[0763] input:

[0764] Optimal driving pattern

[0765] output:

[0766] Driving assistance based on applied driving patterns

[0767] (Application example 2)

[0768] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0769] Many current vehicles lack systems that effectively detect driver fatigue and emotional changes in real time and support safe driving. As a result, there is a high risk of accidents occurring due to long driving times or emotional arousal. Furthermore, conventional systems lack a means to effectively communicate warnings to drivers and do not contribute sufficiently to accident prevention. Furthermore, systems that generate and apply optimal driving patterns for individual drivers are also rare. Effective methods to solve these problems are needed.

[0770] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving preprocessed data and using an artificial intelligence model to analyze the driver's state, means for notifying the driver, surrounding vehicles, and bases of danger information based on the analysis results, and means for using a wearable device to notify the driver of visual and audio warnings in real time. This makes it possible to support the driver in safe driving.

[0771] The "terminal mounted in a vehicle" is a device that is installed inside a vehicle and collects video data and audio data in real time.

[0772] "Video data" refers to visual information obtained from a visual sensor such as a camera.

[0773] "Audio data" refers to acoustic information obtained from an audio sensor such as a microphone.

[0774] "Preprocessing" refers to the process of converting collected data into a form suitable for subsequent analysis.

[0775] An "artificial intelligence model" is a model that includes deep learning algorithms used to analyze the driver's state.

[0776] A "server" is a central processing unit that receives, analyzes, and stores pre-processed data.

[0777] "Danger information" refers to information that warns or alerts drivers about their fatigue level or dangerous driving behavior.

[0778] "Storage" is the act of recording collected and analyzed data in a retrievable form.

[0779] An "optimal driving pattern" is a guideline for driving behavior that is generated to maximize driver safety and efficiency.

[0780] "Applying to vehicle systems" refers to reflecting the generated optimal driving patterns in actual driving operations and control systems.

[0781] A "wearable device" is a device worn by a driver that provides visual and audio notifications.

[0782] This invention provides a system that uses a terminal installed in a vehicle to collect video and audio data in real time, preprocess the data, and transmit it to a server. The server analyzes the driver's condition based on the preprocessed data using an artificial intelligence model and notifies the driver of hazardous situations. The server also stores the driving data, generates optimal driving patterns, and applies them to the vehicle system, supporting safe driving.

[0783] Data collection and preprocessing

[0784] The device uses a camera and microphone to collect in-car video and audio data in real time. The collected video data is converted to grayscale using software libraries such as OpenCV, and the audio data is converted to a buffer format using libraries such as SoundDevice. This preprocesses the collected data into a format optimal for subsequent analysis.

[0785] Analysis by AI models and sentiment engines

[0786] The server receives the preprocessed video and audio data and analyzes the driver's state using an AI model and emotion engine. Specifically, the AI ​​model uses deep learning technology to analyze the driver's facial features and voice patterns to detect signs of driver fatigue or dangerous driving. The emotion engine also analyzes the driver's facial expressions and voice tone to detect changes in emotion (anger, joy, sadness, etc.).

[0787] Risk information notification

[0788] The server notifies the driver of danger information in real time based on the analysis results. Specifically, it provides visual and audio warnings using a wearable device (e.g., smart glasses) worn by the driver. For example, if the driver is fatigued, a message saying "Fatigue detected. Please take a break" is displayed and an audio notification is given. At the same time, a warning message is sent via wireless communication to surrounding vehicles and bases such as logistics centers.

[0789] Data storage and generation of optimal driving patterns

[0790] The server stores all collected and analyzed driving data. This stored data is used for later analysis. Specifically, past driving data is statistically analyzed to generate an optimal driving pattern for each individual driver. This optimal driving pattern is then applied to the vehicle system to reduce driver fatigue and promote safe driving.

[0791] Specific examples

[0792] For example, if a driver has been driving continuously for a long period of time, the device will detect signs of fatigue from their facial expressions and voice. The collected data is sent to a server, where it is analyzed by an artificial intelligence model and emotion engine. If fatigue or agitation is confirmed, the server will issue a warning to the driver, saying, "Fatigue has been detected. Please take a break and stay calm." At the same time, a warning message is sent to surrounding vehicles, and a situation report is sent to the base station. Furthermore, the server can use the collected driving data to generate optimal driving patterns for the driver in the future and apply them to the vehicle system, further improving driver safety.

[0793] Prompt Sentence Examples

[0794] Create an AI model that can detect driver fatigue from video and audio data and provide real-time warnings during long driving sessions. Consider the following steps:

[0795] 1. Convert the video data to grayscale and perform preprocessing.

[0796] 2. Convert the audio data into a buffer format and perform preprocessing.

[0797] 3. The pre-processed data is sent to the server and analyzed by the AI ​​model and emotion engine.

[0798] 4. If fatigue or stress is detected, the driver will be notified with appropriate warning messages via audio and visual means.

[0799] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0800] Step 1:

[0801] The device uses a camera and microphone inside the vehicle to collect video and audio data in real time. Specifically, the camera captures the driver's face and the surrounding environment, while the microphone records the driver's voice and surrounding sounds. The input is video data from the camera and audio data from the microphone, which are then imported into the system in real time.

[0802] Step 2:

[0803] The video data collected by the device is converted to grayscale using OpenCV, and the audio data is converted to buffer format using SoundDevice. Specifically, color data is converted to grayscale, and audio signals are converted to digital data format. The input is color data and raw audio data, and the output is grayscale video data and buffer format audio data.

[0804] Step 3:

[0805] The preprocessed video and audio data are sent to the server. Specifically, the preprocessed data is packaged in JSON format and sent to the server via an HTTP request. The input is grayscale video data and buffered audio data, and the output is the data packet sent to the server.

[0806] Step 4:

[0807] The server inputs the received data into an artificial intelligence model and emotion engine to analyze the driver's condition. Specifically, it uses facial feature recognition algorithms and voice pattern analysis to detect fatigue and changes in emotion. The input is preprocessed video and audio data, and the output is the analysis result regarding the driver's condition.

[0808] Step 5:

[0809] The server generates a message to warn the driver in real time based on the analysis results and sends it to the wearable device. Specifically, if fatigue or dangerous driving is detected, an appropriate warning message is sent visually and audibly. The input is the analysis results, and the output is the warning message.

[0810] Step 6:

[0811] The server saves the analysis results and operational data. Specifically, it stores the results in a database and keeps them as records for later analysis. The input is the analysis results and operational data, and the output is the data saved in the database.

[0812] Step 7:

[0813] The server analyzes the stored driving data and generates optimal driving patterns. Specifically, it uses statistical methods and machine learning algorithms to optimize driving patterns and generate individual patterns for each driver. The input is the stored driving data, and the output is the optimal driving pattern.

[0814] Step 8:

[0815] The generated optimal driving pattern is applied to the vehicle system. Specifically, it is reflected in the vehicle's control system and appropriate feedback is provided to the driver. The input is the optimal driving pattern, and the output is the operation of the driving assistance system according to the applied driving pattern.

[0816] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0817] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0818] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0819] [Third embodiment]

[0820] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0821] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0822] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0823] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0824] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0825] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0826] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0827] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0828] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0829] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0830] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0831] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0832] This invention is a system in which a terminal installed in a vehicle collects video and audio data in real time and preprocesses the data. The preprocessed data is sent to a server, where the driver's condition is analyzed using an artificial intelligence model. Based on the analysis results, danger information is then sent to the driver, surrounding vehicles, and base stations. The server also stores the driving data, generates optimal driving patterns, and reflects them in the vehicle system. This system makes it possible to monitor the driver's safety in real time and support safe driving.

[0833] Data collection and preprocessing

[0834] The terminal collects video and audio data in real time using the camera and microphone installed in the vehicle. The collected data is first pre-processed on the terminal. Specifically, video data is converted to grayscale and audio data is converted to a buffer format. This pre-processing makes the data suitable for analysis by artificial intelligence models.

[0835] Analysis by AI model

[0836] The server receives the preprocessed data and inputs it into an AI model, which uses deep learning technology to analyze the driver's facial features and voice patterns to detect signs of fatigue or dangerous driving, such as the frequency with which the driver opens and closes their eyes, facial expressions, and the tone and rhythm of their voice.

[0837] Risk information notification

[0838] Based on the analysis results of the AI ​​model, the server issues a real-time warning if the driver is fatigued or if dangerous driving behavior is detected. Specifically, a warning message is displayed on the driver's device and an audio alert is given. The server also warns surrounding vehicles via wireless communication and sends a situation report to a base (e.g., a logistics center).

[0839] Data storage and generation of optimal driving patterns

[0840] The server stores all collected and analyzed driving data. This stored data is used for later analysis. Specifically, it statistically analyzes past driving data to generate an optimal driving pattern for the driver. This optimal driving pattern is then applied to the vehicle system to reduce driver fatigue and promote safe driving.

[0841] Specific examples

[0842] For example, if a driver has been driving continuously for a long period of time, the device will detect signs of fatigue from their facial expressions and voice. The collected data is sent to a server, where an artificial intelligence model analyzes it. If fatigue is confirmed, the server will issue a warning to the driver saying, "Fatigue has been detected. Please take a break." At the same time, a message is sent to surrounding vehicles saying, "There is a possibility of dangerous driving. Please be careful," and a report is sent to the base saying, "Driver ID 123 is in a state of fatigue, so please take appropriate action." The server will also use the collected driving data to generate optimal driving patterns for the driver in the future and apply them to the vehicle system, further improving driver safety.

[0843] The processing flow will be explained below.

[0844] Step 1:

[0845] The device collects video and audio data inside the vehicle. Specifically, a camera installed inside the vehicle captures the driver's video and a microphone records the driver's voice. This data is acquired in real time.

[0846] Step 2:

[0847] The terminal preprocesses the collected video and audio data. Specifically, it converts video data to grayscale and audio data to an appropriate format (e.g., integer buffer). This completes the preprocessing required for subsequent analysis.

[0848] Step 3:

[0849] The device sends the pre-processed data to the server, where it is encrypted for secure transmission and uploaded to the server, ready for real-time driver status monitoring.

[0850] Step 4:

[0851] The server inputs the received pre-processed data into an AI model for analysis. The AI ​​model uses deep learning technology to analyze the driver's facial features and voice patterns, thereby detecting signs of driver fatigue and dangerous driving.

[0852] Step 5:

[0853] The server determines risk information based on the analysis results of the AI ​​model. Specifically, if the fatigue score or dangerous driving score indicated by the model exceeds a certain threshold, it determines that the driver is fatigued.

[0854] Step 6:

[0855] The server issues a warning to the driver based on the results of the assessment. Specifically, it sends a notification message to the device saying, "Fatigue has been detected. Please take a break," and issues a voice alert. It also sends wireless communications to nearby vehicles to warn them.

[0856] Step 7:

[0857] The server reports the driver's condition to the base. Specifically, it sends a notification saying, "Driver ID 123 is in a fatigued state, so please take appropriate action." This allows the base to understand the driver's condition in real time.

[0858] Step 8:

[0859] The server stores the collected and analyzed driving data, which is then used for later analysis and optimization. Specifically, the driving data is stored in a database and later statistically analyzed.

[0860] Step 9:

[0861] The server generates optimal driving patterns based on the stored driving data. Specifically, it analyzes past driving data and derives efficient and safe driving methods for the driver.

[0862] Step 10:

[0863] The server applies the generated optimal driving pattern to the vehicle system. Specifically, the pattern is reflected in the vehicle's various control systems, reducing the burden on the driver and supporting safe driving.

[0864] Example 1

[0865] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0866] Conventional vehicle driving monitoring systems have had difficulty accurately grasping the driver's condition in real time and issuing appropriate warnings or notifications. In particular, there is a lack of technology that can quickly detect signs of driver fatigue or dangerous driving and reliably issue warnings, making it difficult to prevent serious traffic accidents.

[0867] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0868] In this invention, the server includes means for receiving the preprocessed data and using a generative AI model to analyze the driver's condition, means for notifying the driver, surrounding vehicles, and bases of danger information based on the analysis results, means for saving the analysis results and driving data, means for analyzing the driver's eye opening and closing frequency and facial expressions from the video data, and means for analyzing the tone and rhythm of the voice from the audio data. This makes it possible to accurately grasp the driver's condition in real time and issue warnings and notifications quickly and appropriately.

[0869] A "terminal" is a device that is installed in a vehicle and collects video data and audio data in real time.

[0870] The "server" is a central computing device that receives the pre-processed data, analyzes the driver's state using a generative AI model, and issues warnings or notifications based on the results.

[0871] A "generative AI model" is a type of artificial intelligence used to analyze video and audio data of a driver and evaluate the driver's condition.

[0872] The "pre-processing means" is a process that converts the collected video data to grayscale and converts the audio data into a suitable buffer format.

[0873] "Means of analysis" refers to the process of inputting the collected pre-processed data into an AI model and evaluating the driver's condition by analyzing the frequency of eye opening and closing, facial expressions, and tone and rhythm of voice.

[0874] "Means for notifying danger information" refers to a means of sending warning messages and alerts to drivers, surrounding vehicles, and bases such as logistics centers based on the analysis results of the AI ​​model.

[0875] "Means for storing data" refers to the process of storing the analyzed driving data in a database or the like and using it for later analysis or for generating optimal driving patterns.

[0876] The "means for generating optimal driving patterns" is a process of statistically analyzing past driving data to derive driving patterns that reduce driver fatigue and promote safe driving.

[0877] The "means for applying the driving pattern" is a process for reflecting the generated optimal driving pattern in the vehicle system.

[0878] "Collecting in real time" refers to obtaining video and audio data instantly without any specific time delay during driving.

[0879] "Video data" refers to visual information that records the inside of a vehicle and the driver's face and expressions.

[0880] "Voice data" refers to auditory information that records the acoustic environment inside the vehicle and the driver's voice.

[0881] "Converting to grayscale" is the process of converting color video data to black and white brightness only.

[0882] "Converting to a buffer format" is the process of temporarily storing audio data and later converting it into a format that is easy to analyze and transmit.

[0883] "Driver status" is an assessment of the driver's current state, based on an analysis of their fatigue, attention, facial expressions, and voice patterns, and is necessary for driving behavior.

[0884] This invention is a system in which a terminal installed in a vehicle collects video and audio data in real time and preprocesses that data. The preprocessed data is sent to a server, where the driver's condition is analyzed using a generative AI model. Based on the analysis results, danger information is then sent to the driver, surrounding vehicles, and base stations. The server also stores the driving data, generates optimal driving patterns, and reflects them in the vehicle system. This system makes it possible to monitor the driver's safety in real time and support safe driving.

[0885] Data collection and preprocessing

[0886] The device uses a camera and microphone installed in the vehicle to collect video and audio data in real time. The video data is captured mainly of the driver's face, while the audio data collects the driver's speech and surrounding sounds. The collected video data is converted to grayscale, and the audio data is converted to a buffer format, making the data suitable for analysis by the generative AI model.

[0887] Examples:

[0888] A camera records the driver's facial expressions in real time, and a microphone picks up the driver's tone of voice.

[0889] The collected color data is quickly converted to grayscale data, and the audio data is stored in a buffer every second.

[0890] Analysis by AI model

[0891] The server receives the preprocessed data and inputs it into a generative AI model. The generative AI model uses deep learning techniques to analyze the driver's facial features and voice patterns to detect signs of driver fatigue or unsafe driving. For example, it analyzes the frequency of the driver's eye opening and closing, facial expressions, and voice tone and rhythm.

[0892] Examples:

[0893] The server analyzes the frequency with which the driver opens and closes their eyelids from the video data to detect signs of fatigue.

[0894] Changes in the tone of the driver's voice are analyzed from audio data to detect signs of lack of attention.

[0895] Example prompts for generative AI models:

[0896] "Analyze pre-processed video and audio data to detect driver fatigue from facial expressions."

[0897] Risk information notification

[0898] Based on the analysis results of the generative AI model, the server issues a real-time warning if the driver is fatigued or if dangerous driving behavior is detected. Specifically, a warning message is displayed on the driver's device and an audio alert is given. The server also warns surrounding vehicles via wireless communication and sends situation reports to bases such as logistics centers.

[0899] Examples:

[0900] The server warns the driver, "Signs of dangerous driving have been detected. Please take a break."

[0901] Sends a message to surrounding vehicles saying, "Possible dangerous driving. Please be careful."

[0902] Example prompts for generative AI models:

[0903] "Generate a warning message to encourage drivers to take a break."

[0904] Data storage and generation of optimal driving patterns

[0905] The server stores all collected and analyzed driving data. This stored data is used for later analysis. Specifically, it statistically analyzes past driving data to generate an optimal driving pattern for the driver. This optimal driving pattern is then applied to the vehicle system to reduce driver fatigue and promote safe driving.

[0906] Examples:

[0907] The server stores each driver's analytical data in a driving database and identifies them with a unique ID.

[0908] Past driving data is analyzed to generate optimal driving patterns, such as "increasing rest time," and these are reflected in the vehicle's navigation system.

[0909] Example prompts for generative AI models:

[0910] "Analyze past driving data and generate optimal driving patterns."

[0911] The above is an embodiment of the present invention. This system can accurately grasp the driver's condition in real time and provide necessary warnings and optimizations to support safe driving.

[0912] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0913] (Program processing flow)

[0914] Step 1:

[0915] The device activates the camera and microphone installed in the vehicle and collects video and audio data in real time.

[0916] Input: In-car video and audio

[0917] Specific behavior:

[0918] The camera captures footage centered on the driver's face.

[0919] A microphone collects audio, including the driver's voice.

[0920] Output: Collected video and audio data

[0921] Step 2:

[0922] The terminal converts the collected video data into grayscale and converts the audio data into a buffer format.

[0923] Input: Collected video and audio data

[0924] Specific behavior:

[0925] Converts video data to grayscale.

[0926] The audio data is divided into fixed time intervals and stored in a buffer.

[0927] Output: Grayscale video data and buffered audio data

[0928] Step 3:

[0929] The terminal transmits the pre-processed data to the server.

[0930] Input: Grayscale video data and buffered audio data

[0931] Specific behavior:

[0932] Upload data to the server using Wi-Fi or mobile data.

[0933] You will receive a confirmation message of the submission.

[0934] Output: Preprocessed data sent to the server

[0935] Step 4:

[0936] The server inputs the received pre-processed data into the generative AI model and analyzes the driver's condition.

[0937] Input: Preprocessed data (grayscaled video data and buffered audio data)

[0938] Specific behavior:

[0939] Send the preprocessed data to the generative AI model.

[0940] The generative AI model analyzes the driver's eye opening and closing frequency, facial expressions, and voice tone and rhythm.

[0941] Output: Driver status analysis results

[0942] Step 5:

[0943] Based on the analysis results of the generative AI model, the server issues a warning in real time if the driver is fatigued or if dangerous driving behavior is detected.

[0944] Input: Driver condition analysis results

[0945] Specific behavior:

[0946] A warning message is sent to the driver's device.

[0947] Issues an audio alert.

[0948] Alerts are sent to surrounding vehicles and bases via radio communication.

[0949] Outputs: Warning messages, audio alerts, and wireless communication notifications

[0950] Step 6:

[0951] The server stores all collected and analyzed driving data.

[0952] Input: Driver status analysis results and other driving data

[0953] Specific behavior:

[0954] Save the analyzed data in a database.

[0955] A unique ID is assigned to the stored data so that it can be searched for later.

[0956] Output: Saved operating data

[0957] Step 7:

[0958] The server statistically analyzes past driving data based on the stored data and generates an optimal driving pattern for the driver.

[0959] Input: Saved historical driving data

[0960] Specific behavior:

[0961] Analyze historical data using statistical analysis algorithms.

[0962] Generate optimal driving patterns and provide feedback to vehicle systems.

[0963] Output: Generated optimal driving pattern

[0964] This allows the system to accurately grasp the driver's condition in real time and provide prompt and appropriate warnings and optimizations.

[0965] (Application example 1)

[0966] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0967] Conventional in-vehicle driver monitoring systems have difficulty accurately assessing the driver's condition in real time and issuing appropriate warnings. Furthermore, the lack of a system that monitors the driver's condition and sends appropriate alerts is a problem, particularly in autonomous vehicles. The present invention aims to solve these problems and provide a system that ensures the safety of the driver and surrounding vehicles.

[0968] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0969] In this invention, the server includes a means for receiving the preprocessed data and using an artificial intelligence model to analyze the driver's condition, a means for notifying the driver, surrounding vehicles, and bases of danger information based on the analysis results, and a means for storing the analysis results and driving data. This allows the server to monitor the driver's condition in real time and issue appropriate warnings to ensure the safety of the driver and those around them.

[0970] A "terminal" is a device that is installed in a vehicle and collects video and audio data in real time and performs preprocessing.

[0971] "Video data" refers to video information captured using a camera showing the conditions inside and outside the vehicle.

[0972] "Audio data" refers to audio information obtained using a microphone of the audio environment inside the vehicle, particularly the driver's voice and surrounding noise.

[0973] "Pre-processing" refers to the process of converting the collected video data to grayscale and converting the audio data into a suitable buffer format.

[0974] The "server" is a central control computer that receives the pre-processed data and analyzes the driver's condition using artificial intelligence models.

[0975] An "artificial intelligence model" is a software algorithm that uses deep learning techniques to analyze data and analyse the driver's facial features and voice patterns.

[0976] "Analysis results" are information derived from artificial intelligence models about driver fatigue and signs of dangerous driving.

[0977] "Notification" refers to the act of the server transmitting danger information to the driver, surrounding vehicles, and base stations based on the analysis results.

[0978] "Storage" refers to the act of recording the analysis results and operational data in a database or the like for use in later analysis.

[0979] A "driving pattern" is a guideline that indicates the optimal driving method for a driver, generated based on past driving data.

[0980] "Smart glasses" are wearable devices that are installed in the driver's seat and collect video and audio data.

[0981] A "deep learning model" is an algorithm based on a neural network with multiple layers that runs on a server.

[0982] An "alert" is a warning or caution message to the driver that is conveyed through audio or visual means.

[0983] The embodiments of the present invention will be described in detail below.

[0984] System configuration

[0985] The system of the present invention mainly comprises the following elements:

[0986] Terminals installed in vehicles

[0987] server

[0988] Smart Glasses

[0989] Deep Learning Model

[0990] Data collection and preprocessing

[0991] The terminal collects video and audio data in real time using the vehicle's onboard camera and microphone. The smart glasses acquire and preprocess the data, converting video data to grayscale and audio data to a suitable buffer format.

[0992] Data transmission and analysis

[0993] The pre-processed data is sent to a server via a computer installed inside the vehicle. The server then analyzes the received data using a deep learning model. This model analyzes the driver's facial features and voice patterns to detect signs of fatigue or dangerous driving. The artificial intelligence model uses frameworks such as TensorFlow and PyTorch.

[0994] Risk information notification

[0995] Based on the analysis results of the AI ​​model, the server issues a real-time warning if the driver is fatigued or if dangerous driving behavior is detected. Specifically, a warning message is displayed on the driver's device and an audio alert is given. The server also warns surrounding vehicles via wireless communication and sends a situation report to the base station.

[0996] Data storage and driving pattern generation

[0997] The server stores all collected and analyzed driving data, using cloud storage and databases. This stored data is then used for further analysis. Specifically, it statistically analyzes past driving data and generates an optimal driving pattern for the driver. This driving pattern is then applied to the vehicle systems to reduce driver fatigue and promote safe driving.

[0998] Specific examples

[0999] For example, if a driver has been driving continuously for a long period of time, the smart glasses will detect changes in the driver's facial expression and voice and send the data to a server. The server will then use an artificial intelligence model to analyze the data and, if it determines that the driver is fatigued, will issue a warning to the driver saying, "Fatigue has been detected. Please take a break." At the same time, a message will be sent to surrounding vehicles saying, "Possible dangerous driving. Please be careful," and a report will be sent to the base saying, "Driver ID 123 is fatigued, so please take appropriate action." The server will also use the collected driving data to generate optimal driving patterns for the driver in the future and apply them to the vehicle system, further improving safety.

[1000] Prompt Sentence Examples

[1001] An example prompt is:

[1002] "Please build an AI model system that uses the driver's facial features and voice patterns to analyze their fatigue level and risky driving behavior in real time and issue a warning to the vehicle system. Based on the invention, data will be collected from the camera and microphone and preprocessed into grayscale images and buffered audio data. Then, the data will be sent to the server and analyzed by deep learning. Please also provide specific instructions and examples of warning messages."

[1003] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1004] Step 1:

[1005] The device uses a camera and microphone to collect video and audio data from inside the vehicle in real time.

[1006] Input: In-vehicle video and audio

[1007] Output: Raw collected data (video and audio)

[1008] Specifically, the device activates the camera and microphone to continuously capture the driver's facial expressions and voice.

[1009] Step 2:

[1010] The terminal converts the collected video data to grayscale and converts the audio data to an appropriate buffer format.

[1011] Input: Raw collected data (video and audio)

[1012] Output: Preprocessed data (grayscale image and audio in buffer format)

[1013] Specifically, the video data is converted to grayscale using the OpenCV library, and the audio data is converted using Python's buffer function.

[1014] Step 3:

[1015] The pre-processed data is sent to a server via a computer inside the vehicle.

[1016] Input: Preprocessed data (grayscale images and audio in buffer format)

[1017] Output: Data sent to the server

[1018] Specifically, the preprocessed data is uploaded to the server using an HTTP POST request.

[1019] Step 4:

[1020] The data received by the server is input into an artificial intelligence model (deep learning model) to analyze the driver's condition.

[1021] Input: Received data (grayscale image and buffered audio)

[1022] Output: Analysis results (status information based on the driver's facial features and voice patterns)

[1023] Specifically, it analyzes data using deep learning frameworks such as TensorFlow and PyTorch to detect signs of fatigue and dangerous driving.

[1024] Step 5:

[1025] Based on the analysis results, the server displays a warning message on the driver's device and issues an audio alert, while also wirelessly alerting surrounding vehicles and sending a situation report to the base.

[1026] Input: Analysis results

[1027] Outputs: Warning messages, audio alerts, radio communication messages, situation reports

[1028] Specifically, it alerts the driver using SMS or application notifications, alerts surrounding vehicles using wireless communication protocols, and sends reports to a base station via email or dashboard.

[1029] Step 6:

[1030] The server stores the analysis results and operational data in a database.

[1031] Input: Analysis results and operating data

[1032] Output: Saved data

[1033] Specifically, the data is recorded in cloud storage or a relational database.

[1034] Step 7:

[1035] The server statistically analyzes the stored driving data, generates an optimal driving pattern for the driver, and applies it to the vehicle system.

[1036] Input: Saved data

[1037] Output: Optimal driving pattern, applied driving pattern

[1038] Specifically, it uses machine learning algorithms to analyze data, form driving patterns, and push them to the vehicle's systems for application.

[1039] The above are the specific processing steps of the system that realizes the application example.

[1040] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1041] This invention is a system in which a terminal installed in a vehicle collects video and audio data in real time and preprocesses that data. The preprocessed data is sent to a server, where the driver's condition is analyzed using an artificial intelligence model and an emotion engine. Based on the analysis results, danger information is then sent to the driver, surrounding vehicles, and base stations. The server also stores the driving data, generates optimal driving patterns, and reflects them in the vehicle system. This system makes it possible to monitor the driver's condition in real time and support safe driving.

[1042] Data collection and preprocessing

[1043] The terminal collects video and audio data in real time using the camera and microphone installed in the vehicle. The collected data is pre-processed on the terminal. Specifically, the video data is converted to grayscale and the audio data is converted to a buffer format. This completes the pre-processing required for subsequent analysis.

[1044] Analysis by AI models and sentiment engines

[1045] The server receives the preprocessed data and inputs it into the AI ​​model and emotion engine. The AI ​​model uses deep learning technology to analyze the driver's facial features and voice patterns to detect signs of fatigue or dangerous driving. Meanwhile, the emotion engine analyzes the driver's emotions from video and audio data. For example, it analyzes the driver's facial expressions and tone of voice to recognize emotions such as anger, joy, and sadness.

[1046] Risk information notification

[1047] Based on the analysis results of the AI ​​model and emotion engine, the server issues a real-time warning if the driver is fatigued or if dangerous driving behavior is detected. Specifically, it sends a notification message to the device, such as "Fatigue has been detected. Please take a break," and issues a voice alert. If the driver appears emotional (angry, sad, etc.), it provides appropriate feedback, such as "Please stay calm." Surrounding vehicles are alerted via wireless communication, and a situation report is sent to the base (e.g., logistics center).

[1048] Data storage and generation of optimal driving patterns

[1049] The server stores all collected and analyzed driving data. This stored data is used for later analysis. Specifically, it statistically analyzes past driving data to generate an optimal driving pattern for the driver. This optimal driving pattern is then applied to the vehicle system to reduce driver fatigue and promote safe driving.

[1050] Specific examples

[1051] For example, if a driver has been driving continuously for a long period of time, the device will detect signs of fatigue or anger from their facial expressions and voice. The collected data is sent to a server, where it is analyzed by an artificial intelligence model and emotion engine. If fatigue or anger is confirmed, the server will issue a warning to the driver, saying, "Fatigue has been detected. Please take a break and remain calm." At the same time, a message is sent to surrounding vehicles saying, "There is a possibility of dangerous driving. Please be careful," and a report is sent to the base saying, "Driver ID 123 is in a state of fatigue and agitation. Please take appropriate action." The server will also use the collected driving data to generate optimal driving patterns for the driver in the future and apply them to the vehicle system, further improving driver safety.

[1052] The processing flow will be explained below.

[1053] Step 1:

[1054] The device is installed inside the vehicle and uses a camera and microphone to collect video and audio data. Specifically, the camera captures the driver's face and the microphone records the driver's voice. This data is acquired in real time.

[1055] Step 2:

[1056] The device converts the video data collected by the device into grayscale and converts the audio data into an appropriate format. Specifically, the video data is converted into grayscale for easier processing, and the audio data is converted into a buffer format.

[1057] Step 3:

[1058] The device sends the pre-processed data to the server, where it is encrypted and then uploaded to the server, ensuring secure data transmission.

[1059] Step 4:

[1060] The server inputs the received data into an AI model and emotion engine. The AI ​​model uses deep learning technology to analyze the driver's facial features and voice patterns to detect signs of fatigue or dangerous driving. Meanwhile, the emotion engine recognizes the driver's emotions (joy, anger, sadness, etc.) from video and audio data.

[1061] Step 5:

[1062] The server receives the analysis results of the AI ​​model and emotion engine and determines the driver's state. Specifically, if the fatigue score or emotion score exceeds a certain threshold, it determines that the driver is in a fatigued or emotionally aroused state.

[1063] Step 6:

[1064] The server issues a warning to the driver based on the results of the assessment. Specifically, it sends a notification message to the device saying, "Fatigue has been detected. Please take a break. Please stay calm." It also issues a voice alert. It also sends a warning message to surrounding vehicles via wireless communication.

[1065] Step 7:

[1066] The server reports the driver's condition to the base. Specifically, it sends a message saying, "Driver ID 123 is in a state of fatigue and emotional arousal, so please take appropriate action." This allows the base to understand the driver's condition in real time.

[1067] Step 8:

[1068] The server stores the collected and analyzed driving data. The stored data is used for later analysis and optimization. Specifically, the driving data is stored in a database and later statistically analyzed.

[1069] Step 9:

[1070] The server generates optimal driving patterns based on the stored driving data. Specifically, it analyzes past driving data and suggests efficient and safe driving methods for the driver.

[1071] Step 10:

[1072] The server applies the generated optimal driving pattern to the vehicle system. Specifically, it reflects the optimal driving pattern in the vehicle's various control systems, reducing the burden on the driver and supporting safe driving.

[1073] Example 2

[1074] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1075] In recent years, there has been an increasing demand for systems that support safe driving. However, existing systems face challenges in accurately monitoring the driver's fatigue and emotional state in real time and providing appropriate feedback. Furthermore, existing systems lack the ability to effectively utilize driving data and generate optimal driving patterns. As a result, they are unable to respond quickly to changes in the driver's fatigue and emotions, which can increase the risk of traffic accidents.

[1076] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a preprocessing means for converting video data collected by the terminal into grayscale and audio data into a buffer format, a means for receiving the preprocessed data and using an artificial intelligence model to analyze the driver's condition, a means for analyzing the driver's emotions and fatigue state using the artificial intelligence model and an emotion engine, a means for notifying the driver, surrounding vehicles, and base stations of danger information based on the analysis results, a means for storing the analysis results and driving data, a means for generating an optimal driving pattern based on the stored driving data, and a means for applying the generated driving pattern to the vehicle system. This makes it possible to accurately monitor the driver's condition in real time and provide appropriate feedback. Furthermore, by effectively utilizing the stored driving data and generating and applying an optimal driving pattern for the driver, it is possible to promote safe driving and reduce the risk of traffic accidents.

[1077] A "terminal" is a device that is installed in a vehicle and collects video data and audio data in real time.

[1078] "Video data" refers to visual information that captures the driver's face and the interior of the vehicle, collected by a camera mounted on the device.

[1079] "Voice data" refers to the driver's voice and acoustic information inside the vehicle collected by a microphone installed in the terminal.

[1080] The "pre-processing means" is a processing function that converts collected video data into grayscale and audio data into a buffer format.

[1081] The "server" is a central management device that receives pre-processed data, analyzes it using an artificial intelligence model and emotion engine, and notifies and stores various information based on the analysis results.

[1082] The "artificial intelligence model" is an algorithm that uses deep learning technology to analyze a driver's facial features and voice patterns to detect signs of fatigue or dangerous driving.

[1083] The "emotion engine" is an algorithm that analyzes the driver's emotions from video and audio data and recognizes emotional states such as anger, joy, and sadness.

[1084] "Analysis results" are the assessments and diagnostic results of the driver's condition produced by the artificial intelligence model and emotion engine.

[1085] The "notification means" is a function that sends warnings and report messages to the driver, surrounding vehicles, and base stations based on the analysis results.

[1086] "Storage means" refers to a function for recording and storing analysis results and operating data.

[1087] The "optimal driving pattern" is a guideline for driving behavior that is generated through statistical analysis of stored driving data to improve driver safety and efficiency.

[1088] "Vehicle system" means a set of electronic devices and functions installed within a vehicle for monitoring and controlling driver behavior.

[1089] This invention is a system in which a terminal installed in a vehicle collects video and audio data in real time and preprocesses the data. The preprocessed data is sent to a server, where the driver's state is analyzed using an artificial intelligence model and an emotion engine. Based on the analysis results, danger information is then sent to the driver, surrounding vehicles, and base stations. The server then stores the driving data, generates optimal driving patterns, and applies them to the vehicle system.

[1090] Data collection and preprocessing

[1091] The terminal collects video and audio data in real time using a camera and microphone installed in the vehicle. Specific hardware includes a high-resolution camera and a highly sensitive microphone. The collected data is preprocessed on the terminal. Specifically, the video data is converted to grayscale, and the audio data is converted to a buffer format. This completes the preprocessing required for subsequent analysis.

[1092] Sending data

[1093] The terminal transmits the pre-processed data to the server. Specifically, the data is transmitted using wireless communication technology. Examples of specific technologies used here include LTE and WiFi. The collected data is appropriately compressed and transmitted efficiently.

[1094] Analysis by AI models and sentiment engines

[1095] The server receives the preprocessed data and inputs it into the AI ​​model and emotion engine. The AI ​​model uses deep learning technology to analyze the driver's facial features and voice patterns to detect signs of fatigue or dangerous driving. Specific software used is TensorFlow and PyTorch. Meanwhile, the emotion engine analyzes the driver's emotions from video and audio data. For example, it analyzes the driver's facial expressions and tone of voice to recognize emotions such as anger, joy, and sadness.

[1096] Risk information notification

[1097] Based on the analysis results of the AI ​​model and emotion engine, the server issues a real-time warning if the driver is fatigued or if dangerous driving behavior is detected. Specifically, it sends a notification message to the device, such as "Fatigue has been detected. Please take a break," and issues a voice alert. If the driver appears emotional, it also provides appropriate feedback, such as "Please stay calm." Surrounding vehicles are alerted via wireless communication, and a situation report is sent to the base (e.g., logistics center).

[1098] Data storage and generation of optimal driving patterns

[1099] The server stores all collected and analyzed driving data. This stored data is used for later analysis. Specifically, it statistically analyzes past driving data to generate an optimal driving pattern for the driver. This optimal driving pattern is then applied to the vehicle system to reduce driver fatigue and promote safe driving.

[1100] Specific examples

[1101] For example, if a driver has been driving continuously for a long period of time, the device will detect signs of fatigue or anger from their facial expressions and voice. The collected data is sent to a server, where it is analyzed by an artificial intelligence model and emotion engine. If fatigue or anger is confirmed, the server will issue a warning to the driver, saying, "Fatigue has been detected. Please take a break and remain calm." At the same time, a message is sent to surrounding vehicles saying, "There is a possibility of dangerous driving. Please be careful," and a report is sent to the base saying, "Driver ID 123 is in a state of fatigue and agitation. Please take appropriate action." The server will also use the collected driving data to generate optimal driving patterns for the driver in the future and apply them to the vehicle system, further improving driver safety.

[1102] Prompt Sentence Examples

[1103] "Your role is to understand the operation of the driver support system and analyze the following driving data: Detect emotions and signs of fatigue from the driver's facial expressions and voice data. Based on the results, generate warning messages for the driver and notifications to surrounding vehicles and bases."

[1104] Inquiry Data:

[1105] Video data: Grayscale image of the driver's face

[1106] Audio data: Driver's audio buffer

[1107] The above is a specific embodiment of this system.

[1108] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1109] Step 1: Collect data

[1110] The device uses a camera and microphone installed in the vehicle to collect video and audio data in real time. Specifically, the camera captures the driver's face and obtains high-resolution video data. The microphone captures the driver's voice and any sounds from inside the vehicle and stores them as audio data. This data is input into the device as soon as it is collected.

[1111] input:

[1112] Video data: Driver's face and interior of the vehicle

[1113] Voice data: Driver's voice and acoustic information inside the vehicle

[1114] output:

[1115] Raw video and audio data collected

[1116] Step 2: Preprocessing the data

[1117] The terminal converts the collected video data into grayscale format and audio data into buffer format. Specifically, color information is removed from the video data, and only brightness information is retained. Audio data is divided into segments of a certain length and stored in a buffer. This reduces the amount of data and makes subsequent analysis more efficient.

[1118] input:

[1119] Raw video data

[1120] Raw audio data

[1121] output:

[1122] Video data converted to grayscale

[1123] Audio data converted into buffer format

[1124] Step 3: Sending data

[1125] The device then sends the preprocessed data to the server. Specifically, the data is divided into packets and transmitted to the server using wireless communication technology (e.g., LTE or WiFi). The data is also compressed during transmission, improving communication efficiency.

[1126] input:

[1127] Video data converted to grayscale

[1128] Audio data converted into buffer format

[1129] output:

[1130] Pre-processed data sent to the server

[1131] Step 4: Analysis by AI models and sentiment engines

[1132] The server receives the preprocessed data and inputs it into an artificial intelligence model and emotion engine. Specifically, it uses deep learning algorithms to analyze the driver's facial features and voice patterns to detect signs of driver fatigue or dangerous driving. It then uses the emotion engine to analyze the driver's emotions.

[1133] input:

[1134] Grayscale video data

[1135] Buffered audio data

[1136] output:

[1137] Analysis of driver fatigue and emotional state

[1138] Step 5: Risk notification

[1139] Based on the analysis results, the server notifies the driver, surrounding vehicles, and the base station of any danger information. Specifically, it sends a real-time warning to the driver's device, such as "Fatigue detected. Please take a break." At the same time, it sends a warning message to surrounding vehicles via wireless communication and sends a detailed situation report to the base station.

[1140] input:

[1141] Analysis of driver fatigue and emotional state

[1142] output:

[1143] Driver warning message

[1144] Warning message to surrounding vehicles

[1145] Report the situation to the base

[1146] Step 6: Save your data

[1147] The server stores the analysis results and collected data, specifically by recording this information in a database for long-term data storage and subsequent analysis.

[1148] input:

[1149] Operational data and analysis results

[1150] output:

[1151] Stored database records

[1152] Step 7: Generate optimal driving patterns

[1153] The server statistically analyzes the stored data and generates optimal driving patterns. Specifically, it analyzes past driving data to determine driving behaviors suitable for a specific driver and generates patterns for reducing fatigue and driving safely.

[1154] input:

[1155] Saved driving data

[1156] output:

[1157] Optimal driving pattern

[1158] Step 8: Application to vehicle systems

[1159] The server applies the generated optimal driving pattern to the vehicle system, sending instructions to the vehicle's control system and navigating the vehicle in the optimal driving method for the driver in real time.

[1160] input:

[1161] Optimal driving pattern

[1162] output:

[1163] Driving assistance based on applied driving patterns

[1164] (Application example 2)

[1165] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1166] Many current vehicles lack systems that effectively detect driver fatigue and emotional changes in real time and support safe driving. As a result, there is a high risk of accidents occurring due to long driving times or emotional arousal. Furthermore, conventional systems lack a means to effectively communicate warnings to drivers and do not contribute sufficiently to accident prevention. Furthermore, systems that generate and apply optimal driving patterns for individual drivers are also rare. Effective methods to solve these problems are needed.

[1167] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving preprocessed data and using an artificial intelligence model to analyze the driver's state, means for notifying the driver, surrounding vehicles, and bases of danger information based on the analysis results, and means for using a wearable device to notify the driver of visual and audio warnings in real time. This makes it possible to support the driver in safe driving.

[1168] The "terminal mounted in a vehicle" is a device that is installed inside a vehicle and collects video data and audio data in real time.

[1169] "Video data" refers to visual information obtained from a visual sensor such as a camera.

[1170] "Audio data" refers to acoustic information obtained from an audio sensor such as a microphone.

[1171] "Preprocessing" refers to the process of converting collected data into a form suitable for subsequent analysis.

[1172] An "artificial intelligence model" is a model that includes deep learning algorithms used to analyze the driver's state.

[1173] A "server" is a central processing unit that receives, analyzes, and stores pre-processed data.

[1174] "Danger information" refers to information that warns or alerts drivers about their fatigue level or dangerous driving behavior.

[1175] "Storage" is the act of recording collected and analyzed data in a retrievable form.

[1176] An "optimal driving pattern" is a guideline for driving behavior that is generated to maximize driver safety and efficiency.

[1177] "Applying to vehicle systems" refers to reflecting the generated optimal driving patterns in actual driving operations and control systems.

[1178] A "wearable device" is a device worn by a driver that provides visual and audio notifications.

[1179] This invention provides a system that uses a terminal installed in a vehicle to collect video and audio data in real time, preprocess the data, and transmit it to a server. The server analyzes the driver's condition based on the preprocessed data using an artificial intelligence model and notifies the driver of hazardous situations. The server also stores the driving data, generates optimal driving patterns, and applies them to the vehicle system, supporting safe driving.

[1180] Data collection and preprocessing

[1181] The device uses a camera and microphone to collect in-car video and audio data in real time. The collected video data is converted to grayscale using software libraries such as OpenCV, and the audio data is converted to a buffer format using libraries such as SoundDevice. This preprocesses the collected data into a format optimal for subsequent analysis.

[1182] Analysis by AI models and sentiment engines

[1183] The server receives the preprocessed video and audio data and analyzes the driver's state using an AI model and emotion engine. Specifically, the AI ​​model uses deep learning technology to analyze the driver's facial features and voice patterns to detect signs of driver fatigue or dangerous driving. The emotion engine also analyzes the driver's facial expressions and voice tone to detect changes in emotion (anger, joy, sadness, etc.).

[1184] Risk information notification

[1185] The server notifies the driver of danger information in real time based on the analysis results. Specifically, it provides visual and audio warnings using a wearable device (e.g., smart glasses) worn by the driver. For example, if the driver is fatigued, a message saying "Fatigue detected. Please take a break" is displayed and an audio notification is given. At the same time, a warning message is sent via wireless communication to surrounding vehicles and bases such as logistics centers.

[1186] Data storage and generation of optimal driving patterns

[1187] The server stores all collected and analyzed driving data. This stored data is used for later analysis. Specifically, past driving data is statistically analyzed to generate an optimal driving pattern for each individual driver. This optimal driving pattern is then applied to the vehicle system to reduce driver fatigue and promote safe driving.

[1188] Specific examples

[1189] For example, if a driver has been driving continuously for a long period of time, the device will detect signs of fatigue from their facial expressions and voice. The collected data is sent to a server, where it is analyzed by an artificial intelligence model and emotion engine. If fatigue or agitation is confirmed, the server will issue a warning to the driver, saying, "Fatigue has been detected. Please take a break and stay calm." At the same time, a warning message is sent to surrounding vehicles, and a situation report is sent to the base station. Furthermore, the server can use the collected driving data to generate optimal driving patterns for the driver in the future and apply them to the vehicle system, further improving driver safety.

[1190] Prompt Sentence Examples

[1191] Create an AI model that can detect driver fatigue from video and audio data and provide real-time warnings during long driving sessions. Consider the following steps:

[1192] 1. Convert the video data to grayscale and perform preprocessing.

[1193] 2. Convert the audio data into a buffer format and perform preprocessing.

[1194] 3. The pre-processed data is sent to the server and analyzed by the AI ​​model and emotion engine.

[1195] 4. If fatigue or stress is detected, the driver will be notified with appropriate warning messages via audio and visual means.

[1196] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1197] Step 1:

[1198] The device uses a camera and microphone inside the vehicle to collect video and audio data in real time. Specifically, the camera captures the driver's face and the surrounding environment, while the microphone records the driver's voice and surrounding sounds. The input is video data from the camera and audio data from the microphone, which are then imported into the system in real time.

[1199] Step 2:

[1200] The video data collected by the device is converted to grayscale using OpenCV, and the audio data is converted to buffer format using SoundDevice. Specifically, color data is converted to grayscale, and audio signals are converted to digital data format. The input is color data and raw audio data, and the output is grayscale video data and buffer format audio data.

[1201] Step 3:

[1202] The preprocessed video and audio data are sent to the server. Specifically, the preprocessed data is packaged in JSON format and sent to the server via an HTTP request. The input is grayscale video data and buffered audio data, and the output is the data packet sent to the server.

[1203] Step 4:

[1204] The server inputs the received data into an artificial intelligence model and emotion engine to analyze the driver's condition. Specifically, it uses facial feature recognition algorithms and voice pattern analysis to detect fatigue and changes in emotion. The input is preprocessed video and audio data, and the output is the analysis result regarding the driver's condition.

[1205] Step 5:

[1206] The server generates a message to warn the driver in real time based on the analysis results and sends it to the wearable device. Specifically, if fatigue or dangerous driving is detected, an appropriate warning message is sent visually and audibly. The input is the analysis results, and the output is the warning message.

[1207] Step 6:

[1208] The server saves the analysis results and operational data. Specifically, it stores the results in a database and keeps them as records for later analysis. The input is the analysis results and operational data, and the output is the data saved in the database.

[1209] Step 7:

[1210] The server analyzes the stored driving data and generates optimal driving patterns. Specifically, it uses statistical methods and machine learning algorithms to optimize driving patterns and generate individual patterns for each driver. The input is the stored driving data, and the output is the optimal driving pattern.

[1211] Step 8:

[1212] The generated optimal driving pattern is applied to the vehicle system. Specifically, it is reflected in the vehicle's control system and appropriate feedback is provided to the driver. The input is the optimal driving pattern, and the output is the operation of the driving assistance system according to the applied driving pattern.

[1213] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1214] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1215] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1216] [Fourth embodiment]

[1217] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1218] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1219] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1220] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1221] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1222] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1223] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1224] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1225] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1226] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1227] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1228] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1229] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1230] This invention is a system in which a terminal installed in a vehicle collects video and audio data in real time and preprocesses the data. The preprocessed data is sent to a server, where the driver's condition is analyzed using an artificial intelligence model. Based on the analysis results, danger information is then sent to the driver, surrounding vehicles, and base stations. The server also stores the driving data, generates optimal driving patterns, and reflects them in the vehicle system. This system makes it possible to monitor the driver's safety in real time and support safe driving.

[1231] Data collection and preprocessing

[1232] The terminal collects video and audio data in real time using the camera and microphone installed in the vehicle. The collected data is first pre-processed on the terminal. Specifically, video data is converted to grayscale and audio data is converted to a buffer format. This pre-processing makes the data suitable for analysis by artificial intelligence models.

[1233] Analysis by AI model

[1234] The server receives the preprocessed data and inputs it into an AI model, which uses deep learning technology to analyze the driver's facial features and voice patterns to detect signs of fatigue or dangerous driving, such as the frequency with which the driver opens and closes their eyes, facial expressions, and the tone and rhythm of their voice.

[1235] Risk information notification

[1236] Based on the analysis results of the AI ​​model, the server issues a real-time warning if the driver is fatigued or if dangerous driving behavior is detected. Specifically, a warning message is displayed on the driver's device and an audio alert is given. The server also warns surrounding vehicles via wireless communication and sends a situation report to a base (e.g., a logistics center).

[1237] Data storage and generation of optimal driving patterns

[1238] The server stores all collected and analyzed driving data. This stored data is used for later analysis. Specifically, it statistically analyzes past driving data to generate an optimal driving pattern for the driver. This optimal driving pattern is then applied to the vehicle system to reduce driver fatigue and promote safe driving.

[1239] Specific examples

[1240] For example, if a driver has been driving continuously for a long period of time, the device will detect signs of fatigue from their facial expressions and voice. The collected data is sent to a server, where an artificial intelligence model analyzes it. If fatigue is confirmed, the server will issue a warning to the driver saying, "Fatigue has been detected. Please take a break." At the same time, a message is sent to surrounding vehicles saying, "There is a possibility of dangerous driving. Please be careful," and a report is sent to the base saying, "Driver ID 123 is in a state of fatigue, so please take appropriate action." The server will also use the collected driving data to generate optimal driving patterns for the driver in the future and apply them to the vehicle system, further improving driver safety.

[1241] The processing flow will be explained below.

[1242] Step 1:

[1243] The device collects video and audio data inside the vehicle. Specifically, a camera installed inside the vehicle captures the driver's video and a microphone records the driver's voice. This data is acquired in real time.

[1244] Step 2:

[1245] The terminal preprocesses the collected video and audio data. Specifically, it converts video data to grayscale and audio data to an appropriate format (e.g., integer buffer). This completes the preprocessing required for subsequent analysis.

[1246] Step 3:

[1247] The device sends the pre-processed data to the server, where it is encrypted for secure transmission and uploaded to the server, ready for real-time driver status monitoring.

[1248] Step 4:

[1249] The server inputs the received pre-processed data into an AI model for analysis. The AI ​​model uses deep learning technology to analyze the driver's facial features and voice patterns, thereby detecting signs of driver fatigue and dangerous driving.

[1250] Step 5:

[1251] The server determines risk information based on the analysis results of the AI ​​model. Specifically, if the fatigue score or dangerous driving score indicated by the model exceeds a certain threshold, it determines that the driver is fatigued.

[1252] Step 6:

[1253] The server issues a warning to the driver based on the results of the assessment. Specifically, it sends a notification message to the device saying, "Fatigue has been detected. Please take a break," and issues a voice alert. It also sends wireless communications to nearby vehicles to warn them.

[1254] Step 7:

[1255] The server reports the driver's condition to the base. Specifically, it sends a notification saying, "Driver ID 123 is in a fatigued state, so please take appropriate action." This allows the base to understand the driver's condition in real time.

[1256] Step 8:

[1257] The server stores the collected and analyzed driving data, which is then used for later analysis and optimization. Specifically, the driving data is stored in a database and later statistically analyzed.

[1258] Step 9:

[1259] The server generates optimal driving patterns based on the stored driving data. Specifically, it analyzes past driving data and derives efficient and safe driving methods for the driver.

[1260] Step 10:

[1261] The server applies the generated optimal driving pattern to the vehicle system. Specifically, the pattern is reflected in the vehicle's various control systems, reducing the burden on the driver and supporting safe driving.

[1262] Example 1

[1263] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1264] Conventional vehicle driving monitoring systems have had difficulty accurately grasping the driver's condition in real time and issuing appropriate warnings or notifications. In particular, there is a lack of technology that can quickly detect signs of driver fatigue or dangerous driving and reliably issue warnings, making it difficult to prevent serious traffic accidents.

[1265] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1266] In this invention, the server includes means for receiving the preprocessed data and using a generative AI model to analyze the driver's condition, means for notifying the driver, surrounding vehicles, and bases of danger information based on the analysis results, means for saving the analysis results and driving data, means for analyzing the driver's eye opening and closing frequency and facial expressions from the video data, and means for analyzing the tone and rhythm of the voice from the audio data. This makes it possible to accurately grasp the driver's condition in real time and issue warnings and notifications quickly and appropriately.

[1267] A "terminal" is a device that is installed in a vehicle and collects video data and audio data in real time.

[1268] The "server" is a central computing device that receives the pre-processed data, analyzes the driver's state using a generative AI model, and issues warnings or notifications based on the results.

[1269] A "generative AI model" is a type of artificial intelligence used to analyze video and audio data of a driver and evaluate the driver's condition.

[1270] The "pre-processing means" is a process that converts the collected video data to grayscale and converts the audio data into a suitable buffer format.

[1271] "Means of analysis" refers to the process of inputting the collected pre-processed data into an AI model and evaluating the driver's condition by analyzing the frequency of eye opening and closing, facial expressions, and tone and rhythm of voice.

[1272] "Means for notifying danger information" refers to a means of sending warning messages and alerts to drivers, surrounding vehicles, and bases such as logistics centers based on the analysis results of the AI ​​model.

[1273] "Means for storing data" refers to the process of storing the analyzed driving data in a database or the like and using it for later analysis or for generating optimal driving patterns.

[1274] The "means for generating optimal driving patterns" is a process of statistically analyzing past driving data to derive driving patterns that reduce driver fatigue and promote safe driving.

[1275] The "means for applying the driving pattern" is a process for reflecting the generated optimal driving pattern in the vehicle system.

[1276] "Collecting in real time" refers to obtaining video and audio data instantly without any specific time delay during driving.

[1277] "Video data" refers to visual information that records the inside of a vehicle and the driver's face and expressions.

[1278] "Voice data" refers to auditory information that records the acoustic environment inside the vehicle and the driver's voice.

[1279] "Converting to grayscale" is the process of converting color video data to black and white brightness only.

[1280] "Converting to a buffer format" is the process of temporarily storing audio data and later converting it into a format that is easy to analyze and transmit.

[1281] "Driver status" is an assessment of the driver's current state, based on an analysis of their fatigue, attention, facial expressions, and voice patterns, and is necessary for driving behavior.

[1282] This invention is a system in which a terminal installed in a vehicle collects video and audio data in real time and preprocesses that data. The preprocessed data is sent to a server, where the driver's condition is analyzed using a generative AI model. Based on the analysis results, danger information is then sent to the driver, surrounding vehicles, and base stations. The server also stores the driving data, generates optimal driving patterns, and reflects them in the vehicle system. This system makes it possible to monitor the driver's safety in real time and support safe driving.

[1283] Data collection and preprocessing

[1284] The device uses a camera and microphone installed in the vehicle to collect video and audio data in real time. The video data is captured mainly of the driver's face, while the audio data collects the driver's speech and surrounding sounds. The collected video data is converted to grayscale, and the audio data is converted to a buffer format, making the data suitable for analysis by the generative AI model.

[1285] Examples:

[1286] A camera records the driver's facial expressions in real time, and a microphone picks up the driver's tone of voice.

[1287] The collected color data is quickly converted to grayscale data, and the audio data is stored in a buffer every second.

[1288] Analysis by AI model

[1289] The server receives the preprocessed data and inputs it into a generative AI model. The generative AI model uses deep learning techniques to analyze the driver's facial features and voice patterns to detect signs of driver fatigue or unsafe driving. For example, it analyzes the frequency of the driver's eye opening and closing, facial expressions, and voice tone and rhythm.

[1290] Examples:

[1291] The server analyzes the frequency with which the driver opens and closes their eyelids from the video data to detect signs of fatigue.

[1292] Changes in the tone of the driver's voice are analyzed from audio data to detect signs of lack of attention.

[1293] Example prompts for generative AI models:

[1294] "Analyze pre-processed video and audio data to detect driver fatigue from facial expressions."

[1295] Risk information notification

[1296] Based on the analysis results of the generative AI model, the server issues a real-time warning if the driver is fatigued or if dangerous driving behavior is detected. Specifically, a warning message is displayed on the driver's device and an audio alert is given. The server also warns surrounding vehicles via wireless communication and sends situation reports to bases such as logistics centers.

[1297] Examples:

[1298] The server warns the driver, "Signs of dangerous driving have been detected. Please take a break."

[1299] Sends a message to surrounding vehicles saying, "Possible dangerous driving. Please be careful."

[1300] Example prompts for generative AI models:

[1301] "Generate a warning message to encourage drivers to take a break."

[1302] Data storage and generation of optimal driving patterns

[1303] The server stores all collected and analyzed driving data. This stored data is used for later analysis. Specifically, it statistically analyzes past driving data to generate an optimal driving pattern for the driver. This optimal driving pattern is then applied to the vehicle system to reduce driver fatigue and promote safe driving.

[1304] Examples:

[1305] The server stores each driver's analytical data in a driving database and identifies them with a unique ID.

[1306] Past driving data is analyzed to generate optimal driving patterns, such as "increasing rest time," and these are reflected in the vehicle's navigation system.

[1307] Example prompts for generative AI models:

[1308] "Analyze past driving data and generate optimal driving patterns."

[1309] The above is an embodiment of the present invention. This system can accurately grasp the driver's condition in real time and provide necessary warnings and optimizations to support safe driving.

[1310] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1311] (Program processing flow)

[1312] Step 1:

[1313] The device activates the camera and microphone installed in the vehicle and collects video and audio data in real time.

[1314] Input: In-car video and audio

[1315] Specific behavior:

[1316] The camera captures footage centered on the driver's face.

[1317] A microphone collects audio, including the driver's voice.

[1318] Output: Collected video and audio data

[1319] Step 2:

[1320] The terminal converts the collected video data into grayscale and converts the audio data into a buffer format.

[1321] Input: Collected video and audio data

[1322] Specific behavior:

[1323] Converts video data to grayscale.

[1324] The audio data is divided into fixed time intervals and stored in a buffer.

[1325] Output: Grayscale video data and buffered audio data

[1326] Step 3:

[1327] The terminal transmits the pre-processed data to the server.

[1328] Input: Grayscale video data and buffered audio data

[1329] Specific behavior:

[1330] Upload data to the server using Wi-Fi or mobile data.

[1331] You will receive a confirmation message of the submission.

[1332] Output: Preprocessed data sent to the server

[1333] Step 4:

[1334] The server inputs the received pre-processed data into the generative AI model and analyzes the driver's condition.

[1335] Input: Preprocessed data (grayscaled video data and buffered audio data)

[1336] Specific behavior:

[1337] Send the preprocessed data to the generative AI model.

[1338] The generative AI model analyzes the driver's eye opening and closing frequency, facial expressions, and voice tone and rhythm.

[1339] Output: Driver status analysis results

[1340] Step 5:

[1341] Based on the analysis results of the generative AI model, the server issues a warning in real time if the driver is fatigued or if dangerous driving behavior is detected.

[1342] Input: Driver condition analysis results

[1343] Specific behavior:

[1344] A warning message is sent to the driver's device.

[1345] Issues an audio alert.

[1346] Alerts are sent to surrounding vehicles and bases via radio communication.

[1347] Outputs: Warning messages, audio alerts, and wireless communication notifications

[1348] Step 6:

[1349] The server stores all collected and analyzed driving data.

[1350] Input: Driver status analysis results and other driving data

[1351] Specific behavior:

[1352] Save the analyzed data in a database.

[1353] A unique ID is assigned to the stored data so that it can be searched for later.

[1354] Output: Saved operating data

[1355] Step 7:

[1356] The server statistically analyzes past driving data based on the stored data and generates an optimal driving pattern for the driver.

[1357] Input: Saved historical driving data

[1358] Specific behavior:

[1359] Analyze historical data using statistical analysis algorithms.

[1360] Generate optimal driving patterns and provide feedback to vehicle systems.

[1361] Output: Generated optimal driving pattern

[1362] This allows the system to accurately grasp the driver's condition in real time and provide prompt and appropriate warnings and optimizations.

[1363] (Application example 1)

[1364] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1365] Conventional in-vehicle driver monitoring systems have difficulty accurately assessing the driver's condition in real time and issuing appropriate warnings. Furthermore, the lack of a system that monitors the driver's condition and sends appropriate alerts is a problem, particularly in autonomous vehicles. The present invention aims to solve these problems and provide a system that ensures the safety of the driver and surrounding vehicles.

[1366] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1367] In this invention, the server includes a means for receiving the preprocessed data and using an artificial intelligence model to analyze the driver's condition, a means for notifying the driver, surrounding vehicles, and bases of danger information based on the analysis results, and a means for storing the analysis results and driving data. This allows the server to monitor the driver's condition in real time and issue appropriate warnings to ensure the safety of the driver and those around them.

[1368] A "terminal" is a device that is installed in a vehicle and collects video and audio data in real time and performs preprocessing.

[1369] "Video data" refers to video information captured using a camera showing the conditions inside and outside the vehicle.

[1370] "Audio data" refers to audio information obtained using a microphone of the audio environment inside the vehicle, particularly the driver's voice and surrounding noise.

[1371] "Pre-processing" refers to the process of converting the collected video data to grayscale and converting the audio data into a suitable buffer format.

[1372] The "server" is a central control computer that receives the pre-processed data and analyzes the driver's condition using artificial intelligence models.

[1373] An "artificial intelligence model" is a software algorithm that uses deep learning techniques to analyze data and analyse the driver's facial features and voice patterns.

[1374] "Analysis results" are information derived from artificial intelligence models about driver fatigue and signs of dangerous driving.

[1375] "Notification" refers to the act of the server transmitting danger information to the driver, surrounding vehicles, and base stations based on the analysis results.

[1376] "Storage" refers to the act of recording the analysis results and operational data in a database or the like for use in later analysis.

[1377] A "driving pattern" is a guideline that indicates the optimal driving method for a driver, generated based on past driving data.

[1378] "Smart glasses" are wearable devices that are installed in the driver's seat and collect video and audio data.

[1379] A "deep learning model" is an algorithm based on a neural network with multiple layers that runs on a server.

[1380] An "alert" is a warning or caution message to the driver that is conveyed through audio or visual means.

[1381] The embodiments of the present invention will be described in detail below.

[1382] System configuration

[1383] The system of the present invention mainly comprises the following elements:

[1384] Terminals installed in vehicles

[1385] server

[1386] Smart Glasses

[1387] Deep Learning Model

[1388] Data collection and preprocessing

[1389] The terminal collects video and audio data in real time using the vehicle's onboard camera and microphone. The smart glasses acquire and preprocess the data, converting video data to grayscale and audio data to a suitable buffer format.

[1390] Data transmission and analysis

[1391] The pre-processed data is sent to a server via a computer installed inside the vehicle. The server then analyzes the received data using a deep learning model. This model analyzes the driver's facial features and voice patterns to detect signs of fatigue or dangerous driving. The artificial intelligence model uses frameworks such as TensorFlow and PyTorch.

[1392] Risk information notification

[1393] Based on the analysis results of the AI ​​model, the server issues a real-time warning if the driver is fatigued or if dangerous driving behavior is detected. Specifically, a warning message is displayed on the driver's device and an audio alert is given. The server also warns surrounding vehicles via wireless communication and sends a situation report to the base station.

[1394] Data storage and driving pattern generation

[1395] The server stores all collected and analyzed driving data, using cloud storage and databases. This stored data is then used for further analysis. Specifically, it statistically analyzes past driving data and generates an optimal driving pattern for the driver. This driving pattern is then applied to the vehicle systems to reduce driver fatigue and promote safe driving.

[1396] Specific examples

[1397] For example, if a driver has been driving continuously for a long period of time, the smart glasses will detect changes in the driver's facial expression and voice and send the data to a server. The server will then use an artificial intelligence model to analyze the data and, if it determines that the driver is fatigued, will issue a warning to the driver saying, "Fatigue has been detected. Please take a break." At the same time, a message will be sent to surrounding vehicles saying, "Possible dangerous driving. Please be careful," and a report will be sent to the base saying, "Driver ID 123 is fatigued, so please take appropriate action." The server will also use the collected driving data to generate optimal driving patterns for the driver in the future and apply them to the vehicle system, further improving safety.

[1398] Prompt Sentence Examples

[1399] An example prompt is:

[1400] "Please build an AI model system that uses the driver's facial features and voice patterns to analyze their fatigue level and risky driving behavior in real time and issue a warning to the vehicle system. Based on the invention, data will be collected from the camera and microphone and preprocessed into grayscale images and buffered audio data. Then, the data will be sent to the server and analyzed by deep learning. Please also provide specific instructions and examples of warning messages."

[1401] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1402] Step 1:

[1403] The device uses a camera and microphone to collect video and audio data from inside the vehicle in real time.

[1404] Input: In-vehicle video and audio

[1405] Output: Raw collected data (video and audio)

[1406] Specifically, the device activates the camera and microphone to continuously capture the driver's facial expressions and voice.

[1407] Step 2:

[1408] The terminal converts the collected video data to grayscale and converts the audio data to an appropriate buffer format.

[1409] Input: Raw collected data (video and audio)

[1410] Output: Preprocessed data (grayscale image and audio in buffer format)

[1411] Specifically, the video data is converted to grayscale using the OpenCV library, and the audio data is converted using Python's buffer function.

[1412] Step 3:

[1413] The pre-processed data is sent to a server via a computer inside the vehicle.

[1414] Input: Preprocessed data (grayscale images and audio in buffer format)

[1415] Output: Data sent to the server

[1416] Specifically, the preprocessed data is uploaded to the server using an HTTP POST request.

[1417] Step 4:

[1418] The data received by the server is input into an artificial intelligence model (deep learning model) to analyze the driver's condition.

[1419] Input: Received data (grayscale image and buffered audio)

[1420] Output: Analysis results (status information based on the driver's facial features and voice patterns)

[1421] Specifically, it analyzes data using deep learning frameworks such as TensorFlow and PyTorch to detect signs of fatigue and dangerous driving.

[1422] Step 5:

[1423] Based on the analysis results, the server displays a warning message on the driver's device and issues an audio alert, while also wirelessly alerting surrounding vehicles and sending a situation report to the base.

[1424] Input: Analysis results

[1425] Outputs: Warning messages, audio alerts, radio communication messages, situation reports

[1426] Specifically, it alerts the driver using SMS or application notifications, alerts surrounding vehicles using wireless communication protocols, and sends reports to a base station via email or dashboard.

[1427] Step 6:

[1428] The server stores the analysis results and operational data in a database.

[1429] Input: Analysis results and operating data

[1430] Output: Saved data

[1431] Specifically, the data is recorded in cloud storage or a relational database.

[1432] Step 7:

[1433] The server statistically analyzes the stored driving data, generates an optimal driving pattern for the driver, and applies it to the vehicle system.

[1434] Input: Saved data

[1435] Output: Optimal driving pattern, applied driving pattern

[1436] Specifically, it uses machine learning algorithms to analyze data, form driving patterns, and push them to the vehicle's systems for application.

[1437] The above are the specific processing steps of the system that realizes the application example.

[1438] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1439] This invention is a system in which a terminal installed in a vehicle collects video and audio data in real time and preprocesses that data. The preprocessed data is sent to a server, where the driver's condition is analyzed using an artificial intelligence model and an emotion engine. Based on the analysis results, danger information is then sent to the driver, surrounding vehicles, and base stations. The server also stores the driving data, generates optimal driving patterns, and reflects them in the vehicle system. This system makes it possible to monitor the driver's condition in real time and support safe driving.

[1440] Data collection and preprocessing

[1441] The terminal collects video and audio data in real time using the camera and microphone installed in the vehicle. The collected data is pre-processed on the terminal. Specifically, the video data is converted to grayscale and the audio data is converted to a buffer format. This completes the pre-processing required for subsequent analysis.

[1442] Analysis by AI models and sentiment engines

[1443] The server receives the preprocessed data and inputs it into the AI ​​model and emotion engine. The AI ​​model uses deep learning technology to analyze the driver's facial features and voice patterns to detect signs of fatigue or dangerous driving. Meanwhile, the emotion engine analyzes the driver's emotions from video and audio data. For example, it analyzes the driver's facial expressions and tone of voice to recognize emotions such as anger, joy, and sadness.

[1444] Risk information notification

[1445] Based on the analysis results of the AI ​​model and emotion engine, the server issues a real-time warning if the driver is fatigued or if dangerous driving behavior is detected. Specifically, it sends a notification message to the device, such as "Fatigue has been detected. Please take a break," and issues a voice alert. If the driver appears emotional (angry, sad, etc.), it provides appropriate feedback, such as "Please stay calm." Surrounding vehicles are alerted via wireless communication, and a situation report is sent to the base (e.g., logistics center).

[1446] Data storage and generation of optimal driving patterns

[1447] The server stores all collected and analyzed driving data. This stored data is used for later analysis. Specifically, it statistically analyzes past driving data to generate an optimal driving pattern for the driver. This optimal driving pattern is then applied to the vehicle system to reduce driver fatigue and promote safe driving.

[1448] Specific examples

[1449] For example, if a driver has been driving continuously for a long period of time, the device will detect signs of fatigue or anger from their facial expressions and voice. The collected data is sent to a server, where it is analyzed by an artificial intelligence model and emotion engine. If fatigue or anger is confirmed, the server will issue a warning to the driver, saying, "Fatigue has been detected. Please take a break and remain calm." At the same time, a message is sent to surrounding vehicles saying, "There is a possibility of dangerous driving. Please be careful," and a report is sent to the base saying, "Driver ID 123 is in a state of fatigue and agitation. Please take appropriate action." The server will also use the collected driving data to generate optimal driving patterns for the driver in the future and apply them to the vehicle system, further improving driver safety.

[1450] The processing flow will be explained below.

[1451] Step 1:

[1452] The device is installed inside the vehicle and uses a camera and microphone to collect video and audio data. Specifically, the camera captures the driver's face and the microphone records the driver's voice. This data is acquired in real time.

[1453] Step 2:

[1454] The device converts the video data collected by the device into grayscale and converts the audio data into an appropriate format. Specifically, the video data is converted into grayscale for easier processing, and the audio data is converted into a buffer format.

[1455] Step 3:

[1456] The device sends the pre-processed data to the server, where it is encrypted and then uploaded to the server, ensuring secure data transmission.

[1457] Step 4:

[1458] The server inputs the received data into an AI model and emotion engine. The AI ​​model uses deep learning technology to analyze the driver's facial features and voice patterns to detect signs of fatigue or dangerous driving. Meanwhile, the emotion engine recognizes the driver's emotions (joy, anger, sadness, etc.) from video and audio data.

[1459] Step 5:

[1460] The server receives the analysis results of the AI ​​model and emotion engine and determines the driver's state. Specifically, if the fatigue score or emotion score exceeds a certain threshold, it determines that the driver is in a fatigued or emotionally aroused state.

[1461] Step 6:

[1462] The server issues a warning to the driver based on the results of the assessment. Specifically, it sends a notification message to the device saying, "Fatigue has been detected. Please take a break. Please stay calm." It also issues a voice alert. It also sends a warning message to surrounding vehicles via wireless communication.

[1463] Step 7:

[1464] The server reports the driver's condition to the base. Specifically, it sends a message saying, "Driver ID 123 is in a state of fatigue and emotional arousal, so please take appropriate action." This allows the base to understand the driver's condition in real time.

[1465] Step 8:

[1466] The server stores the collected and analyzed driving data. The stored data is used for later analysis and optimization. Specifically, the driving data is stored in a database and later statistically analyzed.

[1467] Step 9:

[1468] The server generates optimal driving patterns based on the stored driving data. Specifically, it analyzes past driving data and suggests efficient and safe driving methods for the driver.

[1469] Step 10:

[1470] The server applies the generated optimal driving pattern to the vehicle system. Specifically, it reflects the optimal driving pattern in the vehicle's various control systems, reducing the burden on the driver and supporting safe driving.

[1471] Example 2

[1472] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1473] In recent years, there has been an increasing demand for systems that support safe driving. However, existing systems face challenges in accurately monitoring the driver's fatigue and emotional state in real time and providing appropriate feedback. Furthermore, existing systems lack the ability to effectively utilize driving data and generate optimal driving patterns. As a result, they are unable to respond quickly to changes in the driver's fatigue and emotions, which can increase the risk of traffic accidents.

[1474] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a preprocessing means for converting video data collected by the terminal into grayscale and audio data into a buffer format, a means for receiving the preprocessed data and using an artificial intelligence model to analyze the driver's condition, a means for analyzing the driver's emotions and fatigue state using the artificial intelligence model and an emotion engine, a means for notifying the driver, surrounding vehicles, and base stations of danger information based on the analysis results, a means for storing the analysis results and driving data, a means for generating an optimal driving pattern based on the stored driving data, and a means for applying the generated driving pattern to the vehicle system. This makes it possible to accurately monitor the driver's condition in real time and provide appropriate feedback. Furthermore, by effectively utilizing the stored driving data and generating and applying an optimal driving pattern for the driver, it is possible to promote safe driving and reduce the risk of traffic accidents.

[1475] A "terminal" is a device that is installed in a vehicle and collects video data and audio data in real time.

[1476] "Video data" refers to visual information that captures the driver's face and the interior of the vehicle, collected by a camera mounted on the device.

[1477] "Voice data" refers to the driver's voice and acoustic information inside the vehicle collected by a microphone installed in the terminal.

[1478] The "pre-processing means" is a processing function that converts collected video data into grayscale and audio data into a buffer format.

[1479] The "server" is a central management device that receives pre-processed data, analyzes it using an artificial intelligence model and emotion engine, and notifies and stores various information based on the analysis results.

[1480] The "artificial intelligence model" is an algorithm that uses deep learning technology to analyze a driver's facial features and voice patterns to detect signs of fatigue or dangerous driving.

[1481] The "emotion engine" is an algorithm that analyzes the driver's emotions from video and audio data and recognizes emotional states such as anger, joy, and sadness.

[1482] "Analysis results" are the assessments and diagnostic results of the driver's condition produced by the artificial intelligence model and emotion engine.

[1483] The "notification means" is a function that sends warnings and report messages to the driver, surrounding vehicles, and base stations based on the analysis results.

[1484] "Storage means" refers to a function for recording and storing analysis results and operating data.

[1485] The "optimal driving pattern" is a guideline for driving behavior that is generated through statistical analysis of stored driving data to improve driver safety and efficiency.

[1486] "Vehicle system" means a set of electronic devices and functions installed within a vehicle for monitoring and controlling driver behavior.

[1487] This invention is a system in which a terminal installed in a vehicle collects video and audio data in real time and preprocesses the data. The preprocessed data is sent to a server, where the driver's state is analyzed using an artificial intelligence model and an emotion engine. Based on the analysis results, danger information is then sent to the driver, surrounding vehicles, and base stations. The server then stores the driving data, generates optimal driving patterns, and applies them to the vehicle system.

[1488] Data collection and preprocessing

[1489] The terminal collects video and audio data in real time using a camera and microphone installed in the vehicle. Specific hardware includes a high-resolution camera and a highly sensitive microphone. The collected data is preprocessed on the terminal. Specifically, the video data is converted to grayscale, and the audio data is converted to a buffer format. This completes the preprocessing required for subsequent analysis.

[1490] Sending data

[1491] The terminal transmits the pre-processed data to the server. Specifically, the data is transmitted using wireless communication technology. Examples of specific technologies used here include LTE and WiFi. The collected data is appropriately compressed and transmitted efficiently.

[1492] Analysis by AI models and sentiment engines

[1493] The server receives the preprocessed data and inputs it into the AI ​​model and emotion engine. The AI ​​model uses deep learning technology to analyze the driver's facial features and voice patterns to detect signs of fatigue or dangerous driving. Specific software used is TensorFlow and PyTorch. Meanwhile, the emotion engine analyzes the driver's emotions from video and audio data. For example, it analyzes the driver's facial expressions and tone of voice to recognize emotions such as anger, joy, and sadness.

[1494] Risk information notification

[1495] Based on the analysis results of the AI ​​model and emotion engine, the server issues a real-time warning if the driver is fatigued or if dangerous driving behavior is detected. Specifically, it sends a notification message to the device, such as "Fatigue has been detected. Please take a break," and issues a voice alert. If the driver appears emotional, it also provides appropriate feedback, such as "Please stay calm." Surrounding vehicles are alerted via wireless communication, and a situation report is sent to the base (e.g., logistics center).

[1496] Data storage and generation of optimal driving patterns

[1497] The server stores all collected and analyzed driving data. This stored data is used for later analysis. Specifically, it statistically analyzes past driving data to generate an optimal driving pattern for the driver. This optimal driving pattern is then applied to the vehicle system to reduce driver fatigue and promote safe driving.

[1498] Specific examples

[1499] For example, if a driver has been driving continuously for a long period of time, the device will detect signs of fatigue or anger from their facial expressions and voice. The collected data is sent to a server, where it is analyzed by an artificial intelligence model and emotion engine. If fatigue or anger is confirmed, the server will issue a warning to the driver, saying, "Fatigue has been detected. Please take a break and remain calm." At the same time, a message is sent to surrounding vehicles saying, "There is a possibility of dangerous driving. Please be careful," and a report is sent to the base saying, "Driver ID 123 is in a state of fatigue and agitation. Please take appropriate action." The server will also use the collected driving data to generate optimal driving patterns for the driver in the future and apply them to the vehicle system, further improving driver safety.

[1500] Prompt Sentence Examples

[1501] "Your role is to understand the operation of the driver support system and analyze the following driving data: Detect emotions and signs of fatigue from the driver's facial expressions and voice data. Based on the results, generate warning messages for the driver and notifications to surrounding vehicles and bases."

[1502] Inquiry Data:

[1503] Video data: Grayscale image of the driver's face

[1504] Audio data: Driver's audio buffer

[1505] The above is a specific embodiment of this system.

[1506] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1507] Step 1: Collect data

[1508] The device uses a camera and microphone installed in the vehicle to collect video and audio data in real time. Specifically, the camera captures the driver's face and obtains high-resolution video data. The microphone captures the driver's voice and any sounds from inside the vehicle and stores them as audio data. This data is input into the device as soon as it is collected.

[1509] input:

[1510] Video data: Driver's face and interior of the vehicle

[1511] Voice data: Driver's voice and acoustic information inside the vehicle

[1512] output:

[1513] Raw video and audio data collected

[1514] Step 2: Preprocessing the data

[1515] The terminal converts the collected video data into grayscale format and audio data into buffer format. Specifically, color information is removed from the video data, and only brightness information is retained. Audio data is divided into segments of a certain length and stored in a buffer. This reduces the amount of data and makes subsequent analysis more efficient.

[1516] input:

[1517] Raw video data

[1518] Raw audio data

[1519] output:

[1520] Video data converted to grayscale

[1521] Audio data converted into buffer format

[1522] Step 3: Sending data

[1523] The device then sends the preprocessed data to the server. Specifically, the data is divided into packets and transmitted to the server using wireless communication technology (e.g., LTE or WiFi). The data is also compressed during transmission, improving communication efficiency.

[1524] input:

[1525] Video data converted to grayscale

[1526] Audio data converted into buffer format

[1527] output:

[1528] Pre-processed data sent to the server

[1529] Step 4: Analysis by AI models and sentiment engines

[1530] The server receives the preprocessed data and inputs it into an artificial intelligence model and emotion engine. Specifically, it uses deep learning algorithms to analyze the driver's facial features and voice patterns to detect signs of driver fatigue or dangerous driving. It then uses the emotion engine to analyze the driver's emotions.

[1531] input:

[1532] Grayscale video data

[1533] Buffered audio data

[1534] output:

[1535] Analysis of driver fatigue and emotional state

[1536] Step 5: Risk notification

[1537] Based on the analysis results, the server notifies the driver, surrounding vehicles, and the base station of any danger information. Specifically, it sends a real-time warning to the driver's device, such as "Fatigue detected. Please take a break." At the same time, it sends a warning message to surrounding vehicles via wireless communication and sends a detailed situation report to the base station.

[1538] input:

[1539] Analysis of driver fatigue and emotional state

[1540] output:

[1541] Driver warning message

[1542] Warning message to surrounding vehicles

[1543] Report the situation to the base

[1544] Step 6: Save your data

[1545] The server stores the analysis results and collected data, specifically by recording this information in a database for long-term data storage and subsequent analysis.

[1546] input:

[1547] Operational data and analysis results

[1548] output:

[1549] Stored database records

[1550] Step 7: Generate optimal driving patterns

[1551] The server statistically analyzes the stored data and generates optimal driving patterns. Specifically, it analyzes past driving data to determine driving behaviors suitable for a specific driver and generates patterns for reducing fatigue and driving safely.

[1552] input:

[1553] Saved driving data

[1554] output:

[1555] Optimal driving pattern

[1556] Step 8: Application to vehicle systems

[1557] The server applies the generated optimal driving pattern to the vehicle system, sending instructions to the vehicle's control system and navigating the vehicle in the optimal driving method for the driver in real time.

[1558] input:

[1559] Optimal driving pattern

[1560] output:

[1561] Driving assistance based on applied driving patterns

[1562] (Application example 2)

[1563] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1564] Many current vehicles lack systems that effectively detect driver fatigue and emotional changes in real time and support safe driving. As a result, there is a high risk of accidents occurring due to long driving times or emotional arousal. Furthermore, conventional systems lack a means to effectively communicate warnings to drivers and do not contribute sufficiently to accident prevention. Furthermore, systems that generate and apply optimal driving patterns for individual drivers are also rare. Effective methods to solve these problems are needed.

[1565] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving preprocessed data and using an artificial intelligence model to analyze the driver's state, means for notifying the driver, surrounding vehicles, and bases of danger information based on the analysis results, and means for using a wearable device to notify the driver of visual and audio warnings in real time. This makes it possible to support the driver in safe driving.

[1566] The "terminal mounted in a vehicle" is a device that is installed inside a vehicle and collects video data and audio data in real time.

[1567] "Video data" refers to visual information obtained from a visual sensor such as a camera.

[1568] "Audio data" refers to acoustic information obtained from an audio sensor such as a microphone.

[1569] "Preprocessing" refers to the process of converting collected data into a form suitable for subsequent analysis.

[1570] An "artificial intelligence model" is a model that includes deep learning algorithms used to analyze the driver's state.

[1571] A "server" is a central processing unit that receives, analyzes, and stores pre-processed data.

[1572] "Danger information" refers to information that warns or alerts drivers about their fatigue level or dangerous driving behavior.

[1573] "Storage" is the act of recording collected and analyzed data in a retrievable form.

[1574] An "optimal driving pattern" is a guideline for driving behavior that is generated to maximize driver safety and efficiency.

[1575] "Applying to vehicle systems" refers to reflecting the generated optimal driving patterns in actual driving operations and control systems.

[1576] A "wearable device" is a device worn by a driver that provides visual and audio notifications.

[1577] This invention provides a system that uses a terminal installed in a vehicle to collect video and audio data in real time, preprocess the data, and transmit it to a server. The server analyzes the driver's condition based on the preprocessed data using an artificial intelligence model and notifies the driver of hazardous situations. The server also stores the driving data, generates optimal driving patterns, and applies them to the vehicle system, supporting safe driving.

[1578] Data collection and preprocessing

[1579] The device uses a camera and microphone to collect in-car video and audio data in real time. The collected video data is converted to grayscale using software libraries such as OpenCV, and the audio data is converted to a buffer format using libraries such as SoundDevice. This preprocesses the collected data into a format optimal for subsequent analysis.

[1580] Analysis by AI models and sentiment engines

[1581] The server receives the preprocessed video and audio data and analyzes the driver's state using an AI model and emotion engine. Specifically, the AI ​​model uses deep learning technology to analyze the driver's facial features and voice patterns to detect signs of driver fatigue or dangerous driving. The emotion engine also analyzes the driver's facial expressions and voice tone to detect changes in emotion (anger, joy, sadness, etc.).

[1582] Risk information notification

[1583] The server notifies the driver of danger information in real time based on the analysis results. Specifically, it provides visual and audio warnings using a wearable device (e.g., smart glasses) worn by the driver. For example, if the driver is fatigued, a message saying "Fatigue detected. Please take a break" is displayed and an audio notification is given. At the same time, a warning message is sent via wireless communication to surrounding vehicles and bases such as logistics centers.

[1584] Data storage and generation of optimal driving patterns

[1585] The server stores all collected and analyzed driving data. This stored data is used for later analysis. Specifically, past driving data is statistically analyzed to generate an optimal driving pattern for each individual driver. This optimal driving pattern is then applied to the vehicle system to reduce driver fatigue and promote safe driving.

[1586] Specific examples

[1587] For example, if a driver has been driving continuously for a long period of time, the device will detect signs of fatigue from their facial expressions and voice. The collected data is sent to a server, where it is analyzed by an artificial intelligence model and emotion engine. If fatigue or agitation is confirmed, the server will issue a warning to the driver, saying, "Fatigue has been detected. Please take a break and stay calm." At the same time, a warning message is sent to surrounding vehicles, and a situation report is sent to the base station. Furthermore, the server can use the collected driving data to generate optimal driving patterns for the driver in the future and apply them to the vehicle system, further improving driver safety.

[1588] Prompt Sentence Examples

[1589] Create an AI model that can detect driver fatigue from video and audio data and provide real-time warnings during long driving sessions. Consider the following steps:

[1590] 1. Convert the video data to grayscale and perform preprocessing.

[1591] 2. Convert the audio data into a buffer format and perform preprocessing.

[1592] 3. The pre-processed data is sent to the server and analyzed by the AI ​​model and emotion engine.

[1593] 4. If fatigue or stress is detected, the driver will be notified with appropriate warning messages via audio and visual means.

[1594] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1595] Step 1:

[1596] The device uses a camera and microphone inside the vehicle to collect video and audio data in real time. Specifically, the camera captures the driver's face and the surrounding environment, while the microphone records the driver's voice and surrounding sounds. The input is video data from the camera and audio data from the microphone, which are then imported into the system in real time.

[1597] Step 2:

[1598] The video data collected by the device is converted to grayscale using OpenCV, and the audio data is converted to buffer format using SoundDevice. Specifically, color data is converted to grayscale, and audio signals are converted to digital data format. The input is color data and raw audio data, and the output is grayscale video data and buffer format audio data.

[1599] Step 3:

[1600] The preprocessed video and audio data are sent to the server. Specifically, the preprocessed data is packaged in JSON format and sent to the server via an HTTP request. The input is grayscale video data and buffered audio data, and the output is the data packet sent to the server.

[1601] Step 4:

[1602] The server inputs the received data into an artificial intelligence model and emotion engine to analyze the driver's condition. Specifically, it uses facial feature recognition algorithms and voice pattern analysis to detect fatigue and changes in emotion. The input is preprocessed video and audio data, and the output is the analysis result regarding the driver's condition.

[1603] Step 5:

[1604] The server generates a message to warn the driver in real time based on the analysis results and sends it to the wearable device. Specifically, if fatigue or dangerous driving is detected, an appropriate warning message is sent visually and audibly. The input is the analysis results, and the output is the warning message.

[1605] Step 6:

[1606] The server saves the analysis results and operational data. Specifically, it stores the results in a database and keeps them as records for later analysis. The input is the analysis results and operational data, and the output is the data saved in the database.

[1607] Step 7:

[1608] The server analyzes the stored driving data and generates optimal driving patterns. Specifically, it uses statistical methods and machine learning algorithms to optimize driving patterns and generate individual patterns for each driver. The input is the stored driving data, and the output is the optimal driving pattern.

[1609] Step 8:

[1610] The generated optimal driving pattern is applied to the vehicle system. Specifically, it is reflected in the vehicle's control system and appropriate feedback is provided to the driver. The input is the optimal driving pattern, and the output is the operation of the driving assistance system according to the applied driving pattern.

[1611] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1612] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1613] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1614] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1615] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1616] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1617] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1618] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1619] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1620] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1621] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1622] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1623] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1624] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1625] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1626] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1627] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1628] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1629] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1630] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1631] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1632] The following is further disclosed regarding the above embodiment.

[1633] (Claim 1)

[1634] A terminal mounted in the vehicle collects video data and audio data in real time;

[1635] means for pre-processing video data and audio data collected by the terminal;

[1636] a server that receives the pre-processed data and uses an artificial intelligence model to analyze the driver's state;

[1637] A means for notifying the driver, surrounding vehicles, and bases of danger information based on the analysis results;

[1638] a means for storing the analysis results and operational data;

[1639] A means for generating an optimal driving pattern based on the stored driving data;

[1640] means for applying the generated driving pattern to a vehicle system;

[1641] A system including:

[1642] (Claim 2)

[1643] 10. The system of claim 1, further comprising means for converting the video data collected by the terminal to grayscale.

[1644] (Claim 3)

[1645] 10. The system of claim 1, further comprising means for converting the collected audio data into a suitable buffer.

[1646] "Example 1"

[1647] (Claim 1)

[1648] A terminal mounted in the vehicle collects video data and audio data in real time;

[1649] means for pre-processing video data and audio data collected by the terminal;

[1650] a server that receives the preprocessed data and uses a generative AI model to analyze the driver's state;

[1651] A means for notifying the driver, surrounding vehicles, and bases of danger information based on the analysis results;

[1652] a means for storing the analysis results and operational data;

[1653] A means for generating an optimal driving pattern based on the stored driving data;

[1654] means for applying the generated driving pattern to a vehicle system;

[1655] A method for analyzing the frequency of eye opening and closing and facial expressions of the driver from video data,

[1656] A means for analyzing the tone and rhythm of speech from the speech data;

[1657] A system including:

[1658] (Claim 2)

[1659] 10. The system of claim 1, further comprising means for converting the video data collected by the terminal to grayscale.

[1660] (Claim 3)

[1661] 10. The system of claim 1, further comprising means for converting the collected audio data into a suitable buffer.

[1662] "Application Example 1"

[1663] (Claim 1)

[1664] A terminal mounted in the vehicle collects video data and audio data in real time;

[1665] means for pre-processing video data and audio data collected by the terminal;

[1666] a server that receives the pre-processed data and uses an artificial intelligence model to analyze the driver's state;

[1667] A means for notifying the driver, surrounding vehicles, and bases of danger information based on the analysis results;

[1668] a means for storing the analysis results and operational data;

[1669] A means for generating an optimal driving pattern based on the stored driving data;

[1670] means for applying the generated driving pattern to a vehicle system;

[1671] Smart glasses installed in the driver's seat collect and preprocess video and audio data,

[1672] means for transmitting to a computer mounted within the vehicle;

[1673] A deep learning model on the server analyzes the data and adjusts the autonomous driving mode or sends an alert to the driver as needed.

[1674] A system including:

[1675] (Claim 2)

[1676] 10. The system of claim 1, further comprising means for converting the video data collected by the terminal to grayscale.

[1677] (Claim 3)

[1678] 10. The system of claim 1, further comprising means for converting the collected audio data into a suitable buffer.

[1679] "Example 2: Combining Emotion Engines"

[1680] (Claim 1)

[1681] A terminal mounted in the vehicle collects video data and audio data in real time;

[1682] a pre-processing means for converting video data collected by the terminal into grayscale and audio data into a buffer format;

[1683] a server that receives the pre-processed data and uses an artificial intelligence model to analyze the driver's state;

[1684] means for analyzing the driver's emotions and fatigue state using an artificial intelligence model and an emotion engine;

[1685] A means for notifying the driver, surrounding vehicles, and base stations of danger information based on the analysis results;

[1686] a means for storing analysis results and operational data;

[1687] A means for generating an optimal driving pattern based on the stored driving data;

[1688] means for applying the generated driving pattern to a vehicle system;

[1689] A system including:

[1690] (Claim 2)

[1691] 10. The system of claim 1, further comprising means for converting the video data collected by the terminal to grayscale.

[1692] (Claim 3)

[1693] 10. The system of claim 1, further comprising means for converting the collected voice data into a buffer format.

[1694] "Application example 2 when combining emotion engines"

[1695] (Claim 1)

[1696] A terminal mounted in the vehicle collects video data and audio data in real time;

[1697] means for pre-processing video data and audio data collected by the terminal;

[1698] a server that receives the pre-processed data and uses an artificial intelligence model to analyze the driver's state;

[1699] A means for notifying the driver, surrounding vehicles, and bases of danger information based on the analysis results;

[1700] a means for storing the analysis results and operational data;

[1701] A means for generating an optimal driving pattern based on the stored driving data;

[1702] means for applying the generated driving pattern to a vehicle system;

[1703] a means for using the wearable device to notify the driver of real-time visual and audio warnings;

[1704] A system including:

[1705] (Claim 2)

[1706] 10. The system of claim 1, further comprising means for converting the video data collected by the terminal to grayscale.

[1707] (Claim 3)

[1708] 10. The system of claim 1, further comprising means for converting the collected audio data into a suitable buffer. [Explanation of symbols]

[1709] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A terminal mounted in the vehicle collects video data and audio data in real time; means for pre-processing video data and audio data collected by the terminal; a server that receives the pre-processed data and uses an artificial intelligence model to analyze the driver's state; A means for notifying the driver, surrounding vehicles, and bases of danger information based on the analysis results; a means for storing the analysis results and operational data; A means for generating an optimal driving pattern based on the stored driving data; means for applying the generated driving pattern to a vehicle system; A system including:

2. 10. The system of claim 1, further comprising means for converting the video data collected by the terminal to grayscale.

3. 2. The system of claim 1, further comprising means for converting the collected voice data into a suitable buffer.

Images