system

Abstract

We provide the system. [Solution] A means of capturing rear-view video of a vehicle using a rear camera, A method for detecting dangerous driving and aggressive driving by analyzing rear-view video captured by AI in real time, A means of sending a warning message to the driver in response to detected abnormal driving behavior and providing advice on countermeasures, A means of uploading and saving the detected abnormal operation data to the cloud, A voice support system for reporting to external organizations such as the police, based on the driver's instructions, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In recent years, accidents caused by aggressive driving and dangerous driving have been increasing, and efficient and prompt countermeasures against this are required. However, conventional driving recorders only record driving situations and lack real-time warning and automatic reporting functions. Therefore, there is a problem that the driver cannot respond immediately when facing a dangerous situation. In addition, there is no mechanism for quickly saving evidence of abnormal driving on the cloud and easily reporting it to the police or the like as needed. Solving these problems is an object of the present invention.

Means for Solving the Problems

[0005] To solve the above problems, the present invention provides the following means: It includes a means for capturing rearward video using a rear camera mounted on the vehicle, analyzing the video in real time using AI, and detecting dangerous driving and aggressive driving. It also includes a means for immediately sending a warning message to the driver and providing advice on countermeasures in response to abnormal driving detected by the AI. Furthermore, it includes a means for uploading and storing the data of the abnormal driving to the cloud. In addition, it includes a voice support means for notifying external organizations such as the police based on the driver's instructions. This makes it possible to provide an environment in which drivers can respond quickly and appropriately to dangerous situations, thereby significantly improving safety.

[0006] Definition of Terms

[0007] A "rear camera" is a camera device installed to capture images of the area behind a vehicle.

[0008] "Capture" refers to the process by which a camera device acquires and records video data.

[0009] "AI" refers to artificial intelligence, a technology that can perform specific tasks through machine learning and data analysis.

[0010] "Real-time" means that processing is performed instantly with little to no delay.

[0011] "Analysis" is the process of examining acquired data in detail to identify specific patterns or anomalies.

[0012] "Dangerous driving" refers to driving behavior that endangers other vehicles or pedestrians through speed or control that violates traffic laws.

[0013] "Aggressive driving" refers to driving behavior that intentionally causes fear or anxiety to other vehicles by approaching them too closely or tailing them.

[0014] A "warning message" is a text or voice message for notifying a driver when danger is detected.

[0015] A "countermeasure advice" is a proposal that includes specific action instructions for a driver to take against the detected dangerous driving.

[0016] "Cloud" refers to remote servers and data storage systems accessible through the Internet.

[0017] "Upload" is an operation or process of transferring data from a local device to a cloud server.

[0018] "Save" is a process of safely recording the acquired data for later or future use.

[0019] "Voice support" is a system that provides information to users through voice and supports operations.

[0020] "Report" is an act of notifying an external organization with authority such as the police about a specific abnormal situation or emergency.

Brief Description of Drawings

[0021] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6]It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [[ID=2,3]] [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Embodiments for Carrying Out the Invention

[0022] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0023] First, the terms used in the following description will be explained.

[0024] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), and APU (Accelerated Processing Unit).

[0025] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0026] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0027] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0028] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0029] [First Embodiment]

[0030] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0031] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0032] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0033] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0034] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0035] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0036] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0037] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0038] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0039] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0040] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0041] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0042] Modes for carrying out the invention

[0043] The system according to the present invention is an integrated system that includes a rear camera, AI-based video analysis, driver warnings, data storage in the cloud, and notification to the police, etc. The processing of the program of this system will be described below in natural language.

[0044] System program processing

[0045] Upon system startup, the terminal automatically activates the rear camera and continuously captures rearward video. This video data is encoded and sent to the server using a secure communication protocol. The server receives the transmitted video data and performs preprocessing for input into the AI ​​engine. This preprocessing includes frame extraction, resizing, and normalization.

[0046] The AI ​​engine analyzes pre-processed video data to detect dangerous driving and aggressive driving. The analysis extracts features such as abnormal following distances, sudden speed changes, and frequent lane changes, and uses a machine learning model to calculate an anomaly score. Based on these analysis results, the server generates a warning message if abnormal driving is detected.

[0047] The generated warning message is sent to the terminal in real time. The terminal displays the received warning message to the driver and also provides an audio notification. For example, the warning message might read, "There is a vehicle approaching dangerously close from behind. Change lanes to avoid it," and the same message might be announced aloud.

[0048] Furthermore, the data of detected abnormal operation is uploaded to and stored in the cloud. This includes video frames and metadata (date, time, location information, etc.) from the time the abnormality was detected. This makes it possible to review the details of the detected abnormal operation at a later date.

[0049] If the driver deems the situation serious, they are given the option to report it to the police through the terminal's interface. When the driver selects the reporting option, simply pressing the report button activates voice support, and pre-configured information (such as location and license plate number) is sent to the police via voice and text.

[0050] Specific example

[0051] For example, suppose a vehicle approaches abnormally closely from behind while driving on a highway. In this case, the rear camera captures the image behind the vehicle, and the AI ​​engine detects the abnormal proximity. The server notifies the terminal that an abnormal proximity has been detected, and the terminal displays the message, "There is a vehicle approaching abnormally closely from behind. Change lanes to avoid it," and also notifies the driver of the same message by voice. This situation is saved to the cloud and can be reviewed later. If the driver presses the report button, the police are automatically notified, and a notification is sent stating, "This vehicle is currently traveling at point XX, and is experiencing aggressive driving from behind. The vehicle number is XX-1234."

[0052] In this way, the system ensures driver safety in real time and provides an environment that allows for a rapid and appropriate response to abnormal situations.

[0053] The following describes the processing flow.

[0054] Program processing steps

[0055] Step 1:

[0056] Rear camera activation and video capture start

[0057] Terminal: When the vehicle engine starts, the rear camera automatically activates. After activation, the rear camera begins capturing rearward video and generates video data frame by frame.

[0058] Step 2:

[0059] Sending video data

[0060] Terminal: Encodes captured video frames at high speed and sends them to the server using a secure communication protocol.

[0061] Step 3:

[0062] Receiving and preprocessing video data

[0063] Server: Receives video data, stores it in a buffer, and performs preprocessing to prepare it for input into the AI ​​engine. This preprocessing includes frame extraction, resizing, and normalization.

[0064] Step 4:

[0065] AI analysis

[0066] Server: Pre-processed video data is input into the AI ​​engine to analyze patterns of dangerous driving and aggressive driving. Specifically, features such as abnormal following distances, sudden speed changes, and frequent lane changes are extracted, and an abnormality score is calculated using a machine learning model.

[0067] Step 5:

[0068] Generating warning messages

[0069] Server: When the AI ​​engine detects abnormal driving, it generates a warning message. This message includes the type of dangerous driving detected and the actions the driver should take.

[0070] Step 6:

[0071] Sending a warning

[0072] Server: Sends generated warning messages to terminals in real time.

[0073] Step 7:

[0074] Warning display and audio notification

[0075] Terminal: Displays received warning messages to the driver and also provides voice notifications. For example, it displays a message such as, "There is a vehicle approaching dangerously close from behind. Change lanes to avoid it," and also provides voice notifications.

[0076] Step 8:

[0077] Cloud storage of data

[0078] Server: Uploads and stores data on detected abnormal operation to the cloud. This data includes video frames, date and time, location information, etc.

[0079] Step 9:

[0080] Presentation of reporting options

[0081] Terminal: Presents the driver with the option of whether or not to report to the police. A message such as "Do you want to report this?" is displayed on the screen.

[0082] Step 10:

[0083] Execution of the report

[0084] User: The driver selects the reporting option.

[0085] Device: When the reporting button is pressed, voice support is activated, and detailed information, including location and vehicle information, is sent to the police via voice and text.

[0086] This allows drivers to receive real-time warnings and quickly contact the police as needed, significantly improving safety.

[0087] (Example 1)

[0088] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0089] In recent years, accidents caused by dangerous driving and intimidating driving have been increasing, and there is a need to detect these driving behaviors in real time and respond quickly. However, current dashcams and driver assistance systems are insufficient in their ability to adequately detect dangerous driving behavior behind the vehicle and warn the driver. Therefore, a system is needed to ensure driver safety and respond immediately to abnormal driving.

[0090] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0091] In this invention, the server includes means for capturing rear-view video of the transport equipment using a rear camera, means for analyzing the captured rear-view video in real time using artificial intelligence to detect dangerous driving or intimidating driving, means for sending a warning message to the driver and providing advice on countermeasures for detected abnormal driving, auxiliary means for saving driver safety data to a storage device based on the analysis results, means for saving video data to an external storage device using a secure communication protocol, and voice support means for notifying an external organization at the driver's instruction. This enables the detection of dangerous driving behavior behind the vehicle in real time, prompting the driver to be warned and encouraging appropriate action, and furthermore, data on abnormal driving can be saved to the cloud and reported to an external organization as needed.

[0092] A "rear camera" is a video capture device used to capture images from the rear of a vehicle.

[0093] "Artificial intelligence" is a general term that includes machine learning algorithms and deep learning technologies used to analyze captured rear-view video footage and detect abnormal driving behavior.

[0094] "Dangerous driving" refers to driving behavior that is unstable or abnormal compared to normal driving conditions.

[0095] "Intimidating driving" refers to driving behavior that intentionally causes fear or anxiety to other drivers or pedestrians.

[0096] A "warning message" is notification information intended to prompt the driver to take action in response to detected abnormal driving behavior.

[0097] "Driver" refers to a person operating a vehicle.

[0098] A "storage device" refers to hardware or storage media used to store data.

[0099] "External storage devices" refer to external data storage media, such as cloud storage, used to save data outside of the local system.

[0100] "Auxiliary means" refers to additional functions or technologies used to operate the main function.

[0101] A "secure communication protocol" refers to technologies and standards for ensuring secure data communication. Specifically, this includes protocols such as HTTPS.

[0102] "Reporting" refers to the act of transmitting information about abnormal driving to an external organization when such behavior is detected.

[0103] "Voice support" refers to a function that uses voice to provide information to users or to report information to external organizations.

[0104] "External organizations" refer to organizations including the police and other public institutions.

[0105] Modes for carrying out the invention

[0106] The system according to the present invention is an integrated system that includes a rear camera, artificial intelligence-based video analysis, driver warnings, data storage in the cloud, and notification to an external organization. Specific embodiments of this system are described below.

[0107] Rear camera activation and video capture

[0108] When the system starts up, the terminal automatically activates the rear camera, which captures video footage from behind the transport equipment. This video data is encoded in real time and converted to a stream format (e.g., H.264).

[0109] Sending video data

[0110] The encoded video data is sent from the terminal to the server using a secure communication protocol (e.g., HTTPS). The terminal buffers the video data, divides it into appropriate packet sizes, encrypts it, and sends it to the server.

[0111] Preprocessing of video data

[0112] The server preprocesses the received video data. This preprocessing includes frame extraction, resizing, and normalization. The preprocessed data is then input into the artificial intelligence engine.

[0113] Analysis of video data

[0114] The server uses an artificial intelligence engine to analyze pre-processed video data. The analysis extracts features such as abnormal following distances, sudden speed changes, and frequent lane changes, and calculates an anomaly score using machine learning models (e.g., TensorFlow, PyTorch). This allows for the detection of dangerous driving and aggressive driving behavior.

[0115] Generating and sending warning messages

[0116] Based on the analysis results, the server generates a warning message if abnormal operation is detected. The generated warning message is sent to the terminal in real time.

[0117] Driver warning notification

[0118] The device notifies the driver of any received warning messages. The notification is displayed on the screen and also alerts the driver with an audio alert. For example, a message such as "There is a vehicle approaching dangerously close from behind. Change lanes to avoid it" will be displayed, and the same information will be announced by voice.

[0119] Cloud storage of abnormal driving data

[0120] The server uploads and stores the detected abnormal operation data to the cloud. This data includes video frames and metadata (date and time, location information, etc.) at the time of the anomaly detection. The data is stored using cloud storage services (e.g., AWS S3, Google Cloud Storage).

[0121] Driver's report to the police

[0122] If the driver deems the situation serious, they will be given the option to call the police through the terminal's interface. If the driver selects the calling option, simply pressing the call button will activate voice support, and pre-configured information (location, license plate number, etc.) will be sent to the police in both voice and text format.

[0123] Specific example

[0124] For example, if a vehicle approaches abnormally closely from behind while driving on a highway, the rear camera captures the image behind the vehicle, and the artificial intelligence engine detects the abnormal proximity. The server notifies the terminal that an abnormal proximity has been detected, and the terminal displays the message, "There is a vehicle approaching abnormally closely from behind. Change lanes to avoid it," and also notifies the driver of the same message by voice. This situation is saved to the cloud and can be reviewed later. If the driver presses the report button, the police are automatically notified, and a notification is sent stating, "This vehicle is currently traveling at point XX, and is experiencing aggressive driving from behind. The vehicle number is XX-1234."

[0125] Examples of prompts for generative AI models

[0126] For example, analysis using a generated AI model can be performed by using the following prompt statement.

[0127] You are developing an advanced driver assistance system. This system integrates a rear camera, AI-powered video analysis, driver warnings, cloud data storage, and notification to external organizations. Explain the program's processes and provide specific examples for the following scenarios.

[0128] scenario:

[0129] 1. Please explain how the device activates the rear camera and captures rearward video when the system starts up.

[0130] 2. Explain how the artificial intelligence engine analyzes video data to detect dangerous or aggressive driving.

[0131] 3. Please explain in detail the mechanism for displaying and notifying drivers of warnings.

[0132] 4. Please explain the data storage and reporting process for abnormal operation to external organizations.

[0133] Specific example:

[0134] Describe a scenario in which a vehicle approaches unusually closely from behind while driving on a highway, and specifically explain how each process functions in that situation.

[0135] By inputting this prompt into the AI ​​model, it is possible to generate a detailed explanation of the system's operation and various functions.

[0136] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0137] System program processing flow

[0138] Step 1:

[0139] The device automatically activates the rear camera when the system starts up.

[0140] Specific operation: The terminal powers on the rear camera via internal control software and instructs the camera to begin capturing rearward video. The rear camera continuously captures rearward video and generates encoded stream data (e.g., in H.264 format).

[0141] Input: System power signal.

[0142] Output: Rear camera capture started, encoded video data.

[0143] Step 2:

[0144] The terminal sends the encoded video data to the server using a secure communication protocol (e.g., HTTPS).

[0145] Specific operation: The terminal buffers the video data, divides it into packets, and encrypts them. It then sends them to the server via the network stack.

[0146] Input: Encoded video data from the rear camera.

[0147] Output: Encrypted video data sent to the server.

[0148] Step 3:

[0149] The server preprocesses the received video data.

[0150] Specific operation: The server receives data, decodes the packets, and divides them into frames. Each frame is resized and normalized. After that, the pre-processed data is input into the AI ​​engine.

[0151] Input: Encrypted video data packets.

[0152] Output: Preprocessed video data (resized and normalized frames).

[0153] Step 4:

[0154] The server analyzes the pre-processed video data.

[0155] Specific operation: The AI ​​engine extracts features to detect abnormalities such as insufficient following distance, sudden speed changes, and frequent lane changes. A machine learning model (e.g., TensorFlow, PyTorch) is used to calculate an anomaly score and determine whether the driving is dangerous or aggressive.

[0156] Input: Pre-processed video data.

[0157] Output: Abnormal operation score and judgment result.

[0158] Step 5:

[0159] Based on the analysis results, the server generates a warning message if abnormal operation is detected.

[0160] Specific operation: When abnormal operation is detected, the server generates a warning in text format and creates a warning message for the driver.

[0161] Input: Score and judgment result for abnormal operation.

[0162] Output: Warning message.

[0163] Step 6:

[0164] The server sends the generated warning message to the terminal.

[0165] Specific operation: The server sends a warning message to the terminal in real time.

[0166] Input: Warning message.

[0167] Output: Send to the terminal.

[0168] Step 7:

[0169] The terminal notifies the driver of the received warning message.

[0170] Specific operation: The device displays a warning message on its screen and notifies the driver of the same content via an audio alert. For example, it might inform the driver, "There is a vehicle approaching dangerously close from behind. Change lanes to avoid it."

[0171] Input: Warning message.

[0172] Output: Display, audio alert.

[0173] Step 8:

[0174] The server uploads and stores the detected abnormal operation data to the cloud.

[0175] Specific operation: The server collects video frames and metadata (date, time, location information, etc.) from the time of anomaly detection into storage data and uploads it to a cloud storage service (e.g., AWS S3, Google Cloud Storage).

[0176] Input: Anomaly detection data (video frames, metadata).

[0177] Output: Data stored in cloud storage.

[0178] Step 9:

[0179] If the driver deems the situation serious, they will notify the police via the device.

[0180] Specific operation: When the driver presses the call button, the device activates its voice assistance function and sends pre-configured information (location information, license plate number, etc.) to the police in both voice and text format.

[0181] Input: Driver presses the call button, saved information.

[0182] Output: Automatic notification to the police (voice and text format).

[0183] (Application Example 1)

[0184] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0185] In recent years, aggressive driving and dangerous driving have become a social problem, making prevention and response to these issues urgent. Furthermore, in order to ensure the safety of individuals, including pedestrians, there is a need for a system that can detect abnormal behavior in real time and provide appropriate warnings and countermeasures. The present invention aims to provide a system that ensures the safety of a wider range of individuals by utilizing not only automobiles but also smart glasses.

[0186] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0187] In this invention, the server includes means for capturing rear-view video of a vehicle using a rear camera, means for analyzing the captured rear-view video in real time using AI to detect dangerous driving and aggressive driving, means for sending a warning message to the driver and providing advice on countermeasures for detected abnormal driving, means for uploading and storing the detected abnormal driving data to the cloud, means for providing voice support for notifying external organizations such as the police at the driver's instruction, means for capturing rear-view video using a rear camera mounted on smart glasses to detect approaching abnormal behavior, and means for providing the user with a visual and audible warning message when approaching abnormal behavior is detected. This enables universal safety assurance not only for vehicle drivers but also for pedestrians.

[0188] A "rear camera" is a camera device installed in cars or smart glasses that captures images of the area behind the vehicle.

[0189] "AI" is an abbreviation for artificial intelligence, a technology that uses machine learning and deep learning to analyze data and detect patterns and anomalies.

[0190] "Dangerous driving" refers to driving behaviors that endanger other road users, such as sudden changes in speed or frequent lane changes.

[0191] "Aggressive driving" refers to driving behavior that intentionally causes fear to other drivers, such as overtaking other vehicles and slamming on the brakes, or driving dangerously close to them.

[0192] A "warning message" is a message that provides a warning or evacuation order to the driver or user when dangerous driving or aggressive driving is detected.

[0193] "Cloud" refers to external data storage and computing resources provided via the internet.

[0194] "Voice support means" refers to functions that use voice to provide information to users or to report to external organizations.

[0195] "Smart glasses" are glasses-type wearable devices that have computer functions and assist the user's visual information.

[0196] "Abnormal behavior" refers to actions that deviate from normal behavioral patterns and may pose a danger to the user or others.

[0197] "Means of providing warning messages visually and audibly" refers to functions that use displays and speakers to convey warning information to users visually and audibly.

[0198] The system according to the present invention is an integrated system for ensuring the safety of automobile drivers and pedestrians. This system includes a rear camera, AI-based video analysis, warning messages to the user, data storage to the cloud, and notification to the police, etc. Embodiments of this system will be described below.

[0199] System hardware and software configuration

[0200] Hardware:

[0201] Smart glasses (e.g., typical wearable devices)

[0202] Rear camera (mounted in cars and smart glasses)

[0203] Speaker (for voice notifications)

[0204] GPS module (for acquiring location information)

[0205] software:

[0206] Video capture API (for acquiring video data)

[0207] AI models (for video analysis, e.g., TensorFlow, PyTorch)

[0208] Cloud storage (for data storage, e.g., AWS S3)

[0209] Secure communication protocols (e.g., HTTPS)

[0210] Reporting API (e.g. Twilio, Firebase)

[0211] System Operation Overview

[0212] 1. Startup and video capture

[0213] When smart glasses or a car are activated, the rear camera automatically activates and captures footage of the area behind the vehicle. The video data is encoded in real time.

[0214] 2. Video data preprocessing

[0215] The server receives the video data and performs preprocessing. This includes frame extraction, resizing, and normalization.

[0216] 3. AI analysis

[0217] Pre-processed video data is analyzed by an AI engine. The AI ​​engine detects abnormal behavior (such as dangerous driving, road rage, or suspicious individuals approaching).

[0218] 4. Generate and notify warning messages

[0219] When abnormal behavior is detected, the server generates a warning message and notifies the user through smart glasses or the car's display and speakers. For example, the message might say, "There is a vehicle approaching you dangerously closely from behind. Change lanes to avoid it."

[0220] 5. Data storage and reporting

[0221] Data on detected abnormal behavior is stored in the cloud. This includes location information, date and time, and video frames. When a user presses the report button, voice support is activated, and location information and video data are automatically sent to the police.

[0222] Specific example

[0223] For example, if a pedestrian wearing smart glasses is walking along a school route and a suspicious person approaches from behind, the rear camera captures the situation. The AI ​​engine detects the abnormal approach and displays a message saying, "Someone is approaching you from behind. Please move to a safe place," and also provides an audio notification with the same message. If the user presses the emergency call button, a report is sent to the police stating, "I am currently walking at location XX, and a suspicious person is approaching from behind."

[0224] Example of a prompt

[0225] Please create a smart glasses application that works as follows: Use the smart glasses' rear camera to capture rearward video, and have AI detect abnormal behavior (following, sudden approach). Implement a function to provide visual and audible warnings upon detection, and automatically notify the police in emergencies.

[0226] In summary, this system provides a technical solution to ensure the safety of vehicle drivers and pedestrians in real time and to respond quickly and appropriately.

[0227] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0228] Step 1:

[0229] Smart glasses or a car terminal are activated. The terminal automatically activates the rear camera and captures rearward video in real time. The captured video data is encoded and sent to the server using a secure communication protocol (e.g., HTTPS). The input is video data from the rear camera, and the output is encoded video data.

[0230] Step 2:

[0231] The server preprocesses the received video data. Specifically, it extracts frames, resizes them, and normalizes them. This preprocessing prepares the video for efficient analysis by the AI ​​engine. The input is encoded video data, and the output is preprocessed video data.

[0232] Step 3:

[0233] The server's AI engine analyzes pre-processed video data. The AI ​​engine uses a machine learning model to detect abnormal behavior (such as dangerous driving, road rage, or suspicious individuals approaching). Specifically, it extracts features such as distance between vehicles, speed changes, and lane change frequency, and calculates an abnormality score. The input is pre-processed video data, and the output is the result of detecting abnormal behavior.

[0234] Step 4:

[0235] The server generates a warning message if abnormal behavior is detected. The warning message is provided visually and audibly. The generated warning message is sent to the terminal using a secure communication protocol. The input is the result of the detected abnormal behavior, and the output is the warning message.

[0236] Step 5:

[0237] The device notifies the user of received warning messages. Specifically, it conveys warning messages visually and audibly using the smart glasses' display and speaker. For example, it might say, "There is a vehicle approaching you dangerously close from behind. Change lanes to avoid it." The input is the warning message, and the output is the notification to the user.

[0238] Step 6:

[0239] The server saves data on detected abnormal behavior to the cloud. The saved data includes date and time, location information, and video frames. Cloud storage (e.g., AWS S3) is used for data storage. The input is the detected abnormal behavior and associated metadata, and the output is the data stored in the cloud.

[0240] Step 7:

[0241] When a user presses the emergency call button, the device's voice support function is activated. Pre-configured location information and video data are transmitted to the police via voice and text. The input consists of the user's call instructions and related data (location information, video data), and the output is a notification to the police.

[0242] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0243] Modes for carrying out the invention

[0244] The system according to the present invention is an integrated system that combines a rear camera, AI-based video analysis, driver warnings, data storage to the cloud, and notification to the police, with an emotion engine that recognizes the user's emotions. The processing of this system's program will be described below in natural language.

[0245] System program processing

[0246] When the system starts up, the terminal activates the rear camera and continuously captures video footage of the area behind the vehicle. This video data is encoded and sent to the server, which receives the video data and performs preprocessing for analysis by the AI ​​engine. The AI ​​engine analyzes the preprocessed data to detect dangerous driving and aggressive driving.

[0247] Based on the analysis results, the server generates a warning message if abnormal operation is detected. The message is sent to the terminal in real time, and the terminal notifies the driver of the received message. Furthermore, data on abnormal operation is stored in the cloud and can be reviewed later.

[0248] The present invention also includes a system for recognizing the driver's emotions. Specifically, a camera mounted on the terminal captures an image of the driver's face and transmits this data to a server. The server uses an emotion engine to analyze the image of the driver's face and recognize their emotional state. For example, it can detect states such as tension, stress, and relaxation.

[0249] Based on recognized emotions, the content of warning messages and advice is adjusted. For example, if a driver is stressed, more detailed and considerate advice may be provided. Furthermore, emotional states are included as additional information in data stored in the cloud, allowing for later analysis.

[0250] Furthermore, it includes a feature that automatically prompts a police alert if the driver is experiencing particular tension or stress. This feature allows for quick external assistance to be sought if the driver's emotional state is deemed to be a factor detrimental to safe driving.

[0251] Specific example

[0252] For example, suppose you are driving on a highway and a vehicle approaches you dangerously closely from behind. In this case, the rear camera captures the image behind you, and the server's AI engine detects the dangerous proximity. The server generates a warning message such as, "There is a vehicle approaching you dangerously closely from behind. Change lanes to avoid it," and sends it to your device.

[0253] Simultaneously, a camera is operating to capture video of the driver's face, and this video data is sent to a server. The server's emotion engine analyzes this data to recognize that the driver is stressed. Based on this, it checks whether the warning message is considerate of the driver's emotional state and adjusts it as needed.

[0254] Furthermore, the details of this abnormal driving and the driver's emotional state are saved to the cloud. Also, if the driver presses the report button, the system sends information to the police stating, "This vehicle is currently traveling at location XX and is experiencing aggressive driving from behind. The vehicle's license plate number is XX-1234. The driver is extremely distressed."

[0255] In this way, this system not only detects dangerous driving but also provides comprehensive safety measures that take into account the driver's emotional state.

[0256] The following describes the processing flow.

[0257] Modes for carrying out the invention

[0258] System program processing

[0259] Step 1:

[0260] Rear camera activation and video capture start

[0261] Terminal: When the vehicle engine starts, the rear camera automatically activates. After activation, the rear camera begins capturing rearward video and generates video data frame by frame.

[0262] Step 2:

[0263] Sending video data

[0264] Terminal: Encodes captured video frames at high speed and sends them to the server using a secure communication protocol.

[0265] Step 3:

[0266] Receiving and preprocessing video data

[0267] Server: Receives video data, stores it in a buffer, and performs preprocessing to prepare it for input into the AI ​​engine. This preprocessing includes frame extraction, resizing, and normalization.

[0268] Step 4:

[0269] AI analysis

[0270] Server: Pre-processed video data is input into the AI ​​engine to analyze patterns of dangerous driving and aggressive driving. Specifically, features such as abnormal following distances, sudden speed changes, and frequent lane changes are extracted, and an abnormality score is calculated using a machine learning model.

[0271] Step 5:

[0272] Generating warning messages

[0273] Server: When the AI ​​engine detects abnormal driving, it generates a warning message. This message includes the type of dangerous driving detected and the actions the driver should take. For example, it might say, "There is a vehicle approaching dangerously closely from behind. Change lanes to avoid it."

[0274] Step 6:

[0275] Sending a warning

[0276] Server: Transmits the generated warning message to the terminal in real time.

[0277] Step 7:

[0278] Display of warning and voice notification

[0279] Terminal: Displays the received warning message to the driver and also gives a voice notification. For example, a message is displayed on the display, and it is notified by voice as "There is a vehicle approaching abnormally from behind. Please change lanes to avoid it." <000088�>

[0280] Step 8:

[0281] Cloud storage of data

[0282] Server: Uploads and stores the detected abnormal driving data to the cloud. This data includes video frames, date and time, location information, etc.

[0283] Step 9: <000​​​​​​​​​​​​​​​​​​​​​​​​ Adjusting emotion-based warning messages

[0291] Server: Based on recognized emotions, it adjusts the content of warning messages and corrective advice. For example, if the driver is stressed, it provides more polite and detailed advice.

[0292] Step 12:

[0293] Cloud storage of emotional data

[0294] Server: The recognized emotional states are added to data stored in the cloud, making them available for later analysis.

[0295] Step 13:

[0296] Presentation of reporting options

[0297] Terminal: Displays an option on the screen for the driver to report the incident to the police. For example, it might display the option, "Do you want to report this?"

[0298] Step 14:

[0299] Execution of the report

[0300] User: The driver selects the reporting option.

[0301] Terminal: When the reporting button is pressed, voice support is activated, and detailed information, including location, vehicle information, and emotional state, is sent to the police via voice and text. For example, information such as, "This vehicle is currently traveling at location XX and is experiencing aggressive driving from behind. The vehicle number is XX-1234. The driver is extremely agitated," will be sent.

[0302] As described above, this system provides comprehensive safety measures that take into account not only the detection of dangerous driving and aggressive driving, but also the driver's emotional state.

[0303] (Example 2)

[0304] Next, Example 2 will be described. In the following description, the data processing device 12 is referred to as a "server", and the smart device 14 is referred to as a "terminal".

[0305] Conventional automotive safety systems have a function to detect dangerous driving and tailgating driving, but they cannot consider the driver's emotional state, so there is a problem that the driver's stress and tension increase. In addition, although data on abnormal driving is saved, since specific countermeasure advice is not sufficient, there is a limit to improving safety.

[0306] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for capturing a rear view image of the vehicle by a rear camera, means for analyzing the captured rear view image in real time by AI to detect dangerous driving and tailgating driving, means for transmitting a warning message to the driver and providing countermeasure advice for the detected abnormal driving, means for uploading and saving the data of the detected abnormal driving to the cloud, means for capturing an image of the driver's face and performing emotion analysis, and means for adjusting the warning message based on the recognized emotion of the driver. Thereby, not only can dangerous driving and tailgating driving be detected in real time, but also comprehensive safety measures considering the driver's emotional state become possible. [[ID=十六]]

[0307] [[ID=十七]] The "rear camera" is a camera attached to the rear part of the vehicle and is a device for capturing a rear view image. [[ID=二十]]

[0308] [[ID=二十一]] "AI" means artificial intelligence and refers to an algorithm or system that automatically executes a specific task.

[0309] "Dangerous driving" means a driving act that violates the Road Traffic Law and threatens safety.

[0310] "Aggressive driving" refers to intentionally engaging in intimidating driving behavior towards other vehicles.

[0311] A "warning message" is a message that the system issues to the driver to warn them about dangerous driving or aggressive driving.

[0312] "Safety advice" refers to information that provides drivers with specific instructions and advice to help them drive safely.

[0313] "Cloud" refers to a group of servers that provide data storage and processing functions via the internet.

[0314] "Emotional analysis" is a technology that evaluates a driver's emotional state based on facial expressions and other factors using video data of their face.

[0315] "Reporting methods" refer to a function that, in response to the driver's instructions, allows for reporting abnormal driving situations to external organizations such as the police using voice or text.

[0316] "Preprocessing means" refers to techniques that process video data, such as noise reduction and frame selection, before analysis by AI.

[0317] Modes for carrying out the invention

[0318] The system according to the present invention is an integrated safety driving support system that includes a rear camera, an AI engine, an emotion analysis engine, a driver warning message function, data storage to the cloud, and an automatic notification function to the police.

[0319] Hardware and software configuration:

[0320] The rear camera is mounted on the back of the car and continuously captures images of the area behind it.

[0321] The device has the function of encoding video data from the rear camera and sending it to the server.

[0322] The server has hardware and software to preprocess the received video data and analyze it with an AI engine.

[0323] The AI ​​engine includes algorithms that detect dangerous driving and aggressive driving in real time.

[0324] Another camera on the device captures video of the driver's face, providing data to recognize the driver's emotional state.

[0325] The emotion analysis engine is built into the server and analyzes the transmitted facial video data to recognize the driver's emotional state (tension, stress, relaxation, etc.).

[0326] The cloud server securely stores data such as abnormal driving behavior and the driver's emotional state, allowing for later review and analysis.

[0327] The warning message function sends messages generated on the server to the terminal in real time, and the terminal notifies the driver.

[0328] The automatic reporting function has a means of reporting to an external organization (e.g., the police) based on the driver's instructions.

[0329] Specific examples of how the system works:

[0330] Suppose a vehicle is detected approaching abnormally closely from behind while driving on a highway. The rear camera captures the image behind the vehicle, the video data is encoded on the terminal, and then sent to the server.

[0331] The server uses an AI engine to detect abnormal proximity. It generates a warning message such as, "There is a vehicle approaching abnormally closely from behind. Change lanes to avoid it," and sends it to the terminal.

[0332] Simultaneously, a camera capturing the driver's face is also operating, and the video data is sent to a server. An emotion analysis engine analyzes this data and recognizes that the driver is feeling stressed.

[0333] Based on this understanding, the server adjusts the content of the warning message. For example, it might add advice such as, "Stay calm and check behind you."

[0334] All data (including details of abnormal driving and the driver's emotional state) is stored on a cloud server.

[0335] When the driver presses the call button, the automatic call function sends a message to the police saying, "This vehicle is currently traveling at location XX, and is experiencing aggressive driving from behind. The vehicle's license plate number is XX-1234. The driver is extremely distressed."

[0336] Example of a prompt:

[0337] 1. "Please describe in detail the process from detecting abnormal operation to sending a warning message in conjunction with the rear camera."

[0338] 2. "Please explain specific methods for adjusting warning messages using driver facial recognition and emotion analysis."

[0339] Thus, the system according to the present invention not only detects dangerous driving and aggressive driving in real time, but also provides comprehensive safety measures that take into account the driver's emotional state.

[0340] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0341] Step 1: System startup

[0342] The terminal prepares to activate the rear camera simultaneously with the system startup. The input is the system startup signal, and the output is the completion of the rear camera startup.

[0343] Specifically, when the device's power button is pressed, the internal software sends a command to activate the rear camera.

[0344] Step 2: Video capture using the rear camera

[0345] The device activates the rear camera and continuously captures rearward video. The input is real-time video data, and the output is encoded rearward video data.

[0346] Specifically, the rear camera captures the rear view frame by frame and sends it to the terminal. The terminal then encodes this data.

[0347] Step 3: Encode and transmit video data

[0348] The terminal encodes the captured video data. The input is raw video data, and the output is encoded data. After encoding, this data is sent to the server.

[0349] Specifically, the terminal compresses the video data into a format such as MPEG and sends it to the server using a communication protocol.

[0350] Step 4: Video analysis using an AI engine

[0351] The server preprocesses the received encoded video data. The input is encoded video data, and the output is preprocessed data. Preprocessing such as noise reduction and frame selection is performed.

[0352] The server's AI engine analyzes pre-processed data to detect dangerous driving and aggressive driving. The input for the analysis is pre-processed data, and the output is the detection result of abnormal driving.

[0353] Step 5: Generate and send the warning message.

[0354] If abnormal operation is detected, the server generates a warning message. The input is the result of the abnormal operation detection, and the output is the generated warning message. Next, this message is sent to the terminal in real time, and the terminal notifies the driver.

[0355] Specifically, the server creates a warning message based on a template and sends it to the terminal using the TCP / IP protocol. The terminal then displays an audio alert or a pop-up on the screen.

[0356] Step 6: Driver's Emotion Recognition

[0357] The camera on the device captures video of the driver's face. The input is video data of the driver's face, and the output is encoded video data of the face. This is sent to the server.

[0358] The server's emotion engine analyzes facial video data to recognize the driver's emotional state. The input is encoded facial video data, and the output is the result of the emotion analysis.

[0359] Step 7: Adjusting Emotion-Based Warning Messages

[0360] The server adjusts warning messages to be more detailed and thoughtful if the driver is stressed. The input is the result of sentiment analysis, and the output is the adjusted warning message.

[0361] Specifically, the server selects a warning message template and adjusts the content of the advice based on the sentiment analysis results.

[0362] Step 8: Save data to the cloud

[0363] The server stores data on abnormal operation and sentiment analysis on a cloud server. The input is the status of abnormal operation and the results of sentiment analysis, and the output is the data stored in the cloud.

[0364] Specifically, the server uploads data to the cloud storage service via the database.

[0365] Step 9: Automatically report to the police

[0366] When the driver presses the emergency call button, the server automatically initiates the process of notifying the police. The input is the driver's emergency call signal, and the output is the emergency call message sent to the police.

[0367] Specifically, the server generates a notification message containing location and vehicle information, and sends it to the police using the appropriate communication protocol.

[0368] (Application Example 2)

[0369] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0370] Conventional in-vehicle systems were specialized in detecting dangerous driving behavior and abnormal approaches from the rear, but lacked consideration for the driver's emotional state. This made it difficult to respond appropriately when the driver felt tension or stress, potentially hindering safe driving. Furthermore, warnings and advice after detecting abnormal driving were uniform, failing to provide measures tailored to the driver's emotional state. As a result, safety and driver stress reduction were not adequately achieved.

[0371] In Application Example 2, the identification processing by the identification processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for capturing rear-view video of the vehicle using a rear camera, means for analyzing the captured rear-view video in real time using AI to detect dangerous driving and aggressive driving, means for sending a warning message to the driver and providing advice on countermeasures for detected abnormal driving, means for uploading and saving the detected abnormal driving data to the cloud, voice support means for notifying external organizations such as the police at the driver's instruction, means for capturing video of the driver's face and analyzing the driver's emotional state using an emotion recognition engine, and means for adjusting the content of warning messages and countermeasure advice based on the driver's emotional state. This enables appropriate countermeasures according to the driver's emotional state, thereby improving safety and reducing stress.

[0372] A "rear camera" is a camera device used to capture images of the area behind a vehicle.

[0373] The "AI engine" is an artificial intelligence technology that analyzes captured video in real time to detect abnormal driving, aggressive driving, and the driver's emotional state.

[0374] "Dangerous driving" refers to any driving behavior that may endanger other vehicles or pedestrians.

[0375] "Aggressive driving" refers to driving behavior that is harassing or endangers the driver, such as a vehicle following closely behind another vehicle.

[0376] A "warning message" is a message or alert designed to alert the driver to detected abnormal driving behavior.

[0377] "Countermeasure advice" refers to information that presents recommended driving actions and points to note to the driver in response to detected abnormal driving behavior.

[0378] "Cloud" refers to data storage and processing services provided via the internet, and in this context, it refers to remote storage for saving data on detected abnormal operation.

[0379] "Voice support means" refers to a function that provides voice support when the driver instructs an external organization, such as the police, to make a report.

[0380] The "emotion recognition engine" is a technology that analyzes the driver's facial image to recognize their emotional state.

[0381] "Emotional state" refers to the driver's mental and psychological state, including states such as tension, stress, and relaxation.

[0382] "Video data preprocessing" is a technique that transforms video data into a format that is easily accessible for AI analysis.

[0383] "External organizations" refer to public institutions such as the police and other support organizations.

[0384] "Abnormal driving data" refers to all data that, as a result of analyzing video footage acquired from rear cameras and other sources, is determined to indicate abnormal driving.

[0385]

[0386] The system according to the present invention is an integrated system installed in an autonomous vehicle. This system captures rearward images using a rear camera and analyzes the data with an AI engine to detect dangerous driving and aggressive driving. In addition, a camera that captures facial images and an emotion recognition engine are used to recognize the driver's emotional state.

[0387] Specific system configuration

[0388] 1. Hardware configuration:

[0389] Rear camera: A camera used to capture images of the rear of a vehicle.

[0390] Emotion Recognition Camera: A camera that captures video of the driver's face.

[0391] In-car display: A display that shows warning messages and advice to the driver. This can also be replaced by the display on a smartphone or smart glasses.

[0392] 2. Software configuration:

[0393] AI Engine: Uses platforms such as TensorFlow to analyze video from the rear camera and emotion recognition camera.

[0394] Cloud storage: Remote storage for saving data on detected abnormal operation. Amazon AWS or Google Cloud Storage are possible options.

[0395] Communication software: A tool for uploading abnormal driving data to the cloud and sending HTTP requests to external organizations such as the police to report it. Uses libraries such as Python's Requests library.

[0396] System Operation Description

[0397] When the system starts up, the rear camera continuously captures rearward video, and this video data is sent to the AI ​​engine in real time. After the server receives the video data and performs preprocessing, the AI ​​engine detects dangerous driving and aggressive driving.

[0398] If abnormal driving is detected, the server generates a warning message and displays it in real time on the in-vehicle display. For example, it might notify the driver with a warning message such as, "An abnormally close approach is coming from behind. Please be careful." This data is also uploaded to cloud storage and can be reviewed later.

[0399] Furthermore, an emotion-recognition camera captures the driver's face, and the video data is sent to a server. The server's emotion recognition engine analyzes this data to recognize the driver's emotional state (e.g., tension or stress). Based on the recognized emotion, the content of warning messages and corrective advice is adjusted. For example, if the driver is tense, more polite and detailed advice can be provided.

[0400] Specific usage examples

[0401] For example, suppose a vehicle approaches abnormally closely from behind while driving on a highway. In this case, the rear camera captures the image behind the vehicle, and the server's AI engine detects the abnormal approach. The server generates a warning message such as, "There is a vehicle approaching abnormally closely from behind. Change lanes to avoid it," and sends it to the in-car display. At the same time, a camera is also operating to capture the driver's face, and this video data is sent to the server. The server's emotion recognition engine analyzes this data and recognizes that the driver is stressed. Based on this, it checks whether the warning message is considerate of the driver's emotional state and adjusts it as needed. Furthermore, the details of this abnormal driving and the driver's emotional state are stored in the cloud.

[0402] Example of a prompt

[0403] Regarding the rear camera frame: "The system analyzes the rear camera footage to detect abnormally close approaches from behind."

[0404] In the case of a facial emotion recognition camera: "It analyzes the driver's facial image and recognizes emotional states such as tension and stress."

[0405] This allows for appropriate measures to be taken in response to the driver's emotional state, thereby improving safety and reducing stress.

[0406] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0407] Step 1:

[0408] The terminal activates the rear camera and captures the rear view of the vehicle. It receives live video from the rear camera as input, encodes the video data, and sends it to the server. Specifically, it captures the rear camera's video input in real time, compresses the data, and uploads it to the server via the network.

[0409] Step 2:

[0410] The server preprocesses the received video data. It receives encoded video data as input and decodes and preprocesses it into a format that is easy for the AI ​​engine to analyze. Specifically, it decodes the video data and adjusts the frame rate and removes noise as needed.

[0411] Step 3:

[0412] The server analyzes pre-processed video data using an AI engine. It receives pre-processed video data as input and uses an AI model to detect dangerous driving and aggressive driving. Specifically, it inputs the prompt message "Analyze the rear camera footage and detect abnormally close driving from behind" into the generating AI model and obtains the analysis results.

[0413] Step 4:

[0414] The server generates warning messages for detected abnormal operation. It receives analysis results as input and uses templates to create appropriate warning messages. Specifically, depending on the type of abnormal operation, it generates messages such as "An abnormal approach is coming from behind. Please be careful," and sends them to the terminal.

[0415] Step 5:

[0416] The terminal notifies the driver of received warning messages. It receives warning messages from the server as input and displays them on the in-car display or smart glasses. Specifically, it notifies the driver of the warning messages visually and audibly.

[0417] Step 6:

[0418] The device's emotion recognition camera captures the driver's facial image. It receives live video of the driver's face as input, encodes the video data, and sends it to the server. Specifically, the emotion recognition camera captures the driver's facial image in real time, compresses the data, and uploads it to the server.

[0419] Step 7:

[0420] The server preprocesses the received facial video data. It receives encoded facial video data as input and decodes and preprocesses it into a format that is easy for the emotion recognition engine to analyze. Specifically, it decodes the facial video data and adjusts the frame rate and removes noise as needed.

[0421] Step 8:

[0422] The server analyzes pre-processed facial video data using an emotion recognition engine. It receives pre-processed facial video data as input and uses an AI model to recognize the driver's emotional state. Specifically, it inputs the prompt message "Analyze the driver's facial video and recognize emotional states such as tension and stress" into the generating AI model and obtains the analysis results.

[0423] Step 9:

[0424] The server adjusts warning messages and advice based on the driver's emotional state. It receives the result of emotion recognition as input and personalizes warning messages and advice based on that result. For example, if the driver is stressed, the message might be adjusted to something like, "There is a vehicle approaching dangerously close from behind. Please remain calm and take appropriate action."

[0425] Step 10:

[0426] The server stores abnormal driving data and driver emotional states in the cloud. It receives abnormal driving and emotional state data as input and uploads it to cloud storage. Specifically, it converts the data into an appropriate format and sends it to the cloud storage service for storage.

[0427] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0428] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0429] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0430] [Second Embodiment]

[0431] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0432] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0433] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0434] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0435] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0436] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0437] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0438] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0439] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0440] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0441] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0442] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0443] Modes for carrying out the invention

[0444] The system according to the present invention is an integrated system that includes a rear camera, AI-based video analysis, driver warnings, data storage in the cloud, and notification to the police, etc. The processing of the program of this system will be described below in natural language.

[0445] System program processing

[0446] Upon system startup, the terminal automatically activates the rear camera and continuously captures rearward video. This video data is encoded and sent to the server using a secure communication protocol. The server receives the transmitted video data and performs preprocessing for input into the AI ​​engine. This preprocessing includes frame extraction, resizing, and normalization.

[0447] The AI ​​engine analyzes pre-processed video data to detect dangerous driving and aggressive driving. The analysis extracts features such as abnormal following distances, sudden speed changes, and frequent lane changes, and uses a machine learning model to calculate an anomaly score. Based on these analysis results, the server generates a warning message if abnormal driving is detected.

[0448] The generated warning message is sent to the terminal in real time. The terminal displays the received warning message to the driver and also provides an audio notification. For example, the warning message might read, "There is a vehicle approaching dangerously close from behind. Change lanes to avoid it," and the same message might be announced aloud.

[0449] Furthermore, the data of detected abnormal operation is uploaded to and stored in the cloud. This includes video frames and metadata (date, time, location information, etc.) from the time the abnormality was detected. This makes it possible to review the details of the detected abnormal operation at a later date.

[0450] If the driver deems the situation serious, they are given the option to report it to the police through the terminal's interface. When the driver selects the reporting option, simply pressing the report button activates voice support, and pre-configured information (such as location and license plate number) is sent to the police via voice and text.

[0451] Specific example

[0452] For example, suppose a vehicle approaches abnormally closely from behind while driving on a highway. In this case, the rear camera captures the image behind the vehicle, and the AI ​​engine detects the abnormal proximity. The server notifies the terminal that an abnormal proximity has been detected, and the terminal displays the message, "There is a vehicle approaching abnormally closely from behind. Change lanes to avoid it," and also notifies the driver of the same message by voice. This situation is saved to the cloud and can be reviewed later. If the driver presses the report button, the police are automatically notified, and a notification is sent stating, "This vehicle is currently traveling at point XX, and is experiencing aggressive driving from behind. The vehicle number is XX-1234."

[0453] In this way, the system ensures driver safety in real time and provides an environment that allows for a rapid and appropriate response to abnormal situations.

[0454] The following describes the processing flow.

[0455] Program processing steps

[0456] Step 1:

[0457] Rear camera activation and video capture start

[0458] Terminal: When the vehicle engine starts, the rear camera automatically activates. After activation, the rear camera begins capturing rearward video and generates video data frame by frame.

[0459] Step 2:

[0460] Sending video data

[0461] Terminal: Encodes captured video frames at high speed and sends them to the server using a secure communication protocol.

[0462] Step 3:

[0463] Receiving and preprocessing video data

[0464] Server: Receives video data, stores it in a buffer, and performs preprocessing to prepare it for input into the AI ​​engine. This preprocessing includes frame extraction, resizing, and normalization.

[0465] Step 4:

[0466] AI analysis

[0467] Server: Pre-processed video data is input into the AI ​​engine to analyze patterns of dangerous driving and aggressive driving. Specifically, features such as abnormal following distances, sudden speed changes, and frequent lane changes are extracted, and an abnormality score is calculated using a machine learning model.

[0468] Step 5:

[0469] Generating warning messages

[0470] Server: When the AI ​​engine detects abnormal driving, it generates a warning message. This message includes the type of dangerous driving detected and the actions the driver should take.

[0471] Step 6:

[0472] Sending a warning

[0473] Server: Sends generated warning messages to terminals in real time.

[0474] Step 7:

[0475] Warning display and audio notification

[0476] Terminal: Displays received warning messages to the driver and also provides voice notifications. For example, it displays a message such as, "There is a vehicle approaching dangerously close from behind. Change lanes to avoid it," and also provides voice notifications.

[0477] Step 8:

[0478] Cloud storage of data

[0479] Server: Uploads and stores data on detected abnormal operation to the cloud. This data includes video frames, date and time, location information, etc.

[0480] Step 9:

[0481] Presentation of reporting options

[0482] Terminal: Presents the driver with the option of whether or not to report to the police. A message such as "Do you want to report this?" is displayed on the screen.

[0483] Step 10:

[0484] Execution of the report

[0485] User: The driver selects the reporting option.

[0486] Device: When the reporting button is pressed, voice support is activated, and detailed information, including location and vehicle information, is sent to the police via voice and text.

[0487] This allows drivers to receive real-time warnings and quickly contact the police as needed, significantly improving safety.

[0488] (Example 1)

[0489] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0490] In recent years, accidents caused by dangerous driving and intimidating driving have been increasing, and there is a need to detect these driving behaviors in real time and respond quickly. However, current dashcams and driver assistance systems are insufficient in their ability to adequately detect dangerous driving behavior behind the vehicle and warn the driver. Therefore, a system is needed to ensure driver safety and respond immediately to abnormal driving.

[0491] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0492] In this invention, the server includes means for capturing rear-view video of the transport equipment using a rear camera, means for analyzing the captured rear-view video in real time using artificial intelligence to detect dangerous driving or intimidating driving, means for sending a warning message to the driver and providing advice on countermeasures for detected abnormal driving, auxiliary means for saving driver safety data to a storage device based on the analysis results, means for saving video data to an external storage device using a secure communication protocol, and voice support means for notifying an external organization at the driver's instruction. This enables the detection of dangerous driving behavior behind the vehicle in real time, prompting the driver to be warned and encouraging appropriate action, and furthermore, data on abnormal driving can be saved to the cloud and reported to an external organization as needed.

[0493] A "rear camera" is a video capture device used to capture images from the rear of a vehicle.

[0494] "Artificial intelligence" is a general term that includes machine learning algorithms and deep learning technologies used to analyze captured rear-view video footage and detect abnormal driving behavior.

[0495] "Dangerous driving" refers to driving behavior that is unstable or abnormal compared to normal driving conditions.

[0496] "Intimidating driving" refers to driving behavior that intentionally causes fear or anxiety to other drivers or pedestrians.

[0497] A "warning message" is notification information intended to prompt the driver to take action in response to detected abnormal driving behavior.

[0498] "Driver" refers to a person operating a vehicle.

[0499] A "storage device" refers to hardware or storage media used to store data.

[0500] "External storage devices" refer to external data storage media, such as cloud storage, used to save data outside of the local system.

[0501] "Auxiliary means" refers to additional functions or technologies used to operate the main function.

[0502] A "secure communication protocol" refers to technologies and standards for ensuring secure data communication. Specifically, this includes protocols such as HTTPS.

[0503] "Reporting" refers to the act of transmitting information about abnormal driving to an external organization when such behavior is detected.

[0504] "Voice support" refers to a function that uses voice to provide information to users or to report information to external organizations.

[0505] "External organizations" refer to organizations including the police and other public institutions.

[0506] Modes for carrying out the invention

[0507] The system according to the present invention is an integrated system that includes a rear camera, artificial intelligence-based video analysis, driver warnings, data storage in the cloud, and notification to an external organization. Specific embodiments of this system are described below.

[0508] Rear camera activation and video capture

[0509] When the system starts up, the terminal automatically activates the rear camera, which captures video footage from behind the transport equipment. This video data is encoded in real time and converted to a stream format (e.g., H.264).

[0510] Sending video data

[0511] The encoded video data is sent from the terminal to the server using a secure communication protocol (e.g., HTTPS). The terminal buffers the video data, divides it into appropriate packet sizes, encrypts it, and sends it to the server.

[0512] Preprocessing of video data

[0513] The server preprocesses the received video data. This preprocessing includes frame extraction, resizing, and normalization. The preprocessed data is then input into the artificial intelligence engine.

[0514] Analysis of video data

[0515] The server uses an artificial intelligence engine to analyze pre-processed video data. The analysis extracts features such as abnormal following distances, sudden speed changes, and frequent lane changes, and calculates an anomaly score using machine learning models (e.g., TensorFlow, PyTorch). This allows for the detection of dangerous driving and aggressive driving behavior.

[0516] Generating and sending warning messages

[0517] Based on the analysis results, the server generates a warning message if abnormal operation is detected. The generated warning message is sent to the terminal in real time.

[0518] Driver warning notification

[0519] The device notifies the driver of any received warning messages. The notification is displayed on the screen and also alerts the driver with an audio alert. For example, a message such as "There is a vehicle approaching dangerously close from behind. Change lanes to avoid it" will be displayed, and the same information will be announced by voice.

[0520] Cloud storage of abnormal driving data

[0521] The server uploads and stores the detected abnormal operation data to the cloud. This data includes video frames and metadata (date and time, location information, etc.) at the time of the anomaly detection. The data is stored using cloud storage services (e.g., AWS S3, Google Cloud Storage).

[0522] Driver's report to the police

[0523] If the driver deems the situation serious, they will be given the option to call the police through the terminal's interface. If the driver selects the calling option, simply pressing the call button will activate voice support, and pre-configured information (location, license plate number, etc.) will be sent to the police in both voice and text format.

[0524] Specific example

[0525] For example, if a vehicle approaches abnormally closely from behind while driving on a highway, the rear camera captures the image behind the vehicle, and the artificial intelligence engine detects the abnormal proximity. The server notifies the terminal that an abnormal proximity has been detected, and the terminal displays the message, "There is a vehicle approaching abnormally closely from behind. Change lanes to avoid it," and also notifies the driver of the same message by voice. This situation is saved to the cloud and can be reviewed later. If the driver presses the report button, the police are automatically notified, and a notification is sent stating, "This vehicle is currently traveling at point XX, and is experiencing aggressive driving from behind. The vehicle number is XX-1234."

[0526] Examples of prompts for generative AI models

[0527] For example, analysis using a generated AI model can be performed by using the following prompt statement.

[0528] You are developing an advanced driver assistance system. This system integrates a rear camera, AI-powered video analysis, driver warnings, cloud data storage, and notification to external organizations. Explain the program's processes and provide specific examples for the following scenarios.

[0529] scenario:

[0530] 1. Please explain how the device activates the rear camera and captures rearward video when the system starts up.

[0531] 2. Explain how the artificial intelligence engine analyzes video data to detect dangerous or aggressive driving.

[0532] 3. Please explain in detail the mechanism for displaying and notifying drivers of warnings.

[0533] 4. Please explain the data storage and reporting process for abnormal operation to external organizations.

[0534] Specific example:

[0535] Describe a scenario in which a vehicle approaches unusually closely from behind while driving on a highway, and specifically explain how each process functions in that situation.

[0536] By inputting this prompt into the AI ​​model, it is possible to generate a detailed explanation of the system's operation and various functions.

[0537] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0538] System program processing flow

[0539] Step 1:

[0540] The device automatically activates the rear camera when the system starts up.

[0541] Specific operation: The terminal powers on the rear camera via internal control software and instructs the camera to begin capturing rearward video. The rear camera continuously captures rearward video and generates encoded stream data (e.g., in H.264 format).

[0542] Input: System power signal.

[0543] Output: Rear camera capture started, encoded video data.

[0544] Step 2:

[0545] The terminal sends the encoded video data to the server using a secure communication protocol (e.g., HTTPS).

[0546] Specific operation: The terminal buffers the video data, divides it into packets, and encrypts them. It then sends them to the server via the network stack.

[0547] Input: Encoded video data from the rear camera.

[0548] Output: Encrypted video data sent to the server.

[0549] Step 3:

[0550] The server preprocesses the received video data.

[0551] Specific operation: The server receives data, decodes the packets, and divides them into frames. Each frame is resized and normalized. After that, the pre-processed data is input into the AI ​​engine.

[0552] Input: Encrypted video data packets.

[0553] Output: Preprocessed video data (resized and normalized frames).

[0554] Step 4:

[0555] The server analyzes the pre-processed video data.

[0556] Specific operation: The AI ​​engine extracts features to detect abnormalities such as insufficient following distance, sudden speed changes, and frequent lane changes. A machine learning model (e.g., TensorFlow, PyTorch) is used to calculate an anomaly score and determine whether the driving is dangerous or aggressive.

[0557] Input: Pre-processed video data.

[0558] Output: Abnormal operation score and judgment result.

[0559] Step 5:

[0560] Based on the analysis results, the server generates a warning message if abnormal operation is detected.

[0561] Specific operation: When abnormal operation is detected, the server generates a warning in text format and creates a warning message for the driver.

[0562] Input: Score and judgment result for abnormal operation.

[0563] Output: Warning message.

[0564] Step 6:

[0565] The server sends the generated warning message to the terminal.

[0566] Specific operation: The server sends a warning message to the terminal in real time.

[0567] Input: Warning message.

[0568] Output: Send to the terminal.

[0569] Step 7:

[0570] The terminal notifies the driver of the received warning message.

[0571] Specific operation: The device displays a warning message on its screen and notifies the driver of the same content via an audio alert. For example, it might inform the driver, "There is a vehicle approaching dangerously close from behind. Change lanes to avoid it."

[0572] Input: Warning message.

[0573] Output: Display, audio alert.

[0574] Step 8:

[0575] The server uploads and stores the detected abnormal operation data to the cloud.

[0576] Specific operation: The server collects video frames and metadata (date, time, location information, etc.) from the time of anomaly detection into storage data and uploads it to a cloud storage service (e.g., AWS S3, Google Cloud Storage).

[0577] Input: Anomaly detection data (video frames, metadata).

[0578] Output: Data stored in cloud storage.

[0579] Step 9:

[0580] If the driver deems the situation serious, they will notify the police via the device.

[0581] Specific operation: When the driver presses the call button, the device activates its voice assistance function and sends pre-configured information (location information, license plate number, etc.) to the police in both voice and text format.

[0582] Input: Driver presses the call button, saved information.

[0583] Output: Automatic notification to the police (voice and text format).

[0584] (Application Example 1)

[0585] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0586] In recent years, aggressive driving and dangerous driving have become a social problem, making prevention and response to these issues urgent. Furthermore, in order to ensure the safety of individuals, including pedestrians, there is a need for a system that can detect abnormal behavior in real time and provide appropriate warnings and countermeasures. The present invention aims to provide a system that ensures the safety of a wider range of individuals by utilizing not only automobiles but also smart glasses.

[0587] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0588] In this invention, the server includes means for capturing rear-view video of a vehicle using a rear camera, means for analyzing the captured rear-view video in real time using AI to detect dangerous driving and aggressive driving, means for sending a warning message to the driver and providing advice on countermeasures for detected abnormal driving, means for uploading and storing the detected abnormal driving data to the cloud, means for providing voice support for notifying external organizations such as the police at the driver's instruction, means for capturing rear-view video using a rear camera mounted on smart glasses to detect approaching abnormal behavior, and means for providing the user with a visual and audible warning message when approaching abnormal behavior is detected. This enables universal safety assurance not only for vehicle drivers but also for pedestrians.

[0589] A "rear camera" is a camera device installed in cars or smart glasses that captures images of the area behind the vehicle.

[0590] "AI" is an abbreviation for artificial intelligence, a technology that uses machine learning and deep learning to analyze data and detect patterns and anomalies.

[0591] "Dangerous driving" refers to driving behaviors that endanger other road users, such as sudden changes in speed or frequent lane changes.

[0592] "Aggressive driving" refers to driving behavior that intentionally causes fear to other drivers, such as overtaking other vehicles and slamming on the brakes, or driving dangerously close to them.

[0593] A "warning message" is a message that provides a warning or evacuation order to the driver or user when dangerous driving or aggressive driving is detected.

[0594] "Cloud" refers to external data storage and computing resources provided via the internet.

[0595] "Voice support means" refers to functions that use voice to provide information to users or to report to external organizations.

[0596] "Smart glasses" are glasses-type wearable devices that have computer functions and assist the user's visual information.

[0597] "Abnormal behavior" refers to actions that deviate from normal behavioral patterns and may pose a danger to the user or others.

[0598] "Means of providing warning messages visually and audibly" refers to functions that use displays and speakers to convey warning information to users visually and audibly.

[0599] The system according to the present invention is an integrated system for ensuring the safety of automobile drivers and pedestrians. This system includes a rear camera, AI-based video analysis, warning messages to the user, data storage to the cloud, and notification to the police, etc. Embodiments of this system will be described below.

[0600] System hardware and software configuration

[0601] Hardware:

[0602] Smart glasses (e.g., typical wearable devices)

[0603] Rear camera (mounted in cars and smart glasses)

[0604] Speaker (for voice notifications)

[0605] GPS module (for acquiring location information)

[0606] software:

[0607] Video capture API (for acquiring video data)

[0608] AI models (for video analysis, e.g., TensorFlow, PyTorch)

[0609] Cloud storage (for data storage, e.g., AWS S3)

[0610] Secure communication protocols (e.g., HTTPS)

[0611] Reporting API (e.g. Twilio, Firebase)

[0612] System Operation Overview

[0613] 1. Startup and video capture

[0614] When smart glasses or a car are activated, the rear camera automatically activates and captures footage of the area behind the vehicle. The video data is encoded in real time.

[0615] 2. Video data preprocessing

[0616] The server receives the video data and performs preprocessing. This includes frame extraction, resizing, and normalization.

[0617] 3. AI analysis

[0618] Pre-processed video data is analyzed by an AI engine. The AI ​​engine detects abnormal behavior (such as dangerous driving, road rage, or suspicious individuals approaching).

[0619] 4. Generate and notify warning messages

[0620] When abnormal behavior is detected, the server generates a warning message and notifies the user through smart glasses or the car's display and speakers. For example, the message might say, "There is a vehicle approaching you dangerously closely from behind. Change lanes to avoid it."

[0621] 5. Data storage and reporting

[0622] Data on detected abnormal behavior is stored in the cloud. This includes location information, date and time, and video frames. When a user presses the report button, voice support is activated, and location information and video data are automatically sent to the police.

[0623] Specific example

[0624] For example, if a pedestrian wearing smart glasses is walking along a school route and a suspicious person approaches from behind, the rear camera captures the situation. The AI ​​engine detects the abnormal approach and displays a message saying, "Someone is approaching you from behind. Please move to a safe place," and also provides an audio notification with the same message. If the user presses the emergency call button, a report is sent to the police stating, "I am currently walking at location XX, and a suspicious person is approaching from behind."

[0625] Example of a prompt

[0626] Please create a smart glasses application that works as follows: Use the smart glasses' rear camera to capture rearward video, and have AI detect abnormal behavior (following, sudden approach). Implement a function to provide visual and audible warnings upon detection, and automatically notify the police in emergencies.

[0627] In summary, this system provides a technical solution to ensure the safety of vehicle drivers and pedestrians in real time and to respond quickly and appropriately.

[0628] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0629] Step 1:

[0630] Smart glasses or a car terminal are activated. The terminal automatically activates the rear camera and captures rearward video in real time. The captured video data is encoded and sent to the server using a secure communication protocol (e.g., HTTPS). The input is video data from the rear camera, and the output is encoded video data.

[0631] Step 2:

[0632] The server preprocesses the received video data. Specifically, it extracts frames, resizes them, and normalizes them. This preprocessing prepares the video for efficient analysis by the AI ​​engine. The input is encoded video data, and the output is preprocessed video data.

[0633] Step 3:

[0634] The server's AI engine analyzes pre-processed video data. The AI ​​engine uses a machine learning model to detect abnormal behavior (such as dangerous driving, road rage, or suspicious individuals approaching). Specifically, it extracts features such as distance between vehicles, speed changes, and lane change frequency, and calculates an abnormality score. The input is pre-processed video data, and the output is the result of detecting abnormal behavior.

[0635] Step 4:

[0636] The server generates a warning message if abnormal behavior is detected. The warning message is provided visually and audibly. The generated warning message is sent to the terminal using a secure communication protocol. The input is the result of the detected abnormal behavior, and the output is the warning message.

[0637] Step 5:

[0638] The device notifies the user of received warning messages. Specifically, it conveys warning messages visually and audibly using the smart glasses' display and speaker. For example, it might say, "There is a vehicle approaching you dangerously close from behind. Change lanes to avoid it." The input is the warning message, and the output is the notification to the user.

[0639] Step 6:

[0640] The server saves data on detected abnormal behavior to the cloud. The saved data includes date and time, location information, and video frames. Cloud storage (e.g., AWS S3) is used for data storage. The input is the detected abnormal behavior and associated metadata, and the output is the data stored in the cloud.

[0641] Step 7:

[0642] When a user presses the emergency call button, the device's voice support function is activated. Pre-configured location information and video data are transmitted to the police via voice and text. The input consists of the user's call instructions and related data (location information, video data), and the output is a notification to the police.

[0643] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0644] Modes for carrying out the invention

[0645] The system according to the present invention is an integrated system that combines a rear camera, AI-based video analysis, driver warnings, data storage to the cloud, and notification to the police, with an emotion engine that recognizes the user's emotions. The processing of this system's program will be described below in natural language.

[0646] System program processing

[0647] When the system starts up, the terminal activates the rear camera and continuously captures video footage of the area behind the vehicle. This video data is encoded and sent to the server, which receives the video data and performs preprocessing for analysis by the AI ​​engine. The AI ​​engine analyzes the preprocessed data to detect dangerous driving and aggressive driving.

[0648] Based on the analysis results, the server generates a warning message if abnormal operation is detected. The message is sent to the terminal in real time, and the terminal notifies the driver of the received message. Furthermore, data on abnormal operation is stored in the cloud and can be reviewed later.

[0649] The present invention also includes a system for recognizing the driver's emotions. Specifically, a camera mounted on the terminal captures an image of the driver's face and transmits this data to a server. The server uses an emotion engine to analyze the image of the driver's face and recognize their emotional state. For example, it can detect states such as tension, stress, and relaxation.

[0650] Based on recognized emotions, the content of warning messages and advice is adjusted. For example, if a driver is stressed, more detailed and considerate advice may be provided. Furthermore, emotional states are included as additional information in data stored in the cloud, allowing for later analysis.

[0651] Furthermore, it includes a feature that automatically prompts a police alert if the driver is experiencing particular tension or stress. This feature allows for quick external assistance to be sought if the driver's emotional state is deemed to be a factor detrimental to safe driving.

[0652] Specific example

[0653] For example, suppose you are driving on a highway and a vehicle approaches you dangerously closely from behind. In this case, the rear camera captures the image behind you, and the server's AI engine detects the dangerous proximity. The server generates a warning message such as, "There is a vehicle approaching you dangerously closely from behind. Change lanes to avoid it," and sends it to your device.

[0654] Simultaneously, a camera is operating to capture video of the driver's face, and this video data is sent to a server. The server's emotion engine analyzes this data to recognize that the driver is stressed. Based on this, it checks whether the warning message is considerate of the driver's emotional state and adjusts it as needed.

[0655] Furthermore, the details of this abnormal driving and the driver's emotional state are saved to the cloud. Also, if the driver presses the report button, the system sends information to the police stating, "This vehicle is currently traveling at location XX and is experiencing aggressive driving from behind. The vehicle's license plate number is XX-1234. The driver is extremely distressed."

[0656] In this way, this system not only detects dangerous driving but also provides comprehensive safety measures that take into account the driver's emotional state.

[0657] The following describes the processing flow.

[0658] Modes for carrying out the invention

[0659] System program processing

[0660] Step 1:

[0661] Rear camera activation and video capture start

[0662] Terminal: When the vehicle engine starts, the rear camera automatically activates. After activation, the rear camera begins capturing rearward video and generates video data frame by frame.

[0663] Step 2:

[0664] Sending video data

[0665] Terminal: Encodes captured video frames at high speed and sends them to the server using a secure communication protocol.

[0666] Step 3:

[0667] Receiving and preprocessing video data

[0668] Server: Receives video data, stores it in a buffer, and performs preprocessing to prepare it for input into the AI ​​engine. This preprocessing includes frame extraction, resizing, and normalization.

[0669] Step 4:

[0670] AI analysis

[0671] Server: Pre-processed video data is input into the AI ​​engine to analyze patterns of dangerous driving and aggressive driving. Specifically, features such as abnormal following distances, sudden speed changes, and frequent lane changes are extracted, and an abnormality score is calculated using a machine learning model.

[0672] Step 5:

[0673] Generating warning messages

[0674] Server: When the AI ​​engine detects abnormal driving, it generates a warning message. This message includes the type of dangerous driving detected and the actions the driver should take. For example, it might say, "There is a vehicle approaching dangerously closely from behind. Change lanes to avoid it."

[0675] Step 6:

[0676] Sending a warning

[0677] Server: Sends generated warning messages to terminals in real time.

[0678] Step 7:

[0679] Warning display and audio notification

[0680] Terminal: Displays received warning messages to the driver and also provides voice notifications. For example, a message is displayed on the screen, and a voice notification says, "There is a vehicle approaching you dangerously close from behind. Change lanes to avoid it."

[0681] Step 8:

[0682] Cloud storage of data

[0683] Server: Uploads and stores data on detected abnormal operation to the cloud. This data includes video frames, date and time, location information, etc.

[0684] Step 9:

[0685] Video capture of the driver's face

[0686] Terminal: The camera operates to capture video of the driver's face. This video data is encoded and sent to the server.

[0687] Step 10:

[0688] Emotion recognition by an emotion engine

[0689] Server: The server analyzes the received video of the driver's face using an emotion engine to recognize their emotional state. Specifically, it analyzes the driver's facial expressions and condition to determine emotions such as "tension," "stress," and "relaxation."

[0690] Step 11:

[0691] Adjusting emotion-based warning messages

[0692] Server: Based on recognized emotions, it adjusts the content of warning messages and corrective advice. For example, if the driver is stressed, it provides more polite and detailed advice.

[0693] Step 12:

[0694] Cloud storage of emotional data

[0695] Server: The recognized emotional states are added to data stored in the cloud, making them available for later analysis.

[0696] Step 13:

[0697] Presentation of reporting options

[0698] Terminal: Displays an option on the screen for the driver to report the incident to the police. For example, it might display the option, "Do you want to report this?"

[0699] Step 14:

[0700] Execution of the report

[0701] User: The driver selects the reporting option.

[0702] Terminal: When the reporting button is pressed, voice support is activated, and detailed information, including location, vehicle information, and emotional state, is sent to the police via voice and text. For example, information such as, "This vehicle is currently traveling at location XX and is experiencing aggressive driving from behind. The vehicle number is XX-1234. The driver is extremely agitated," will be sent.

[0703] As described above, this system provides comprehensive safety measures that take into account not only the detection of dangerous driving and aggressive driving, but also the driver's emotional state.

[0704] (Example 2)

[0705] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0706] Conventional automotive safety systems have the ability to detect dangerous driving and aggressive driving, but they cannot take into account the driver's emotional state, which can lead to increased stress and tension for the driver. Furthermore, while data on abnormal driving is stored, the lack of sufficient specific advice on countermeasures limits the potential for improving safety.

[0707] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for capturing rear-view video of the vehicle using a rear camera, means for analyzing the captured rear-view video in real time using AI to detect dangerous driving and aggressive driving, means for sending a warning message to the driver regarding the detected abnormal driving and providing advice on countermeasures, means for uploading and saving the detected abnormal driving data to the cloud, means for capturing video of the driver's face and performing emotion analysis, and means for adjusting the warning message based on the observed emotions of the driver. This makes it possible to not only detect dangerous driving and aggressive driving in real time, but also to implement comprehensive safety measures that take into account the driver's emotional state.

[0708] A "rear camera" is a camera mounted on the rear of a vehicle, used to capture images of the area behind the vehicle.

[0709] "AI" stands for Artificial Intelligence, and refers to an algorithm or system that automatically performs specific tasks.

[0710] "Dangerous driving" refers to driving behavior that violates the Road Traffic Act and endangers safety.

[0711] "Aggressive driving" refers to intentionally engaging in intimidating driving behavior towards other vehicles.

[0712] A "warning message" is a message that the system issues to the driver to warn them about dangerous driving or aggressive driving.

[0713] "Safety advice" refers to information that provides drivers with specific instructions and advice to drive safely.

[0714] "Cloud" refers to a group of servers that provide data storage and processing functions via the internet.

[0715] "Emotional analysis" is a technology that evaluates a driver's emotional state based on facial expressions and other factors using video data of their face.

[0716] "Reporting methods" refer to a function that, in response to the driver's instructions, allows for reporting abnormal driving situations to external organizations such as the police using voice or text.

[0717] "Preprocessing means" refers to techniques that process video data, such as noise reduction and frame selection, before analysis by AI.

[0718] Modes for carrying out the invention

[0719] The system according to the present invention is an integrated safety driving support system that includes a rear camera, an AI engine, an emotion analysis engine, a driver warning message function, data storage to the cloud, and an automatic notification function to the police.

[0720] Hardware and software configuration:

[0721] The rear camera is mounted on the back of the car and continuously captures images of the area behind it.

[0722] The device has the function of encoding video data from the rear camera and sending it to the server.

[0723] The server has hardware and software to preprocess the received video data and analyze it with an AI engine.

[0724] The AI ​​engine includes algorithms that detect dangerous driving and aggressive driving in real time.

[0725] Another camera on the device captures video of the driver's face, providing data to recognize the driver's emotional state.

[0726] The emotion analysis engine is built into the server and analyzes the transmitted facial video data to recognize the driver's emotional state (tension, stress, relaxation, etc.).

[0727] The cloud server securely stores data such as abnormal driving behavior and the driver's emotional state, allowing for later review and analysis.

[0728] The warning message function sends messages generated on the server to the terminal in real time, and the terminal notifies the driver.

[0729] The automatic reporting function has a means of reporting to an external organization (e.g., the police) based on the driver's instructions.

[0730] Specific examples of how the system works:

[0731] Suppose a vehicle is detected approaching abnormally closely from behind while driving on a highway. The rear camera captures the image behind the vehicle, the video data is encoded on the terminal, and then sent to the server.

[0732] The server uses an AI engine to detect abnormal proximity. It generates a warning message such as, "There is a vehicle approaching abnormally closely from behind. Change lanes to avoid it," and sends it to the terminal.

[0733] Simultaneously, a camera capturing the driver's face is also operating, and the video data is sent to a server. An emotion analysis engine analyzes this data and recognizes that the driver is feeling stressed.

[0734] Based on this understanding, the server adjusts the content of the warning message. For example, it might add advice such as, "Stay calm and check behind you."

[0735] All data (including details of abnormal driving and the driver's emotional state) is stored on a cloud server.

[0736] When the driver presses the call button, the automatic call function sends a message to the police saying, "This vehicle is currently traveling at location XX, and is experiencing aggressive driving from behind. The vehicle's license plate number is XX-1234. The driver is extremely distressed."

[0737] Example of a prompt:

[0738] 1. "Please describe in detail the process from detecting abnormal operation to sending a warning message in conjunction with the rear camera."

[0739] 2. "Please explain specific methods for adjusting warning messages using driver facial recognition and emotion analysis."

[0740] Thus, the system according to the present invention not only detects dangerous driving and aggressive driving in real time, but also provides comprehensive safety measures that take into account the driver's emotional state.

[0741] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0742] Step 1: System startup

[0743] The terminal prepares to activate the rear camera simultaneously with the system startup. The input is the system startup signal, and the output is the completion of the rear camera startup.

[0744] Specifically, when the device's power button is pressed, the internal software sends a command to activate the rear camera.

[0745] Step 2: Video capture using the rear camera

[0746] The device activates the rear camera and continuously captures rearward video. The input is real-time video data, and the output is encoded rearward video data.

[0747] Specifically, the rear camera captures the rear view frame by frame and sends it to the terminal. The terminal then encodes this data.

[0748] Step 3: Encode and transmit video data

[0749] The terminal encodes the captured video data. The input is raw video data, and the output is encoded data. After encoding, this data is sent to the server.

[0750] Specifically, the terminal compresses the video data into a format such as MPEG and sends it to the server using a communication protocol.

[0751] Step 4: Video analysis using an AI engine

[0752] The server preprocesses the received encoded video data. The input is encoded video data, and the output is preprocessed data. Preprocessing such as noise reduction and frame selection is performed.

[0753] The server's AI engine analyzes pre-processed data to detect dangerous driving and aggressive driving. The input for the analysis is pre-processed data, and the output is the detection result of abnormal driving.

[0754] Step 5: Generate and send the warning message.

[0755] If abnormal operation is detected, the server generates a warning message. The input is the result of the abnormal operation detection, and the output is the generated warning message. Next, this message is sent to the terminal in real time, and the terminal notifies the driver.

[0756] Specifically, the server creates a warning message based on a template and sends it to the terminal using the TCP / IP protocol. The terminal then displays an audio alert or a pop-up on the screen.

[0757] Step 6: Driver's Emotion Recognition

[0758] The camera on the device captures video of the driver's face. The input is video data of the driver's face, and the output is encoded video data of the face. This is sent to the server.

[0759] The server's emotion engine analyzes facial video data to recognize the driver's emotional state. The input is encoded facial video data, and the output is the result of the emotion analysis.

[0760] Step 7: Adjusting Emotion-Based Warning Messages

[0761] The server adjusts warning messages to be more detailed and thoughtful if the driver is stressed. The input is the result of sentiment analysis, and the output is the adjusted warning message.

[0762] Specifically, the server selects a warning message template and adjusts the content of the advice based on the sentiment analysis results.

[0763] Step 8: Save data to the cloud

[0764] The server stores data on abnormal operation and sentiment analysis on a cloud server. The input is the status of abnormal operation and the results of sentiment analysis, and the output is the data stored in the cloud.

[0765] Specifically, the server uploads data to the cloud storage service via the database.

[0766] Step 9: Automatically report to the police

[0767] When the driver presses the emergency call button, the server automatically initiates the process of notifying the police. The input is the driver's emergency call signal, and the output is the emergency call message sent to the police.

[0768] Specifically, the server generates a notification message containing location and vehicle information, and sends it to the police using the appropriate communication protocol.

[0769] (Application Example 2)

[0770] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0771] Conventional in-vehicle systems were specialized in detecting dangerous driving behavior and abnormal approaches from the rear, but lacked consideration for the driver's emotional state. This made it difficult to respond appropriately when the driver felt tension or stress, potentially hindering safe driving. Furthermore, warnings and advice after detecting abnormal driving were uniform, failing to provide measures tailored to the driver's emotional state. As a result, safety and driver stress reduction were not adequately achieved.

[0772] In Application Example 2, the identification processing by the identification processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for capturing rear-view video of the vehicle using a rear camera, means for analyzing the captured rear-view video in real time using AI to detect dangerous driving and aggressive driving, means for sending a warning message to the driver and providing advice on countermeasures for detected abnormal driving, means for uploading and saving the detected abnormal driving data to the cloud, voice support means for notifying external organizations such as the police at the driver's instruction, means for capturing video of the driver's face and analyzing the driver's emotional state using an emotion recognition engine, and means for adjusting the content of warning messages and countermeasure advice based on the driver's emotional state. This enables appropriate countermeasures according to the driver's emotional state, thereby improving safety and reducing stress.

[0773] A "rear camera" is a camera device used to capture images of the area behind a vehicle.

[0774] The "AI engine" is an artificial intelligence technology that analyzes captured video in real time to detect abnormal driving, aggressive driving, and the driver's emotional state.

[0775] "Dangerous driving" refers to any driving behavior that may endanger other vehicles or pedestrians.

[0776] "Aggressive driving" refers to driving behavior that is harassing or endangering to the driver, such as a vehicle following closely behind another vehicle.

[0777] A "warning message" is a message or alert designed to alert the driver to detected abnormal driving behavior.

[0778] "Countermeasure advice" refers to information that presents recommended driving actions and points to note to the driver in response to detected abnormal driving behavior.

[0779] "Cloud" refers to data storage and processing services provided via the internet, and in this context, it refers to remote storage for saving data on detected abnormal operation.

[0780] "Voice support means" refers to a function that provides voice support when the driver instructs an external organization, such as the police, to make a report.

[0781] The "emotion recognition engine" is a technology that analyzes the driver's facial image to recognize their emotional state.

[0782] "Emotional state" refers to the driver's mental and psychological state, including states such as tension, stress, and relaxation.

[0783] "Video data preprocessing" is a technique that transforms video data into a format that is easily accessible for AI analysis.

[0784] "External organizations" refer to public institutions such as the police and other support organizations.

[0785] "Abnormal driving data" refers to all data that, as a result of analyzing video footage acquired from rear cameras and other sources, is determined to indicate abnormal driving.

[0786]

[0787] The system according to the present invention is an integrated system installed in an autonomous vehicle. This system captures rearward images using a rear camera and analyzes the data with an AI engine to detect dangerous driving and aggressive driving. In addition, a camera that captures facial images and an emotion recognition engine are used to recognize the driver's emotional state.

[0788] Specific system configuration

[0789] 1. Hardware configuration:

[0790] Rear camera: A camera used to capture images of the rear of a vehicle.

[0791] Emotion Recognition Camera: A camera that captures video of the driver's face.

[0792] In-car display: A display that shows warning messages and advice to the driver. This can also be replaced by the display on a smartphone or smart glasses.

[0793] 2. Software configuration:

[0794] AI Engine: Uses platforms such as TensorFlow to analyze video from the rear camera and emotion recognition camera.

[0795] Cloud storage: Remote storage for saving data on detected abnormal operation. Amazon AWS or Google Cloud Storage are possible options.

[0796] Communication software: A tool for uploading abnormal driving data to the cloud and sending HTTP requests to external organizations such as the police to report it. Uses libraries such as Python's Requests library.

[0797] System Operation Description

[0798] When the system starts up, the rear camera continuously captures rearward video, and this video data is sent to the AI ​​engine in real time. After the server receives the video data and performs preprocessing, the AI ​​engine detects dangerous driving and aggressive driving.

[0799] If abnormal driving is detected, the server generates a warning message and displays it in real time on the in-vehicle display. For example, it might notify the driver with a warning message such as, "An abnormally close approach is coming from behind. Please be careful." This data is also uploaded to cloud storage and can be reviewed later.

[0800] Furthermore, an emotion-recognition camera captures the driver's face, and the video data is sent to a server. The server's emotion recognition engine analyzes this data to recognize the driver's emotional state (e.g., tension or stress). Based on the recognized emotion, the content of warning messages and corrective advice is adjusted. For example, if the driver is tense, more polite and detailed advice can be provided.

[0801] Specific usage examples

[0802] For example, suppose a vehicle approaches abnormally closely from behind while driving on a highway. In this case, the rear camera captures the image behind the vehicle, and the server's AI engine detects the abnormal approach. The server generates a warning message such as, "There is a vehicle approaching abnormally closely from behind. Change lanes to avoid it," and sends it to the in-car display. At the same time, a camera is also operating to capture the driver's face, and this video data is sent to the server. The server's emotion recognition engine analyzes this data and recognizes that the driver is stressed. Based on this, it checks whether the warning message is considerate of the driver's emotional state and adjusts it as needed. Furthermore, the details of this abnormal driving and the driver's emotional state are stored in the cloud.

[0803] Example of a prompt

[0804] Regarding the rear camera frame: "The system analyzes the rear camera footage to detect abnormally close approaches from behind."

[0805] In the case of a facial emotion recognition camera: "It analyzes the driver's facial image and recognizes emotional states such as tension and stress."

[0806] This allows for appropriate measures to be taken in response to the driver's emotional state, thereby improving safety and reducing stress.

[0807] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0808] Step 1:

[0809] The terminal activates the rear camera and captures the rear view of the vehicle. It receives live video from the rear camera as input, encodes the video data, and sends it to the server. Specifically, it captures the rear camera's video input in real time, compresses the data, and uploads it to the server via the network.

[0810] Step 2:

[0811] The server preprocesses the received video data. It receives encoded video data as input and decodes and preprocesses it into a format that is easy for the AI ​​engine to analyze. Specifically, it decodes the video data and adjusts the frame rate and removes noise as needed.

[0812] Step 3:

[0813] The server analyzes pre-processed video data using an AI engine. It receives pre-processed video data as input and uses an AI model to detect dangerous driving and aggressive driving. Specifically, it inputs the prompt message "Analyze the rear camera footage and detect abnormally close driving from behind" into the generating AI model and obtains the analysis results.

[0814] Step 4:

[0815] The server generates warning messages for detected abnormal operation. It receives analysis results as input and uses templates to create appropriate warning messages. Specifically, depending on the type of abnormal operation, it generates messages such as "An abnormal approach is coming from behind. Please be careful," and sends them to the terminal.

[0816] Step 5:

[0817] The terminal notifies the driver of received warning messages. It receives warning messages from the server as input and displays them on the in-car display or smart glasses. Specifically, it notifies the driver of the warning messages visually and audibly.

[0818] Step 6:

[0819] The device's emotion recognition camera captures the driver's facial image. It receives live video of the driver's face as input, encodes the video data, and sends it to the server. Specifically, the emotion recognition camera captures the driver's facial image in real time, compresses the data, and uploads it to the server.

[0820] Step 7:

[0821] The server preprocesses the received facial video data. It receives encoded facial video data as input and decodes and preprocesses it into a format that is easy for the emotion recognition engine to analyze. Specifically, it decodes the facial video data and adjusts the frame rate and removes noise as needed.

[0822] Step 8:

[0823] The server analyzes pre-processed facial video data using an emotion recognition engine. It receives pre-processed facial video data as input and uses an AI model to recognize the driver's emotional state. Specifically, it inputs the prompt message "Analyze the driver's facial video and recognize emotional states such as tension and stress" into the generating AI model and obtains the analysis results.

[0824] Step 9:

[0825] The server adjusts warning messages and advice based on the driver's emotional state. It receives the result of emotion recognition as input and personalizes warning messages and advice based on that result. For example, if the driver is stressed, the message might be adjusted to something like, "There is a vehicle approaching dangerously close from behind. Please remain calm and take appropriate action."

[0826] Step 10:

[0827] The server stores abnormal driving data and driver emotional states in the cloud. It receives abnormal driving and emotional state data as input and uploads it to cloud storage. Specifically, it converts the data into an appropriate format and sends it to the cloud storage service for storage.

[0828] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0829] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0830] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0831] [Third Embodiment]

[0832] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0833] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0834] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0835] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0836] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0837] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0838] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0839] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0840] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0841] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0842] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0843] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0844] Modes for carrying out the invention

[0845] The system according to the present invention is an integrated system that includes a rear camera, AI-based video analysis, driver warnings, data storage in the cloud, and notification to the police, etc. The processing of the program of this system will be described below in natural language.

[0846] System program processing

[0847] Upon system startup, the terminal automatically activates the rear camera and continuously captures rearward video. This video data is encoded and sent to the server using a secure communication protocol. The server receives the transmitted video data and performs preprocessing for input into the AI ​​engine. This preprocessing includes frame extraction, resizing, and normalization.

[0848] The AI ​​engine analyzes pre-processed video data to detect dangerous driving and aggressive driving. The analysis extracts features such as abnormal following distances, sudden speed changes, and frequent lane changes, and uses a machine learning model to calculate an anomaly score. Based on these analysis results, the server generates a warning message if abnormal driving is detected.

[0849] The generated warning message is sent to the terminal in real time. The terminal displays the received warning message to the driver and also provides an audio notification. For example, the warning message might read, "There is a vehicle approaching dangerously close from behind. Change lanes to avoid it," and the same message might be announced aloud.

[0850] Furthermore, the data of detected abnormal operation is uploaded to and stored in the cloud. This includes video frames and metadata (date, time, location information, etc.) from the time the abnormality was detected. This makes it possible to review the details of the detected abnormal operation at a later date.

[0851] If the driver deems the situation serious, they are given the option to report it to the police through the terminal's interface. When the driver selects the reporting option, simply pressing the report button activates voice support, and pre-configured information (such as location and license plate number) is sent to the police via voice and text.

[0852] Specific example

[0853] For example, suppose a vehicle approaches abnormally closely from behind while driving on a highway. In this case, the rear camera captures the image behind the vehicle, and the AI ​​engine detects the abnormal proximity. The server notifies the terminal that an abnormal proximity has been detected, and the terminal displays the message, "There is a vehicle approaching abnormally closely from behind. Change lanes to avoid it," and also notifies the driver of the same message by voice. This situation is saved to the cloud and can be reviewed later. If the driver presses the report button, the police are automatically notified, and a notification is sent stating, "This vehicle is currently traveling at point XX, and is experiencing aggressive driving from behind. The vehicle number is XX-1234."

[0854] In this way, the system ensures driver safety in real time and provides an environment that allows for a rapid and appropriate response to abnormal situations.

[0855] The following describes the processing flow.

[0856] Program processing steps

[0857] Step 1:

[0858] Rear camera activation and video capture start

[0859] Terminal: When the vehicle engine starts, the rear camera automatically activates. After activation, the rear camera begins capturing rearward video and generates video data frame by frame.

[0860] Step 2:

[0861] Sending video data

[0862] Terminal: Encodes captured video frames at high speed and sends them to the server using a secure communication protocol.

[0863] Step 3:

[0864] Receiving and preprocessing video data

[0865] Server: Receives video data, stores it in a buffer, and performs preprocessing to prepare it for input into the AI ​​engine. This preprocessing includes frame extraction, resizing, and normalization.

[0866] Step 4:

[0867] AI analysis

[0868] Server: Pre-processed video data is input into the AI ​​engine to analyze patterns of dangerous driving and aggressive driving. Specifically, features such as abnormal following distances, sudden speed changes, and frequent lane changes are extracted, and an abnormality score is calculated using a machine learning model.

[0869] Step 5:

[0870] Generating warning messages

[0871] Server: When the AI ​​engine detects abnormal driving, it generates a warning message. This message includes the type of dangerous driving detected and the actions the driver should take.

[0872] Step 6:

[0873] Sending a warning

[0874] Server: Sends generated warning messages to terminals in real time.

[0875] Step 7:

[0876] Warning display and audio notification

[0877] Terminal: Displays received warning messages to the driver and also provides voice notifications. For example, it displays a message such as, "There is a vehicle approaching dangerously close from behind. Change lanes to avoid it," and also provides voice notifications.

[0878] Step 8:

[0879] Cloud storage of data

[0880] Server: Uploads and stores data on detected abnormal operation to the cloud. This data includes video frames, date and time, location information, etc.

[0881] Step 9:

[0882] Presentation of reporting options

[0883] Terminal: Presents the driver with the option of whether or not to report to the police. A message such as "Do you want to report this?" is displayed on the screen.

[0884] Step 10:

[0885] Execution of the report

[0886] User: The driver selects the reporting option.

[0887] Device: When the reporting button is pressed, voice support is activated, and detailed information, including location and vehicle information, is sent to the police via voice and text.

[0888] This allows drivers to receive real-time warnings and quickly contact the police as needed, significantly improving safety.

[0889] (Example 1)

[0890] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0891] In recent years, accidents caused by dangerous driving and intimidating driving have been increasing, and there is a need to detect these driving behaviors in real time and respond quickly. However, current dashcams and driver assistance systems are insufficient in their ability to adequately detect dangerous driving behavior behind the vehicle and warn the driver. Therefore, a system is needed to ensure driver safety and respond immediately to abnormal driving.

[0892] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0893] In this invention, the server includes means for capturing rear-view video of the transport equipment using a rear camera, means for analyzing the captured rear-view video in real time using artificial intelligence to detect dangerous driving or intimidating driving, means for sending a warning message to the driver and providing advice on countermeasures for detected abnormal driving, auxiliary means for saving driver safety data to a storage device based on the analysis results, means for saving video data to an external storage device using a secure communication protocol, and voice support means for notifying an external organization at the driver's instruction. This enables the detection of dangerous driving behavior behind the vehicle in real time, prompting the driver to be warned and encouraging appropriate action, and furthermore, data on abnormal driving can be saved to the cloud and reported to an external organization as needed.

[0894] A "rear camera" is a video capture device used to capture images from the rear of a vehicle.

[0895] "Artificial intelligence" is a general term that includes machine learning algorithms and deep learning technologies used to analyze captured rear-view video footage and detect abnormal driving behavior.

[0896] "Dangerous driving" refers to driving behavior that is unstable or abnormal compared to normal driving conditions.

[0897] "Intimidating driving" refers to driving behavior that intentionally causes fear or anxiety to other drivers or pedestrians.

[0898] A "warning message" is notification information intended to prompt the driver to take action in response to detected abnormal driving behavior.

[0899] "Driver" refers to a person operating a vehicle.

[0900] A "storage device" refers to hardware or storage media used to store data.

[0901] "External storage devices" refer to external data storage media, such as cloud storage, used to save data outside of the local system.

[0902] "Auxiliary means" refers to additional functions or technologies used to operate the main function.

[0903] A "secure communication protocol" refers to technologies and standards for ensuring secure data communication. Specifically, this includes protocols such as HTTPS.

[0904] "Reporting" refers to the act of transmitting information about abnormal driving to an external organization when such behavior is detected.

[0905] "Voice support" refers to a function that uses voice to provide information to users or to report information to external organizations.

[0906] "External organizations" refer to organizations including the police and other public institutions.

[0907] Modes for carrying out the invention

[0908] The system according to the present invention is an integrated system that includes a rear camera, artificial intelligence-based video analysis, driver warnings, data storage in the cloud, and notification to an external organization. Specific embodiments of this system are described below.

[0909] Rear camera activation and video capture

[0910] When the system starts up, the terminal automatically activates the rear camera, which captures video footage from behind the transport equipment. This video data is encoded in real time and converted to a stream format (e.g., H.264).

[0911] Sending video data

[0912] The encoded video data is sent from the terminal to the server using a secure communication protocol (e.g., HTTPS). The terminal buffers the video data, divides it into appropriate packet sizes, encrypts it, and sends it to the server.

[0913] Preprocessing of video data

[0914] The server preprocesses the received video data. This preprocessing includes frame extraction, resizing, and normalization. The preprocessed data is then input into the artificial intelligence engine.

[0915] Analysis of video data

[0916] The server uses an artificial intelligence engine to analyze pre-processed video data. The analysis extracts features such as abnormal following distances, sudden speed changes, and frequent lane changes, and calculates an anomaly score using machine learning models (e.g., TensorFlow, PyTorch). This allows for the detection of dangerous driving and aggressive driving behavior.

[0917] Generating and sending warning messages

[0918] Based on the analysis results, the server generates a warning message if abnormal operation is detected. The generated warning message is sent to the terminal in real time.

[0919] Driver warning notification

[0920] The device notifies the driver of any received warning messages. The notification is displayed on the screen and also alerts the driver with an audio alert. For example, a message such as "There is a vehicle approaching dangerously close from behind. Change lanes to avoid it" will be displayed, and the same information will be announced by voice.

[0921] Cloud storage of abnormal driving data

[0922] The server uploads and stores the detected abnormal operation data to the cloud. This data includes video frames and metadata (date and time, location information, etc.) at the time of the anomaly detection. The data is stored using cloud storage services (e.g., AWS S3, Google Cloud Storage).

[0923] Driver's report to the police

[0924] If the driver deems the situation serious, they will be given the option to call the police through the terminal's interface. If the driver selects the calling option, simply pressing the call button will activate voice support, and pre-configured information (location, license plate number, etc.) will be sent to the police in both voice and text format.

[0925] Specific example

[0926] For example, if a vehicle approaches abnormally closely from behind while driving on a highway, the rear camera captures the image behind the vehicle, and the artificial intelligence engine detects the abnormal proximity. The server notifies the terminal that an abnormal proximity has been detected, and the terminal displays the message, "There is a vehicle approaching abnormally closely from behind. Change lanes to avoid it," and also notifies the driver of the same message by voice. This situation is saved to the cloud and can be reviewed later. If the driver presses the report button, the police are automatically notified, and a notification is sent stating, "This vehicle is currently traveling at point XX, and is experiencing aggressive driving from behind. The vehicle number is XX-1234."

[0927] Examples of prompts for generative AI models

[0928] For example, analysis using a generated AI model can be performed by using the following prompt statement.

[0929] You are developing an advanced driver assistance system. This system integrates a rear camera, AI-powered video analysis, driver warnings, cloud data storage, and notification to external organizations. Explain the program's processes and provide specific examples for the following scenarios.

[0930] scenario:

[0931] 1. Please explain how the device activates the rear camera and captures rearward video when the system starts up.

[0932] 2. Explain how the artificial intelligence engine analyzes video data to detect dangerous or aggressive driving.

[0933] 3. Please explain in detail the mechanism for displaying and notifying drivers of warnings.

[0934] 4. Please explain the data storage and reporting process for abnormal operation to external organizations.

[0935] Specific example:

[0936] Describe a scenario in which a vehicle approaches unusually closely from behind while driving on a highway, and specifically explain how each process functions in that situation.

[0937] By inputting this prompt into the AI ​​model, it is possible to generate a detailed explanation of the system's operation and various functions.

[0938] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0939] System program processing flow

[0940] Step 1:

[0941] The device automatically activates the rear camera when the system starts up.

[0942] Specific operation: The terminal powers on the rear camera via internal control software and instructs the camera to begin capturing rearward video. The rear camera continuously captures rearward video and generates encoded stream data (e.g., in H.264 format).

[0943] Input: System power signal.

[0944] Output: Rear camera capture started, encoded video data.

[0945] Step 2:

[0946] The terminal sends the encoded video data to the server using a secure communication protocol (e.g., HTTPS).

[0947] Specific operation: The terminal buffers the video data, divides it into packets, and encrypts them. It then sends them to the server via the network stack.

[0948] Input: Encoded video data from the rear camera.

[0949] Output: Encrypted video data sent to the server.

[0950] Step 3:

[0951] The server preprocesses the received video data.

[0952] Specific operation: The server receives data, decodes the packets, and divides them into frames. Each frame is resized and normalized. After that, the pre-processed data is input into the AI ​​engine.

[0953] Input: Encrypted video data packets.

[0954] Output: Preprocessed video data (resized and normalized frames).

[0955] Step 4:

[0956] The server analyzes the pre-processed video data.

[0957] Specific operation: The AI ​​engine extracts features to detect abnormalities such as insufficient following distance, sudden speed changes, and frequent lane changes. A machine learning model (e.g., TensorFlow, PyTorch) is used to calculate an anomaly score and determine whether the driving is dangerous or aggressive.

[0958] Input: Pre-processed video data.

[0959] Output: Abnormal operation score and judgment result.

[0960] Step 5:

[0961] Based on the analysis results, the server generates a warning message if abnormal operation is detected.

[0962] Specific operation: When abnormal operation is detected, the server generates a warning in text format and creates a warning message for the driver.

[0963] Input: Score and judgment result for abnormal operation.

[0964] Output: Warning message.

[0965] Step 6:

[0966] The server sends the generated warning message to the terminal.

[0967] Specific operation: The server sends a warning message to the terminal in real time.

[0968] Input: Warning message.

[0969] Output: Send to the terminal.

[0970] Step 7:

[0971] The terminal notifies the driver of the received warning message.

[0972] Specific operation: The device displays a warning message on its screen and notifies the driver of the same content via an audio alert. For example, it might inform the driver, "There is a vehicle approaching dangerously close from behind. Change lanes to avoid it."

[0973] Input: Warning message.

[0974] Output: Display, audio alert.

[0975] Step 8:

[0976] The server uploads and stores the detected abnormal operation data to the cloud.

[0977] Specific operation: The server collects video frames and metadata (date, time, location information, etc.) from the time of anomaly detection into storage data and uploads it to a cloud storage service (e.g., AWS S3, Google Cloud Storage).

[0978] Input: Anomaly detection data (video frames, metadata).

[0979] Output: Data stored in cloud storage.

[0980] Step 9:

[0981] If the driver deems the situation serious, they will notify the police via the device.

[0982] Specific operation: When the driver presses the call button, the device activates its voice assistance function and sends pre-configured information (location information, license plate number, etc.) to the police in both voice and text format.

[0983] Input: Driver presses the call button, saved information.

[0984] Output: Automatic notification to the police (voice and text format).

[0985] (Application Example 1)

[0986] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0987] In recent years, aggressive driving and dangerous driving have become a social problem, making prevention and response to these issues urgent. Furthermore, in order to ensure the safety of individuals, including pedestrians, there is a need for a system that can detect abnormal behavior in real time and provide appropriate warnings and countermeasures. The present invention aims to provide a system that ensures the safety of a wider range of individuals by utilizing not only automobiles but also smart glasses.

[0988] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0989] In this invention, the server includes means for capturing rear-view video of a vehicle using a rear camera, means for analyzing the captured rear-view video in real time using AI to detect dangerous driving and aggressive driving, means for sending a warning message to the driver and providing advice on countermeasures for detected abnormal driving, means for uploading and storing the detected abnormal driving data to the cloud, means for providing voice support for notifying external organizations such as the police at the driver's instruction, means for capturing rear-view video using a rear camera mounted on smart glasses to detect approaching abnormal behavior, and means for providing the user with a visual and audible warning message when approaching abnormal behavior is detected. This enables universal safety assurance not only for vehicle drivers but also for pedestrians.

[0990] A "rear camera" is a camera device installed in cars or smart glasses that captures images of the area behind the vehicle.

[0991] "AI" is an abbreviation for artificial intelligence, a technology that uses machine learning and deep learning to analyze data and detect patterns and anomalies.

[0992] "Dangerous driving" refers to driving behaviors that endanger other road users, such as sudden changes in speed or frequent lane changes.

[0993] "Aggressive driving" refers to driving behavior that intentionally causes fear to other drivers, such as overtaking other vehicles and slamming on the brakes, or driving dangerously close to them.

[0994] A "warning message" is a message that provides a warning or evacuation order to the driver or user when dangerous driving or aggressive driving is detected.

[0995] "Cloud" refers to external data storage and computing resources provided via the internet.

[0996] "Voice support means" refers to functions that use voice to provide information to users or to report to external organizations.

[0997] "Smart glasses" are glasses-type wearable devices that have computer functions and assist the user's visual information.

[0998] "Abnormal behavior" refers to actions that deviate from normal behavioral patterns and may pose a danger to the user or others.

[0999] "Means of providing warning messages visually and audibly" refers to functions that use displays and speakers to convey warning information to users visually and audibly.

[1000] The system according to the present invention is an integrated system for ensuring the safety of automobile drivers and pedestrians. This system includes a rear camera, AI-based video analysis, warning messages to the user, data storage to the cloud, and notification to the police, etc. Embodiments of this system will be described below.

[1001] System hardware and software configuration

[1002] Hardware:

[1003] Smart glasses (e.g., typical wearable devices)

[1004] Rear camera (mounted in cars and smart glasses)

[1005] Speaker (for voice notifications)

[1006] GPS module (for acquiring location information)

[1007] software:

[1008] Video capture API (for acquiring video data)

[1009] AI models (for video analysis, e.g., TensorFlow, PyTorch)

[1010] Cloud storage (for data storage, e.g., AWS S3)

[1011] Secure communication protocols (e.g., HTTPS)

[1012] Reporting API (e.g. Twilio, Firebase)

[1013] System Operation Overview

[1014] 1. Startup and video capture

[1015] When smart glasses or a car are activated, the rear camera automatically activates and captures footage of the area behind the vehicle. The video data is encoded in real time.

[1016] 2. Video data preprocessing

[1017] The server receives the video data and performs preprocessing. This includes frame extraction, resizing, and normalization.

[1018] 3. AI analysis

[1019] Pre-processed video data is analyzed by an AI engine. The AI ​​engine detects abnormal behavior (such as dangerous driving, road rage, or suspicious individuals approaching).

[1020] 4. Generate and notify warning messages

[1021] When abnormal behavior is detected, the server generates a warning message and notifies the user through smart glasses or the car's display and speakers. For example, the message might say, "There is a vehicle approaching you dangerously closely from behind. Change lanes to avoid it."

[1022] 5. Data storage and reporting

[1023] Data on detected abnormal behavior is stored in the cloud. This includes location information, date and time, and video frames. When a user presses the report button, voice support is activated, and location information and video data are automatically sent to the police.

[1024] Specific example

[1025] For example, if a pedestrian wearing smart glasses is walking along a school route and a suspicious person approaches from behind, the rear camera captures the situation. The AI ​​engine detects the abnormal approach and displays a message saying, "Someone is approaching you from behind. Please move to a safe place," and also provides an audio notification with the same message. If the user presses the emergency call button, a report is sent to the police stating, "I am currently walking at location XX, and a suspicious person is approaching from behind."

[1026] Example of a prompt

[1027] Please create a smart glasses application that works as follows: Use the smart glasses' rear camera to capture rearward video, and have AI detect abnormal behavior (following, sudden approach). Implement a function to provide visual and audible warnings upon detection, and automatically notify the police in emergencies.

[1028] In summary, this system provides a technical solution to ensure the safety of vehicle drivers and pedestrians in real time and to respond quickly and appropriately.

[1029] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1030] Step 1:

[1031] Smart glasses or a car terminal are activated. The terminal automatically activates the rear camera and captures rearward video in real time. The captured video data is encoded and sent to the server using a secure communication protocol (e.g., HTTPS). The input is video data from the rear camera, and the output is encoded video data.

[1032] Step 2:

[1033] The server preprocesses the received video data. Specifically, it extracts frames, resizes them, and normalizes them. This preprocessing prepares the video for efficient analysis by the AI ​​engine. The input is encoded video data, and the output is preprocessed video data.

[1034] Step 3:

[1035] The server's AI engine analyzes pre-processed video data. The AI ​​engine uses a machine learning model to detect abnormal behavior (such as dangerous driving, road rage, or suspicious individuals approaching). Specifically, it extracts features such as distance between vehicles, speed changes, and lane change frequency, and calculates an abnormality score. The input is pre-processed video data, and the output is the result of detecting abnormal behavior.

[1036] Step 4:

[1037] The server generates a warning message if abnormal behavior is detected. The warning message is provided visually and audibly. The generated warning message is sent to the terminal using a secure communication protocol. The input is the result of the detected abnormal behavior, and the output is the warning message.

[1038] Step 5:

[1039] The device notifies the user of received warning messages. Specifically, it conveys warning messages visually and audibly using the smart glasses' display and speaker. For example, it might say, "There is a vehicle approaching you dangerously close from behind. Change lanes to avoid it." The input is the warning message, and the output is the notification to the user.

[1040] Step 6:

[1041] The server saves data on detected abnormal behavior to the cloud. The saved data includes date and time, location information, and video frames. Cloud storage (e.g., AWS S3) is used for data storage. The input is the detected abnormal behavior and associated metadata, and the output is the data stored in the cloud.

[1042] Step 7:

[1043] When a user presses the emergency call button, the device's voice support function is activated. Pre-configured location information and video data are transmitted to the police via voice and text. The input consists of the user's call instructions and related data (location information, video data), and the output is a notification to the police.

[1044] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1045] Modes for carrying out the invention

[1046] The system according to the present invention is an integrated system that combines a rear camera, AI-based video analysis, driver warnings, data storage to the cloud, and notification to the police, with an emotion engine that recognizes the user's emotions. The processing of this system's program will be described below in natural language.

[1047] System program processing

[1048] When the system starts up, the terminal activates the rear camera and continuously captures video footage of the area behind the vehicle. This video data is encoded and sent to the server, which receives the video data and performs preprocessing for analysis by the AI ​​engine. The AI ​​engine analyzes the preprocessed data to detect dangerous driving and aggressive driving.

[1049] Based on the analysis results, the server generates a warning message if abnormal operation is detected. The message is sent to the terminal in real time, and the terminal notifies the driver of the received message. Furthermore, data on abnormal operation is stored in the cloud and can be reviewed later.

[1050] The present invention also includes a system for recognizing the driver's emotions. Specifically, a camera mounted on the terminal captures an image of the driver's face and transmits this data to a server. The server uses an emotion engine to analyze the image of the driver's face and recognize their emotional state. For example, it can detect states such as tension, stress, and relaxation.

[1051] Based on recognized emotions, the content of warning messages and advice is adjusted. For example, if a driver is stressed, more detailed and considerate advice may be provided. Furthermore, emotional states are included as additional information in data stored in the cloud, allowing for later analysis.

[1052] Furthermore, it includes a feature that automatically prompts a police alert if the driver is experiencing particular tension or stress. This feature allows for quick external assistance to be sought if the driver's emotional state is deemed to be a factor detrimental to safe driving.

[1053] Specific example

[1054] For example, suppose you are driving on a highway and a vehicle approaches you dangerously closely from behind. In this case, the rear camera captures the image behind you, and the server's AI engine detects the dangerous proximity. The server generates a warning message such as, "There is a vehicle approaching you dangerously closely from behind. Change lanes to avoid it," and sends it to your device.

[1055] Simultaneously, a camera is operating to capture video of the driver's face, and this video data is sent to a server. The server's emotion engine analyzes this data to recognize that the driver is stressed. Based on this, it checks whether the warning message is considerate of the driver's emotional state and adjusts it as needed.

[1056] Furthermore, the details of this abnormal driving and the driver's emotional state are saved to the cloud. Also, if the driver presses the report button, the system sends information to the police stating, "This vehicle is currently traveling at location XX and is experiencing aggressive driving from behind. The vehicle's license plate number is XX-1234. The driver is extremely distressed."

[1057] In this way, this system not only detects dangerous driving but also provides comprehensive safety measures that take into account the driver's emotional state.

[1058] The following describes the processing flow.

[1059] Modes for carrying out the invention

[1060] System program processing

[1061] Step 1:

[1062] Rear camera activation and video capture start

[1063] Terminal: When the vehicle engine starts, the rear camera automatically activates. After activation, the rear camera begins capturing rearward video and generates video data frame by frame.

[1064] Step 2:

[1065] Sending video data

[1066] Terminal: Encodes captured video frames at high speed and sends them to the server using a secure communication protocol.

[1067] Step 3:

[1068] Receiving and preprocessing video data

[1069] Server: Receives video data, stores it in a buffer, and performs preprocessing to prepare it for input into the AI ​​engine. This preprocessing includes frame extraction, resizing, and normalization.

[1070] Step 4:

[1071] AI analysis

[1072] Server: Pre-processed video data is input into the AI ​​engine to analyze patterns of dangerous driving and aggressive driving. Specifically, features such as abnormal following distances, sudden speed changes, and frequent lane changes are extracted, and an abnormality score is calculated using a machine learning model.

[1073] Step 5:

[1074] Generating warning messages

[1075] Server: When the AI ​​engine detects abnormal driving, it generates a warning message. This message includes the type of dangerous driving detected and the actions the driver should take. For example, it might say, "There is a vehicle approaching dangerously closely from behind. Change lanes to avoid it."

[1076] Step 6:

[1077] Sending a warning

[1078] Server: Sends generated warning messages to terminals in real time.

[1079] Step 7:

[1080] Warning display and audio notification

[1081] Terminal: Displays received warning messages to the driver and also provides voice notifications. For example, a message is displayed on the screen, and a voice notification says, "There is a vehicle approaching you dangerously close from behind. Change lanes to avoid it."

[1082] Step 8:

[1083] Cloud storage of data

[1084] Server: Uploads and stores data on detected abnormal operation to the cloud. This data includes video frames, date and time, location information, etc.

[1085] Step 9:

[1086] Video capture of the driver's face

[1087] Terminal: The camera operates to capture video of the driver's face. This video data is encoded and sent to the server.

[1088] Step 10:

[1089] Emotion recognition by an emotion engine

[1090] Server: The server analyzes the received video of the driver's face using an emotion engine to recognize their emotional state. Specifically, it analyzes the driver's facial expressions and condition to determine emotions such as "tension," "stress," and "relaxation."

[1091] Step 11:

[1092] Adjusting emotion-based warning messages

[1093] Server: Based on recognized emotions, it adjusts the content of warning messages and corrective advice. For example, if the driver is stressed, it provides more polite and detailed advice.

[1094] Step 12:

[1095] Cloud storage of emotional data

[1096] Server: The recognized emotional states are added to data stored in the cloud, making them available for later analysis.

[1097] Step 13:

[1098] Presentation of reporting options

[1099] Terminal: Displays an option on the screen for the driver to report the incident to the police. For example, it might display the option, "Do you want to report this?"

[1100] Step 14:

[1101] Execution of the report

[1102] User: The driver selects the reporting option.

[1103] Terminal: When the reporting button is pressed, voice support is activated, and detailed information, including location, vehicle information, and emotional state, is sent to the police via voice and text. For example, information such as, "This vehicle is currently traveling at location XX and is experiencing aggressive driving from behind. The vehicle number is XX-1234. The driver is extremely agitated," will be sent.

[1104] As described above, this system provides comprehensive safety measures that take into account not only the detection of dangerous driving and aggressive driving, but also the driver's emotional state.

[1105] (Example 2)

[1106] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1107] Conventional automotive safety systems have the ability to detect dangerous driving and aggressive driving, but they cannot take into account the driver's emotional state, which can lead to increased stress and tension for the driver. Furthermore, while data on abnormal driving is stored, the lack of sufficient specific advice on countermeasures limits the potential for improving safety.

[1108] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for capturing rear-view video of the vehicle using a rear camera, means for analyzing the captured rear-view video in real time using AI to detect dangerous driving and aggressive driving, means for sending a warning message to the driver regarding the detected abnormal driving and providing advice on countermeasures, means for uploading and saving the detected abnormal driving data to the cloud, means for capturing video of the driver's face and performing emotion analysis, and means for adjusting the warning message based on the observed emotions of the driver. This makes it possible to not only detect dangerous driving and aggressive driving in real time, but also to implement comprehensive safety measures that take into account the driver's emotional state.

[1109] A "rear camera" is a camera mounted on the rear of a vehicle, used to capture images of the area behind the vehicle.

[1110] "AI" stands for Artificial Intelligence, and refers to an algorithm or system that automatically performs specific tasks.

[1111] "Dangerous driving" refers to driving behavior that violates the Road Traffic Act and endangers safety.

[1112] "Aggressive driving" refers to intentionally engaging in intimidating driving behavior towards other vehicles.

[1113] A "warning message" is a message that the system issues to the driver to warn them about dangerous driving or aggressive driving.

[1114] "Safety advice" refers to information that provides drivers with specific instructions and advice to drive safely.

[1115] "Cloud" refers to a group of servers that provide data storage and processing functions via the internet.

[1116] "Emotional analysis" is a technology that evaluates a driver's emotional state based on facial expressions and other factors using video data of their face.

[1117] "Reporting methods" refer to a function that, in response to the driver's instructions, allows for reporting abnormal driving situations to external organizations such as the police using voice or text.

[1118] "Preprocessing means" refers to techniques that process video data, such as noise reduction and frame selection, before analysis by AI.

[1119] Modes for carrying out the invention

[1120] The system according to the present invention is an integrated safety driving support system that includes a rear camera, an AI engine, an emotion analysis engine, a driver warning message function, data storage to the cloud, and an automatic notification function to the police.

[1121] Hardware and software configuration:

[1122] The rear camera is mounted on the back of the car and continuously captures images of the area behind it.

[1123] The device has the function of encoding video data from the rear camera and sending it to the server.

[1124] The server has hardware and software to preprocess the received video data and analyze it with an AI engine.

[1125] The AI ​​engine includes algorithms that detect dangerous driving and aggressive driving in real time.

[1126] Another camera on the device captures video of the driver's face, providing data to recognize the driver's emotional state.

[1127] The emotion analysis engine is built into the server and analyzes the transmitted facial video data to recognize the driver's emotional state (tension, stress, relaxation, etc.).

[1128] The cloud server securely stores data such as abnormal driving behavior and the driver's emotional state, allowing for later review and analysis.

[1129] The warning message function sends messages generated on the server to the terminal in real time, and the terminal notifies the driver.

[1130] The automatic reporting function has a means of reporting to an external organization (e.g., the police) based on the driver's instructions.

[1131] Specific examples of how the system works:

[1132] Suppose a vehicle is detected approaching abnormally closely from behind while driving on a highway. The rear camera captures the image behind the vehicle, the video data is encoded on the terminal, and then sent to the server.

[1133] The server uses an AI engine to detect abnormal proximity. It generates a warning message such as, "There is a vehicle approaching abnormally closely from behind. Change lanes to avoid it," and sends it to the terminal.

[1134] Simultaneously, a camera capturing the driver's face is also operating, and the video data is sent to a server. An emotion analysis engine analyzes this data and recognizes that the driver is feeling stressed.

[1135] Based on this understanding, the server adjusts the content of the warning message. For example, it might add advice such as, "Stay calm and check behind you."

[1136] All data (including details of abnormal driving and the driver's emotional state) is stored on a cloud server.

[1137] When the driver presses the call button, the automatic call function sends a message to the police saying, "This vehicle is currently traveling at location XX, and is experiencing aggressive driving from behind. The vehicle's license plate number is XX-1234. The driver is extremely distressed."

[1138] Example of a prompt:

[1139] 1. "Please describe in detail the process from detecting abnormal operation to sending a warning message in conjunction with the rear camera."

[1140] 2. "Please explain specific methods for adjusting warning messages using driver facial recognition and emotion analysis."

[1141] Thus, the system according to the present invention not only detects dangerous driving and aggressive driving in real time, but also provides comprehensive safety measures that take into account the driver's emotional state.

[1142] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1143] Step 1: System startup

[1144] The terminal prepares to activate the rear camera simultaneously with the system startup. The input is the system startup signal, and the output is the completion of the rear camera startup.

[1145] Specifically, when the device's power button is pressed, the internal software sends a command to activate the rear camera.

[1146] Step 2: Video capture using the rear camera

[1147] The device activates the rear camera and continuously captures rearward video. The input is real-time video data, and the output is encoded rearward video data.

[1148] Specifically, the rear camera captures the rear view frame by frame and sends it to the terminal. The terminal then encodes this data.

[1149] Step 3: Encode and transmit video data

[1150] The terminal encodes the captured video data. The input is raw video data, and the output is encoded data. After encoding, this data is sent to the server.

[1151] Specifically, the terminal compresses the video data into a format such as MPEG and sends it to the server using a communication protocol.

[1152] Step 4: Video analysis using an AI engine

[1153] The server preprocesses the received encoded video data. The input is encoded video data, and the output is preprocessed data. Preprocessing such as noise reduction and frame selection is performed.

[1154] The server's AI engine analyzes pre-processed data to detect dangerous driving and aggressive driving. The input for the analysis is pre-processed data, and the output is the detection result of abnormal driving.

[1155] Step 5: Generate and send the warning message.

[1156] If abnormal operation is detected, the server generates a warning message. The input is the result of the abnormal operation detection, and the output is the generated warning message. Next, this message is sent to the terminal in real time, and the terminal notifies the driver.

[1157] Specifically, the server creates a warning message based on a template and sends it to the terminal using the TCP / IP protocol. The terminal then displays an audio alert or a pop-up on the screen.

[1158] Step 6: Driver's Emotion Recognition

[1159] The camera on the device captures video of the driver's face. The input is video data of the driver's face, and the output is encoded video data of the face. This is sent to the server.

[1160] The server's emotion engine analyzes facial video data to recognize the driver's emotional state. The input is encoded facial video data, and the output is the result of the emotion analysis.

[1161] Step 7: Adjusting Emotion-Based Warning Messages

[1162] The server adjusts warning messages to be more detailed and thoughtful if the driver is stressed. The input is the result of sentiment analysis, and the output is the adjusted warning message.

[1163] Specifically, the server selects a warning message template and adjusts the content of the advice based on the sentiment analysis results.

[1164] Step 8: Save data to the cloud

[1165] The server stores data on abnormal operation and sentiment analysis on a cloud server. The input is the status of abnormal operation and the results of sentiment analysis, and the output is the data stored in the cloud.

[1166] Specifically, the server uploads data to the cloud storage service via the database.

[1167] Step 9: Automatically report to the police

[1168] When the driver presses the emergency call button, the server automatically initiates the process of notifying the police. The input is the driver's emergency call signal, and the output is the emergency call message sent to the police.

[1169] Specifically, the server generates a notification message containing location and vehicle information, and sends it to the police using the appropriate communication protocol.

[1170] (Application Example 2)

[1171] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1172] Conventional in-vehicle systems were specialized in detecting dangerous driving behavior and abnormal approaches from the rear, but lacked consideration for the driver's emotional state. This made it difficult to respond appropriately when the driver felt tension or stress, potentially hindering safe driving. Furthermore, warnings and advice after detecting abnormal driving were uniform, failing to provide measures tailored to the driver's emotional state. As a result, safety and driver stress reduction were not adequately achieved.

[1173] In Application Example 2, the identification processing by the identification processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for capturing rear-view video of the vehicle using a rear camera, means for analyzing the captured rear-view video in real time using AI to detect dangerous driving and aggressive driving, means for sending a warning message to the driver and providing advice on countermeasures for detected abnormal driving, means for uploading and saving the detected abnormal driving data to the cloud, voice support means for notifying external organizations such as the police at the driver's instruction, means for capturing video of the driver's face and analyzing the driver's emotional state using an emotion recognition engine, and means for adjusting the content of warning messages and countermeasure advice based on the driver's emotional state. This enables appropriate countermeasures according to the driver's emotional state, thereby improving safety and reducing stress.

[1174] A "rear camera" is a camera device used to capture images of the area behind a vehicle.

[1175] The "AI engine" is an artificial intelligence technology that analyzes captured video in real time to detect abnormal driving, aggressive driving, and the driver's emotional state.

[1176] "Dangerous driving" refers to any driving behavior that may endanger other vehicles or pedestrians.

[1177] "Aggressive driving" refers to driving behavior that is harassing or endangering to the driver, such as a vehicle following closely behind another vehicle.

[1178] A "warning message" is a message or alert designed to alert the driver to detected abnormal driving behavior.

[1179] "Countermeasure advice" refers to information that presents recommended driving actions and points to note to the driver in response to detected abnormal driving behavior.

[1180] "Cloud" refers to data storage and processing services provided via the internet, and in this context, it refers to remote storage for saving data on detected abnormal operation.

[1181] "Voice support means" refers to a function that provides voice support when the driver instructs an external organization, such as the police, to make a report.

[1182] The "emotion recognition engine" is a technology that analyzes the driver's facial image to recognize their emotional state.

[1183] "Emotional state" refers to the driver's mental and psychological state, including states such as tension, stress, and relaxation.

[1184] "Video data preprocessing" is a technique that transforms video data into a format that is easily accessible for AI analysis.

[1185] "External organizations" refer to public institutions such as the police and other support organizations.

[1186] "Abnormal driving data" refers to all data that, as a result of analyzing video footage acquired from rear cameras and other sources, is determined to indicate abnormal driving.

[1187]

[1188] The system according to the present invention is an integrated system installed in an autonomous vehicle. This system captures rearward images using a rear camera and analyzes the data with an AI engine to detect dangerous driving and aggressive driving. In addition, a camera that captures facial images and an emotion recognition engine are used to recognize the driver's emotional state.

[1189] Specific system configuration

[1190] 1. Hardware configuration:

[1191] Rear camera: A camera used to capture images of the rear of a vehicle.

[1192] Emotion Recognition Camera: A camera that captures video of the driver's face.

[1193] In-car display: A display that shows warning messages and advice to the driver. This can also be replaced by the display on a smartphone or smart glasses.

[1194] 2. Software configuration:

[1195] AI Engine: Uses platforms such as TensorFlow to analyze video from the rear camera and emotion recognition camera.

[1196] Cloud storage: Remote storage for saving data on detected abnormal operation. Amazon AWS or Google Cloud Storage are possible options.

[1197] Communication software: A tool for uploading abnormal driving data to the cloud and sending HTTP requests to external organizations such as the police to report it. Uses libraries such as Python's Requests library.

[1198] System Operation Description

[1199] When the system starts up, the rear camera continuously captures rearward video, and this video data is sent to the AI ​​engine in real time. After the server receives the video data and performs preprocessing, the AI ​​engine detects dangerous driving and aggressive driving.

[1200] If abnormal driving is detected, the server generates a warning message and displays it in real time on the in-vehicle display. For example, it might notify the driver with a warning message such as, "An abnormally close approach is coming from behind. Please be careful." This data is also uploaded to cloud storage and can be reviewed later.

[1201] Furthermore, an emotion-recognition camera captures the driver's face, and the video data is sent to a server. The server's emotion recognition engine analyzes this data to recognize the driver's emotional state (e.g., tension or stress). Based on the recognized emotion, the content of warning messages and corrective advice is adjusted. For example, if the driver is tense, more polite and detailed advice can be provided.

[1202] Specific usage examples

[1203] For example, suppose a vehicle approaches abnormally closely from behind while driving on a highway. In this case, the rear camera captures the image behind the vehicle, and the server's AI engine detects the abnormal approach. The server generates a warning message such as, "There is a vehicle approaching abnormally closely from behind. Change lanes to avoid it," and sends it to the in-car display. At the same time, a camera is also operating to capture the driver's face, and this video data is sent to the server. The server's emotion recognition engine analyzes this data and recognizes that the driver is stressed. Based on this, it checks whether the warning message is considerate of the driver's emotional state and adjusts it as needed. Furthermore, the details of this abnormal driving and the driver's emotional state are stored in the cloud.

[1204] Example of a prompt

[1205] Regarding the rear camera frame: "The system analyzes the rear camera footage to detect abnormally close approaches from behind."

[1206] In the case of a facial emotion recognition camera: "It analyzes the driver's facial image and recognizes emotional states such as tension and stress."

[1207] This allows for appropriate measures to be taken in response to the driver's emotional state, thereby improving safety and reducing stress.

[1208] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1209] Step 1:

[1210] The terminal activates the rear camera and captures the rear view of the vehicle. It receives live video from the rear camera as input, encodes the video data, and sends it to the server. Specifically, it captures the rear camera's video input in real time, compresses the data, and uploads it to the server via the network.

[1211] Step 2:

[1212] The server preprocesses the received video data. It receives encoded video data as input and decodes and preprocesses it into a format that is easy for the AI ​​engine to analyze. Specifically, it decodes the video data and adjusts the frame rate and removes noise as needed.

[1213] Step 3:

[1214] The server analyzes pre-processed video data using an AI engine. It receives pre-processed video data as input and uses an AI model to detect dangerous driving and aggressive driving. Specifically, it inputs the prompt message "Analyze the rear camera footage and detect abnormally close driving from behind" into the generating AI model and obtains the analysis results.

[1215] Step 4:

[1216] The server generates warning messages for detected abnormal operation. It receives analysis results as input and uses templates to create appropriate warning messages. Specifically, depending on the type of abnormal operation, it generates messages such as "An abnormal approach is coming from behind. Please be careful," and sends them to the terminal.

[1217] Step 5:

[1218] The terminal notifies the driver of received warning messages. It receives warning messages from the server as input and displays them on the in-car display or smart glasses. Specifically, it notifies the driver of the warning messages visually and audibly.

[1219] Step 6:

[1220] The device's emotion recognition camera captures the driver's facial image. It receives live video of the driver's face as input, encodes the video data, and sends it to the server. Specifically, the emotion recognition camera captures the driver's facial image in real time, compresses the data, and uploads it to the server.

[1221] Step 7:

[1222] The server preprocesses the received facial video data. It receives encoded facial video data as input and decodes and preprocesses it into a format that is easy for the emotion recognition engine to analyze. Specifically, it decodes the facial video data and adjusts the frame rate and removes noise as needed.

[1223] Step 8:

[1224] The server analyzes pre-processed facial video data using an emotion recognition engine. It receives pre-processed facial video data as input and uses an AI model to recognize the driver's emotional state. Specifically, it inputs the prompt message "Analyze the driver's facial video and recognize emotional states such as tension and stress" into the generating AI model and obtains the analysis results.

[1225] Step 9:

[1226] The server adjusts warning messages and advice based on the driver's emotional state. It receives the result of emotion recognition as input and personalizes warning messages and advice based on that result. For example, if the driver is stressed, the message might be adjusted to something like, "There is a vehicle approaching dangerously close from behind. Please remain calm and take appropriate action."

[1227] Step 10:

[1228] The server stores abnormal driving data and driver emotional states in the cloud. It receives abnormal driving and emotional state data as input and uploads it to cloud storage. Specifically, it converts the data into an appropriate format and sends it to the cloud storage service for storage.

[1229] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1230] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1231] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1232] [Fourth Embodiment]

[1233] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1234] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1235] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1236] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1237] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1238] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1239] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1240] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1241] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1242] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1243] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1244] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1245] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1246] Modes for carrying out the invention

[1247] The system according to the present invention is an integrated system that includes a rear camera, AI-based video analysis, driver warnings, data storage in the cloud, and notification to the police, etc. The processing of the program of this system will be described below in natural language.

[1248] System program processing

[1249] Upon system startup, the terminal automatically activates the rear camera and continuously captures rearward video. This video data is encoded and sent to the server using a secure communication protocol. The server receives the transmitted video data and performs preprocessing for input into the AI ​​engine. This preprocessing includes frame extraction, resizing, and normalization.

[1250] The AI ​​engine analyzes pre-processed video data to detect dangerous driving and aggressive driving. The analysis extracts features such as abnormal following distances, sudden speed changes, and frequent lane changes, and uses a machine learning model to calculate an anomaly score. Based on these analysis results, the server generates a warning message if abnormal driving is detected.

[1251] The generated warning message is sent to the terminal in real time. The terminal displays the received warning message to the driver and also provides an audio notification. For example, the warning message might read, "There is a vehicle approaching dangerously close from behind. Change lanes to avoid it," and the same message might be announced aloud.

[1252] Furthermore, the data of detected abnormal operation is uploaded to and stored in the cloud. This includes video frames and metadata (date, time, location information, etc.) from the time the abnormality was detected. This makes it possible to review the details of the detected abnormal operation at a later date.

[1253] If the driver deems the situation serious, they are given the option to report it to the police through the terminal's interface. When the driver selects the reporting option, simply pressing the report button activates voice support, and pre-configured information (such as location and license plate number) is sent to the police via voice and text.

[1254] Specific example

[1255] For example, suppose a vehicle approaches abnormally closely from behind while driving on a highway. In this case, the rear camera captures the image behind the vehicle, and the AI ​​engine detects the abnormal proximity. The server notifies the terminal that an abnormal proximity has been detected, and the terminal displays the message, "There is a vehicle approaching abnormally closely from behind. Change lanes to avoid it," and also notifies the driver of the same message by voice. This situation is saved to the cloud and can be reviewed later. If the driver presses the report button, the police are automatically notified, and a notification is sent stating, "This vehicle is currently traveling at point XX, and is experiencing aggressive driving from behind. The vehicle number is XX-1234."

[1256] In this way, the system ensures driver safety in real time and provides an environment that allows for a rapid and appropriate response to abnormal situations.

[1257] The following describes the processing flow.

[1258] Program processing steps

[1259] Step 1:

[1260] Rear camera activation and video capture start

[1261] Terminal: When the vehicle engine starts, the rear camera automatically activates. After activation, the rear camera begins capturing rearward video and generates video data frame by frame.

[1262] Step 2:

[1263] Sending video data

[1264] Terminal: Encodes captured video frames at high speed and sends them to the server using a secure communication protocol.

[1265] Step 3:

[1266] Receiving and preprocessing video data

[1267] Server: Receives video data, stores it in a buffer, and performs preprocessing to prepare it for input into the AI ​​engine. This preprocessing includes frame extraction, resizing, and normalization.

[1268] Step 4:

[1269] AI analysis

[1270] Server: Pre-processed video data is input into the AI ​​engine to analyze patterns of dangerous driving and aggressive driving. Specifically, features such as abnormal following distances, sudden speed changes, and frequent lane changes are extracted, and an abnormality score is calculated using a machine learning model.

[1271] Step 5:

[1272] Generating warning messages

[1273] Server: When the AI ​​engine detects abnormal driving, it generates a warning message. This message includes the type of dangerous driving detected and the actions the driver should take.

[1274] Step 6:

[1275] Sending a warning

[1276] Server: Sends generated warning messages to terminals in real time.

[1277] Step 7:

[1278] Warning display and audio notification

[1279] Terminal: Displays received warning messages to the driver and also provides voice notifications. For example, it displays a message such as, "There is a vehicle approaching dangerously close from behind. Change lanes to avoid it," and also provides voice notifications.

[1280] Step 8:

[1281] Cloud storage of data

[1282] Server: Uploads and stores data on detected abnormal operation to the cloud. This data includes video frames, date and time, location information, etc.

[1283] Step 9:

[1284] Presentation of reporting options

[1285] Terminal: Presents the driver with the option of whether or not to report to the police. A message such as "Do you want to report this?" is displayed on the screen.

[1286] Step 10:

[1287] Execution of the report

[1288] User: The driver selects the reporting option.

[1289] Device: When the reporting button is pressed, voice support is activated, and detailed information, including location and vehicle information, is sent to the police via voice and text.

[1290] This allows drivers to receive real-time warnings and quickly contact the police as needed, significantly improving safety.

[1291] (Example 1)

[1292] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1293] In recent years, accidents caused by dangerous driving and intimidating driving have been increasing, and there is a need to detect these driving behaviors in real time and respond quickly. However, current dashcams and driver assistance systems are insufficient in their ability to adequately detect dangerous driving behavior behind the vehicle and warn the driver. Therefore, a system is needed to ensure driver safety and respond immediately to abnormal driving.

[1294] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1295] In this invention, the server includes means for capturing rear-view video of the transport equipment using a rear camera, means for analyzing the captured rear-view video in real time using artificial intelligence to detect dangerous driving or intimidating driving, means for sending a warning message to the driver and providing advice on countermeasures for detected abnormal driving, auxiliary means for saving driver safety data to a storage device based on the analysis results, means for saving video data to an external storage device using a secure communication protocol, and voice support means for notifying an external organization at the driver's instruction. This enables the detection of dangerous driving behavior behind the vehicle in real time, prompting the driver to be warned and encouraging appropriate action, and furthermore, data on abnormal driving can be saved to the cloud and reported to an external organization as needed.

[1296] A "rear camera" is a video capture device used to capture images from the rear of a vehicle.

[1297] "Artificial intelligence" is a general term that includes machine learning algorithms and deep learning technologies used to analyze captured rear-view video footage and detect abnormal driving behavior.

[1298] "Dangerous driving" refers to driving behavior that is unstable or abnormal compared to normal driving conditions.

[1299] "Intimidating driving" refers to driving behavior that intentionally causes fear or anxiety to other drivers or pedestrians.

[1300] A "warning message" is notification information intended to prompt the driver to take action in response to detected abnormal driving behavior.

[1301] "Driver" refers to a person operating a vehicle.

[1302] A "storage device" refers to hardware or storage media used to store data.

[1303] "External storage devices" refer to external data storage media, such as cloud storage, used to save data outside of the local system.

[1304] "Auxiliary means" refers to additional functions or technologies used to operate the main function.

[1305] A "secure communication protocol" refers to technologies and standards for ensuring secure data communication. Specifically, this includes protocols such as HTTPS.

[1306] "Reporting" refers to the act of transmitting information about abnormal driving to an external organization when such behavior is detected.

[1307] "Voice support" refers to a function that uses voice to provide information to users or to report information to external organizations.

[1308] "External organizations" refer to organizations including the police and other public institutions.

[1309] Modes for carrying out the invention

[1310] The system according to the present invention is an integrated system that includes a rear camera, artificial intelligence-based video analysis, driver warnings, data storage in the cloud, and notification to an external organization. Specific embodiments of this system are described below.

[1311] Rear camera activation and video capture

[1312] When the system starts up, the terminal automatically activates the rear camera, which captures video footage from behind the transport equipment. This video data is encoded in real time and converted to a stream format (e.g., H.264).

[1313] Sending video data

[1314] The encoded video data is sent from the terminal to the server using a secure communication protocol (e.g., HTTPS). The terminal buffers the video data, divides it into appropriate packet sizes, encrypts it, and sends it to the server.

[1315] Preprocessing of video data

[1316] The server preprocesses the received video data. This preprocessing includes frame extraction, resizing, and normalization. The preprocessed data is then input into the artificial intelligence engine.

[1317] Analysis of video data

[1318] The server uses an artificial intelligence engine to analyze pre-processed video data. The analysis extracts features such as abnormal following distances, sudden speed changes, and frequent lane changes, and calculates an anomaly score using machine learning models (e.g., TensorFlow, PyTorch). This allows for the detection of dangerous driving and aggressive driving behavior.

[1319] Generating and sending warning messages

[1320] Based on the analysis results, the server generates a warning message if abnormal operation is detected. The generated warning message is sent to the terminal in real time.

[1321] Driver warning notification

[1322] The device notifies the driver of any received warning messages. The notification is displayed on the screen and also alerts the driver with an audio alert. For example, a message such as "There is a vehicle approaching dangerously close from behind. Change lanes to avoid it" will be displayed, and the same information will be announced by voice.

[1323] Cloud storage of abnormal driving data

[1324] The server uploads and stores the detected abnormal operation data to the cloud. This data includes video frames and metadata (date and time, location information, etc.) at the time of the anomaly detection. The data is stored using cloud storage services (e.g., AWS S3, Google Cloud Storage).

[1325] Driver's report to the police

[1326] If the driver deems the situation serious, they will be given the option to call the police through the terminal's interface. If the driver selects the calling option, simply pressing the call button will activate voice support, and pre-configured information (location, license plate number, etc.) will be sent to the police in both voice and text format.

[1327] Specific example

[1328] For example, if a vehicle approaches abnormally closely from behind while driving on a highway, the rear camera captures the image behind the vehicle, and the artificial intelligence engine detects the abnormal proximity. The server notifies the terminal that an abnormal proximity has been detected, and the terminal displays the message, "There is a vehicle approaching abnormally closely from behind. Change lanes to avoid it," and also notifies the driver of the same message by voice. This situation is saved to the cloud and can be reviewed later. If the driver presses the report button, the police are automatically notified, and a notification is sent stating, "This vehicle is currently traveling at point XX, and is experiencing aggressive driving from behind. The vehicle number is XX-1234."

[1329] Examples of prompts for generative AI models

[1330] For example, analysis using a generated AI model can be performed by using the following prompt statement.

[1331] You are developing an advanced driver assistance system. This system integrates a rear camera, AI-powered video analysis, driver warnings, cloud data storage, and notification to external organizations. Explain the program's processes and provide specific examples for the following scenarios.

[1332] scenario:

[1333] 1. Please explain how the device activates the rear camera and captures rearward video when the system starts up.

[1334] 2. Explain how the artificial intelligence engine analyzes video data to detect dangerous or aggressive driving.

[1335] 3. Please explain in detail the mechanism for displaying and notifying drivers of warnings.

[1336] 4. Please explain the data storage and reporting process for abnormal operation to external organizations.

[1337] Specific example:

[1338] Describe a scenario in which a vehicle approaches unusually closely from behind while driving on a highway, and specifically explain how each process functions in that situation.

[1339] By inputting this prompt into the AI ​​model, it is possible to generate a detailed explanation of the system's operation and various functions.

[1340] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1341] System program processing flow

[1342] Step 1:

[1343] The device automatically activates the rear camera when the system starts up.

[1344] Specific operation: The terminal powers on the rear camera via internal control software and instructs the camera to begin capturing rearward video. The rear camera continuously captures rearward video and generates encoded stream data (e.g., in H.264 format).

[1345] Input: System power signal.

[1346] Output: Rear camera capture started, encoded video data.

[1347] Step 2:

[1348] The terminal sends the encoded video data to the server using a secure communication protocol (e.g., HTTPS).

[1349] Specific operation: The terminal buffers the video data, divides it into packets, and encrypts them. It then sends them to the server via the network stack.

[1350] Input: Encoded video data from the rear camera.

[1351] Output: Encrypted video data sent to the server.

[1352] Step 3:

[1353] The server preprocesses the received video data.

[1354] Specific operation: The server receives data, decodes the packets, and divides them into frames. Each frame is resized and normalized. After that, the pre-processed data is input into the AI ​​engine.

[1355] Input: Encrypted video data packets.

[1356] Output: Preprocessed video data (resized and normalized frames).

[1357] Step 4:

[1358] The server analyzes the pre-processed video data.

[1359] Specific operation: The AI ​​engine extracts features to detect abnormalities such as insufficient following distance, sudden speed changes, and frequent lane changes. A machine learning model (e.g., TensorFlow, PyTorch) is used to calculate an anomaly score and determine whether the driving is dangerous or aggressive.

[1360] Input: Pre-processed video data.

[1361] Output: Abnormal operation score and judgment result.

[1362] Step 5:

[1363] Based on the analysis results, the server generates a warning message if abnormal operation is detected.

[1364] Specific operation: When abnormal operation is detected, the server generates a warning in text format and creates a warning message for the driver.

[1365] Input: Score and judgment result for abnormal operation.

[1366] Output: Warning message.

[1367] Step 6:

[1368] The server sends the generated warning message to the terminal.

[1369] Specific operation: The server sends a warning message to the terminal in real time.

[1370] Input: Warning message.

[1371] Output: Send to the terminal.

[1372] Step 7:

[1373] The terminal notifies the driver of the received warning message.

[1374] Specific operation: The device displays a warning message on its screen and notifies the driver of the same content via an audio alert. For example, it might inform the driver, "There is a vehicle approaching dangerously close from behind. Change lanes to avoid it."

[1375] Input: Warning message.

[1376] Output: Display, audio alert.

[1377] Step 8:

[1378] The server uploads and stores the detected abnormal operation data to the cloud.

[1379] Specific operation: The server collects video frames and metadata (date, time, location information, etc.) from the time of anomaly detection into storage data and uploads it to a cloud storage service (e.g., AWS S3, Google Cloud Storage).

[1380] Input: Anomaly detection data (video frames, metadata).

[1381] Output: Data stored in cloud storage.

[1382] Step 9:

[1383] If the driver deems the situation serious, they will notify the police via the device.

[1384] Specific operation: When the driver presses the call button, the device activates its voice assistance function and sends pre-configured information (location information, license plate number, etc.) to the police in both voice and text format.

[1385] Input: Driver presses the call button, saved information.

[1386] Output: Automatic notification to the police (voice and text format).

[1387] (Application Example 1)

[1388] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1389] In recent years, aggressive driving and dangerous driving have become a social problem, making prevention and response to these issues urgent. Furthermore, in order to ensure the safety of individuals, including pedestrians, there is a need for a system that can detect abnormal behavior in real time and provide appropriate warnings and countermeasures. The present invention aims to provide a system that ensures the safety of a wider range of individuals by utilizing not only automobiles but also smart glasses.

[1390] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1391] In this invention, the server includes means for capturing rear-view video of a vehicle using a rear camera, means for analyzing the captured rear-view video in real time using AI to detect dangerous driving and aggressive driving, means for sending a warning message to the driver and providing advice on countermeasures for detected abnormal driving, means for uploading and storing the detected abnormal driving data to the cloud, means for providing voice support for notifying external organizations such as the police at the driver's instruction, means for capturing rear-view video using a rear camera mounted on smart glasses to detect approaching abnormal behavior, and means for providing the user with a visual and audible warning message when approaching abnormal behavior is detected. This enables universal safety assurance not only for vehicle drivers but also for pedestrians.

[1392] A "rear camera" is a camera device installed in cars or smart glasses that captures images of the area behind the vehicle.

[1393] "AI" is an abbreviation for artificial intelligence, a technology that uses machine learning and deep learning to analyze data and detect patterns and anomalies.

[1394] "Dangerous driving" refers to driving behaviors that endanger other road users, such as sudden changes in speed or frequent lane changes.

[1395] "Aggressive driving" refers to driving behavior that intentionally causes fear to other drivers, such as overtaking other vehicles and slamming on the brakes, or driving dangerously close to them.

[1396] A "warning message" is a message that provides a warning or evacuation order to the driver or user when dangerous driving or aggressive driving is detected.

[1397] "Cloud" refers to external data storage and computing resources provided via the internet.

[1398] "Voice support means" refers to functions that use voice to provide information to users or to report to external organizations.

[1399] "Smart glasses" are glasses-type wearable devices that have computer functions and assist the user's visual information.

[1400] "Abnormal behavior" refers to actions that deviate from normal behavioral patterns and may pose a danger to the user or others.

[1401] "Means of providing warning messages visually and audibly" refers to functions that use displays and speakers to convey warning information to users visually and audibly.

[1402] The system according to the present invention is an integrated system for ensuring the safety of automobile drivers and pedestrians. This system includes a rear camera, AI-based video analysis, warning messages to the user, data storage to the cloud, and notification to the police, etc. Embodiments of this system will be described below.

[1403] System hardware and software configuration

[1404] Hardware:

[1405] Smart glasses (e.g., typical wearable devices)

[1406] Rear camera (mounted in cars and smart glasses)

[1407] Speaker (for voice notifications)

[1408] GPS module (for acquiring location information)

[1409] software:

[1410] Video capture API (for acquiring video data)

[1411] AI models (for video analysis, e.g., TensorFlow, PyTorch)

[1412] Cloud storage (for data storage, e.g., AWS S3)

[1413] Secure communication protocols (e.g., HTTPS)

[1414] Reporting API (e.g. Twilio, Firebase)

[1415] System Operation Overview

[1416] 1. Startup and video capture

[1417] When smart glasses or a car are activated, the rear camera automatically activates and captures footage of the area behind the vehicle. The video data is encoded in real time.

[1418] 2. Video data preprocessing

[1419] The server receives the video data and performs preprocessing. This includes frame extraction, resizing, and normalization.

[1420] 3. AI analysis

[1421] Pre-processed video data is analyzed by an AI engine. The AI ​​engine detects abnormal behavior (such as dangerous driving, road rage, or suspicious individuals approaching).

[1422] 4. Generate and notify warning messages

[1423] When abnormal behavior is detected, the server generates a warning message and notifies the user through smart glasses or the car's display and speakers. For example, the message might say, "There is a vehicle approaching you dangerously closely from behind. Change lanes to avoid it."

[1424] 5. Data storage and reporting

[1425] Data on detected abnormal behavior is stored in the cloud. This includes location information, date and time, and video frames. When a user presses the report button, voice support is activated, and location information and video data are automatically sent to the police.

[1426] Specific example

[1427] For example, if a pedestrian wearing smart glasses is walking along a school route and a suspicious person approaches from behind, the rear camera captures the situation. The AI ​​engine detects the abnormal approach and displays a message saying, "Someone is approaching you from behind. Please move to a safe place," and also provides an audio notification with the same message. If the user presses the emergency call button, a report is sent to the police stating, "I am currently walking at location XX, and a suspicious person is approaching from behind."

[1428] Example of a prompt

[1429] Please create a smart glasses application that works as follows: Use the smart glasses' rear camera to capture rearward video, and have AI detect abnormal behavior (following, sudden approach). Implement a function to provide visual and audible warnings upon detection, and automatically notify the police in emergencies.

[1430] In summary, this system provides a technical solution to ensure the safety of vehicle drivers and pedestrians in real time and to respond quickly and appropriately.

[1431] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1432] Step 1:

[1433] Smart glasses or a car terminal are activated. The terminal automatically activates the rear camera and captures rearward video in real time. The captured video data is encoded and sent to the server using a secure communication protocol (e.g., HTTPS). The input is video data from the rear camera, and the output is encoded video data.

[1434] Step 2:

[1435] The server preprocesses the received video data. Specifically, it extracts frames, resizes them, and normalizes them. This preprocessing prepares the video for efficient analysis by the AI ​​engine. The input is encoded video data, and the output is preprocessed video data.

[1436] Step 3:

[1437] The server's AI engine analyzes pre-processed video data. The AI ​​engine uses a machine learning model to detect abnormal behavior (such as dangerous driving, road rage, or suspicious individuals approaching). Specifically, it extracts features such as distance between vehicles, speed changes, and lane change frequency, and calculates an abnormality score. The input is pre-processed video data, and the output is the result of detecting abnormal behavior.

[1438] Step 4:

[1439] The server generates a warning message if abnormal behavior is detected. The warning message is provided visually and audibly. The generated warning message is sent to the terminal using a secure communication protocol. The input is the result of the detected abnormal behavior, and the output is the warning message.

[1440] Step 5:

[1441] The device notifies the user of received warning messages. Specifically, it conveys warning messages visually and audibly using the smart glasses' display and speaker. For example, it might say, "There is a vehicle approaching you dangerously close from behind. Change lanes to avoid it." The input is the warning message, and the output is the notification to the user.

[1442] Step 6:

[1443] The server saves data on detected abnormal behavior to the cloud. The saved data includes date and time, location information, and video frames. Cloud storage (e.g., AWS S3) is used for data storage. The input is the detected abnormal behavior and associated metadata, and the output is the data stored in the cloud.

[1444] Step 7:

[1445] When a user presses the emergency call button, the device's voice support function is activated. Pre-configured location information and video data are transmitted to the police via voice and text. The input consists of the user's call instructions and related data (location information, video data), and the output is a notification to the police.

[1446] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1447] Modes for carrying out the invention

[1448] The system according to the present invention is an integrated system that combines a rear camera, AI-based video analysis, driver warnings, data storage to the cloud, and notification to the police, with an emotion engine that recognizes the user's emotions. The processing of this system's program will be described below in natural language.

[1449] System program processing

[1450] When the system starts up, the terminal activates the rear camera and continuously captures video footage of the area behind the vehicle. This video data is encoded and sent to the server, which receives the video data and performs preprocessing for analysis by the AI ​​engine. The AI ​​engine analyzes the preprocessed data to detect dangerous driving and aggressive driving.

[1451] Based on the analysis results, the server generates a warning message if abnormal operation is detected. The message is sent to the terminal in real time, and the terminal notifies the driver of the received message. Furthermore, data on abnormal operation is stored in the cloud and can be reviewed later.

[1452] The present invention also includes a system for recognizing the driver's emotions. Specifically, a camera mounted on the terminal captures an image of the driver's face and transmits this data to a server. The server uses an emotion engine to analyze the image of the driver's face and recognize their emotional state. For example, it can detect states such as tension, stress, and relaxation.

[1453] Based on recognized emotions, the content of warning messages and advice is adjusted. For example, if a driver is stressed, more detailed and considerate advice may be provided. Furthermore, emotional states are included as additional information in data stored in the cloud, allowing for later analysis.

[1454] Furthermore, it includes a feature that automatically prompts a police alert if the driver is experiencing particular tension or stress. This feature allows for quick external assistance to be sought if the driver's emotional state is deemed to be a factor detrimental to safe driving.

[1455] Specific example

[1456] For example, suppose you are driving on a highway and a vehicle approaches you dangerously closely from behind. In this case, the rear camera captures the image behind you, and the server's AI engine detects the dangerous proximity. The server generates a warning message such as, "There is a vehicle approaching you dangerously closely from behind. Change lanes to avoid it," and sends it to your device.

[1457] Simultaneously, a camera is operating to capture video of the driver's face, and this video data is sent to a server. The server's emotion engine analyzes this data to recognize that the driver is stressed. Based on this, it checks whether the warning message is considerate of the driver's emotional state and adjusts it as needed.

[1458] Furthermore, the details of this abnormal driving and the driver's emotional state are saved to the cloud. Also, if the driver presses the report button, the system sends information to the police stating, "This vehicle is currently traveling at location XX and is experiencing aggressive driving from behind. The vehicle's license plate number is XX-1234. The driver is extremely distressed."

[1459] In this way, this system not only detects dangerous driving but also provides comprehensive safety measures that take into account the driver's emotional state.

[1460] The following describes the processing flow.

[1461] Modes for carrying out the invention

[1462] System program processing

[1463] Step 1:

[1464] Rear camera activation and video capture start

[1465] Terminal: When the vehicle engine starts, the rear camera automatically activates. After activation, the rear camera begins capturing rearward video and generates video data frame by frame.

[1466] Step 2:

[1467] Sending video data

[1468] Terminal: Encodes captured video frames at high speed and sends them to the server using a secure communication protocol.

[1469] Step 3:

[1470] Receiving and preprocessing video data

[1471] Server: Receives video data, stores it in a buffer, and performs preprocessing to prepare it for input into the AI ​​engine. This preprocessing includes frame extraction, resizing, and normalization.

[1472] Step 4:

[1473] AI analysis

[1474] Server: Pre-processed video data is input into the AI ​​engine to analyze patterns of dangerous driving and aggressive driving. Specifically, features such as abnormal following distances, sudden speed changes, and frequent lane changes are extracted, and an abnormality score is calculated using a machine learning model.

[1475] Step 5:

[1476] Generating warning messages

[1477] Server: When the AI ​​engine detects abnormal driving, it generates a warning message. This message includes the type of dangerous driving detected and the actions the driver should take. For example, it might say, "There is a vehicle approaching dangerously closely from behind. Change lanes to avoid it."

[1478] Step 6:

[1479] Sending a warning

[1480] Server: Sends generated warning messages to terminals in real time.

[1481] Step 7:

[1482] Warning display and audio notification

[1483] Terminal: Displays received warning messages to the driver and also provides voice notifications. For example, a message is displayed on the screen, and a voice notification says, "There is a vehicle approaching you dangerously close from behind. Change lanes to avoid it."

[1484] Step 8:

[1485] Cloud storage of data

[1486] Server: Uploads and stores data on detected abnormal operation to the cloud. This data includes video frames, date and time, location information, etc.

[1487] Step 9:

[1488] Video capture of the driver's face

[1489] Terminal: The camera operates to capture video of the driver's face. This video data is encoded and sent to the server.

[1490] Step 10:

[1491] Emotion recognition by an emotion engine

[1492] Server: The server analyzes the received video of the driver's face using an emotion engine to recognize their emotional state. Specifically, it analyzes the driver's facial expressions and condition to determine emotions such as "tension," "stress," and "relaxation."

[1493] Step 11:

[1494] Adjusting emotion-based warning messages

[1495] Server: Based on recognized emotions, it adjusts the content of warning messages and corrective advice. For example, if the driver is stressed, it provides more polite and detailed advice.

[1496] Step 12:

[1497] Cloud storage of emotional data

[1498] Server: The recognized emotional states are added to data stored in the cloud, making them available for later analysis.

[1499] Step 13:

[1500] Presentation of reporting options

[1501] Terminal: Displays an option on the screen for the driver to report the incident to the police. For example, it might display the option, "Do you want to report this?"

[1502] Step 14:

[1503] Execution of the report

[1504] User: The driver selects the reporting option.

[1505] Terminal: When the reporting button is pressed, voice support is activated, and detailed information, including location, vehicle information, and emotional state, is sent to the police via voice and text. For example, information such as, "This vehicle is currently traveling at location XX and is experiencing aggressive driving from behind. The vehicle number is XX-1234. The driver is extremely agitated," will be sent.

[1506] As described above, this system provides comprehensive safety measures that take into account not only the detection of dangerous driving and aggressive driving, but also the driver's emotional state.

[1507] (Example 2)

[1508] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1509] Conventional automotive safety systems have the ability to detect dangerous driving and aggressive driving, but they cannot take into account the driver's emotional state, which can lead to increased stress and tension for the driver. Furthermore, while data on abnormal driving is stored, the lack of sufficient specific advice on countermeasures limits the potential for improving safety.

[1510] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for capturing rear-view video of the vehicle using a rear camera, means for analyzing the captured rear-view video in real time using AI to detect dangerous driving and aggressive driving, means for sending a warning message to the driver regarding the detected abnormal driving and providing advice on countermeasures, means for uploading and saving the detected abnormal driving data to the cloud, means for capturing video of the driver's face and performing emotion analysis, and means for adjusting the warning message based on the observed emotions of the driver. This makes it possible to not only detect dangerous driving and aggressive driving in real time, but also to implement comprehensive safety measures that take into account the driver's emotional state.

[1511] A "rear camera" is a camera mounted on the rear of a vehicle, used to capture images of the area behind the vehicle.

[1512] "AI" stands for Artificial Intelligence, and refers to an algorithm or system that automatically performs specific tasks.

[1513] "Dangerous driving" refers to driving behavior that violates the Road Traffic Act and endangers safety.

[1514] "Aggressive driving" refers to intentionally engaging in intimidating driving behavior towards other vehicles.

[1515] A "warning message" is a message that the system issues to the driver to warn them about dangerous driving or aggressive driving.

[1516] "Safety advice" refers to information that provides drivers with specific instructions and advice to drive safely.

[1517] "Cloud" refers to a group of servers that provide data storage and processing functions via the internet.

[1518] "Emotional analysis" is a technology that evaluates a driver's emotional state based on facial expressions and other factors using video data of their face.

[1519] "Reporting methods" refer to a function that, in response to the driver's instructions, allows for reporting abnormal driving situations to external organizations such as the police using voice or text.

[1520] "Preprocessing means" refers to techniques that process video data, such as noise reduction and frame selection, before analysis by AI.

[1521] Modes for carrying out the invention

[1522] The system according to the present invention is an integrated safety driving support system that includes a rear camera, an AI engine, an emotion analysis engine, a driver warning message function, data storage to the cloud, and an automatic notification function to the police.

[1523] Hardware and software configuration:

[1524] The rear camera is mounted on the back of the car and continuously captures images of the area behind it.

[1525] The device has the function of encoding video data from the rear camera and sending it to the server.

[1526] The server has hardware and software to preprocess the received video data and analyze it with an AI engine.

[1527] The AI ​​engine includes algorithms that detect dangerous driving and aggressive driving in real time.

[1528] Another camera on the device captures video of the driver's face, providing data to recognize the driver's emotional state.

[1529] The emotion analysis engine is built into the server and analyzes the transmitted facial video data to recognize the driver's emotional state (tension, stress, relaxation, etc.).

[1530] The cloud server securely stores data such as abnormal driving behavior and the driver's emotional state, allowing for later review and analysis.

[1531] The warning message function sends messages generated on the server to the terminal in real time, and the terminal notifies the driver.

[1532] The automatic reporting function has a means of reporting to an external organization (e.g., the police) based on the driver's instructions.

[1533] Specific examples of how the system works:

[1534] Suppose a vehicle is detected approaching abnormally closely from behind while driving on a highway. The rear camera captures the image behind the vehicle, the video data is encoded on the terminal, and then sent to the server.

[1535] The server uses an AI engine to detect abnormal proximity. It generates a warning message such as, "There is a vehicle approaching abnormally closely from behind. Change lanes to avoid it," and sends it to the terminal.

[1536] Simultaneously, a camera capturing the driver's face is also operating, and the video data is sent to a server. An emotion analysis engine analyzes this data and recognizes that the driver is feeling stressed.

[1537] Based on this understanding, the server adjusts the content of the warning message. For example, it might add advice such as, "Stay calm and check behind you."

[1538] All data (including details of abnormal driving and the driver's emotional state) is stored on a cloud server.

[1539] When the driver presses the call button, the automatic call function sends a message to the police saying, "This vehicle is currently traveling at location XX, and is experiencing aggressive driving from behind. The vehicle's license plate number is XX-1234. The driver is extremely distressed."

[1540] Example of a prompt:

[1541] 1. "Please describe in detail the process from detecting abnormal operation to sending a warning message in conjunction with the rear camera."

[1542] 2. "Please explain specific methods for adjusting warning messages using driver facial recognition and emotion analysis."

[1543] Thus, the system according to the present invention not only detects dangerous driving and aggressive driving in real time, but also provides comprehensive safety measures that take into account the driver's emotional state.

[1544] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1545] Step 1: System startup

[1546] The terminal prepares to activate the rear camera simultaneously with the system startup. The input is the system startup signal, and the output is the completion of the rear camera startup.

[1547] Specifically, when the device's power button is pressed, the internal software sends a command to activate the rear camera.

[1548] Step 2: Video capture using the rear camera

[1549] The device activates the rear camera and continuously captures rearward video. The input is real-time video data, and the output is encoded rearward video data.

[1550] Specifically, the rear camera captures the rear view frame by frame and sends it to the terminal. The terminal then encodes this data.

[1551] Step 3: Encode and transmit video data

[1552] The terminal encodes the captured video data. The input is raw video data, and the output is encoded data. After encoding, this data is sent to the server.

[1553] Specifically, the terminal compresses the video data into a format such as MPEG and sends it to the server using a communication protocol.

[1554] Step 4: Video analysis using an AI engine

[1555] The server preprocesses the received encoded video data. The input is encoded video data, and the output is preprocessed data. Preprocessing such as noise reduction and frame selection is performed.

[1556] The server's AI engine analyzes pre-processed data to detect dangerous driving and aggressive driving. The input for the analysis is pre-processed data, and the output is the detection result of abnormal driving.

[1557] Step 5: Generate and send the warning message.

[1558] If abnormal operation is detected, the server generates a warning message. The input is the result of the abnormal operation detection, and the output is the generated warning message. Next, this message is sent to the terminal in real time, and the terminal notifies the driver.

[1559] Specifically, the server creates a warning message based on a template and sends it to the terminal using the TCP / IP protocol. The terminal then displays an audio alert or a pop-up on the screen.

[1560] Step 6: Driver's Emotion Recognition

[1561] The camera on the device captures video of the driver's face. The input is video data of the driver's face, and the output is encoded video data of the face. This is sent to the server.

[1562] The server's emotion engine analyzes facial video data to recognize the driver's emotional state. The input is encoded facial video data, and the output is the result of the emotion analysis.

[1563] Step 7: Adjusting Emotion-Based Warning Messages

[1564] The server adjusts warning messages to be more detailed and thoughtful if the driver is stressed. The input is the result of sentiment analysis, and the output is the adjusted warning message.

[1565] Specifically, the server selects a warning message template and adjusts the content of the advice based on the sentiment analysis results.

[1566] Step 8: Save data to the cloud

[1567] The server stores data on abnormal operation and sentiment analysis on a cloud server. The input is the status of abnormal operation and the results of sentiment analysis, and the output is the data stored in the cloud.

[1568] Specifically, the server uploads data to the cloud storage service via the database.

[1569] Step 9: Automatically report to the police

[1570] When the driver presses the emergency call button, the server automatically initiates the process of notifying the police. The input is the driver's emergency call signal, and the output is the emergency call message sent to the police.

[1571] Specifically, the server generates a notification message containing location and vehicle information, and sends it to the police using the appropriate communication protocol.

[1572] (Application Example 2)

[1573] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1574] Conventional in-vehicle systems were specialized in detecting dangerous driving behavior and abnormal approaches from the rear, but lacked consideration for the driver's emotional state. This made it difficult to respond appropriately when the driver felt tension or stress, potentially hindering safe driving. Furthermore, warnings and advice after detecting abnormal driving were uniform, failing to provide measures tailored to the driver's emotional state. As a result, safety and driver stress reduction were not adequately achieved.

[1575] In Application Example 2, the identification processing by the identification processing unit 290 of the data processing device 12 is realized by the following means. In this invention, the server includes means for capturing rear-view video of the vehicle using a rear camera, means for analyzing the captured rear-view video in real time using AI to detect dangerous driving and aggressive driving, means for sending a warning message to the driver and providing advice on countermeasures for detected abnormal driving, means for uploading and saving the detected abnormal driving data to the cloud, voice support means for notifying external organizations such as the police at the driver's instruction, means for capturing video of the driver's face and analyzing the driver's emotional state using an emotion recognition engine, and means for adjusting the content of warning messages and countermeasure advice based on the driver's emotional state. This enables appropriate countermeasures according to the driver's emotional state, thereby improving safety and reducing stress.

[1576] A "rear camera" is a camera device used to capture images of the area behind a vehicle.

[1577] The "AI engine" is an artificial intelligence technology that analyzes captured video in real time to detect abnormal driving, aggressive driving, and the driver's emotional state.

[1578] "Dangerous driving" refers to any driving behavior that may endanger other vehicles or pedestrians.

[1579] "Aggressive driving" refers to driving behavior that is harassing or endangering to the driver, such as a vehicle following closely behind another vehicle.

[1580] A "warning message" is a message or alert designed to alert the driver to detected abnormal driving behavior.

[1581] "Countermeasure advice" refers to information that presents recommended driving actions and points to note to the driver in response to detected abnormal driving behavior.

[1582] "Cloud" refers to data storage and processing services provided via the internet, and in this context, it refers to remote storage for saving data on detected abnormal operation.

[1583] "Voice support means" refers to a function that provides voice support when the driver instructs an external organization, such as the police, to make a report.

[1584] The "emotion recognition engine" is a technology that analyzes the driver's facial image to recognize their emotional state.

[1585] "Emotional state" refers to the driver's mental and psychological state, including states such as tension, stress, and relaxation.

[1586] "Video data preprocessing" is a technique that transforms video data into a format that is easily accessible for AI analysis.

[1587] "External organizations" refer to public institutions such as the police and other support organizations.

[1588] "Abnormal driving data" refers to all data that, as a result of analyzing video footage acquired from rear cameras and other sources, is determined to indicate abnormal driving.

[1589]

[1590] The system according to the present invention is an integrated system installed in an autonomous vehicle. This system captures rearward images using a rear camera and analyzes the data with an AI engine to detect dangerous driving and aggressive driving. In addition, a camera that captures facial images and an emotion recognition engine are used to recognize the driver's emotional state.

[1591] Specific system configuration

[1592] 1. Hardware configuration:

[1593] Rear camera: A camera used to capture images of the rear of a vehicle.

[1594] Emotion Recognition Camera: A camera that captures video of the driver's face.

[1595] In-car display: A display that shows warning messages and advice to the driver. This can also be replaced by the display on a smartphone or smart glasses.

[1596] 2. Software configuration:

[1597] AI Engine: Uses platforms such as TensorFlow to analyze video from the rear camera and emotion recognition camera.

[1598] Cloud storage: Remote storage for saving data on detected abnormal operation. Amazon AWS or Google Cloud Storage are possible options.

[1599] Communication software: A tool for uploading abnormal driving data to the cloud and sending HTTP requests to external organizations such as the police to report it. Uses libraries such as Python's Requests library.

[1600] System Operation Description

[1601] When the system starts up, the rear camera continuously captures rearward video, and this video data is sent to the AI ​​engine in real time. After the server receives the video data and performs preprocessing, the AI ​​engine detects dangerous driving and aggressive driving.

[1602] If abnormal driving is detected, the server generates a warning message and displays it in real time on the in-vehicle display. For example, it might notify the driver with a warning message such as, "An abnormally close approach is coming from behind. Please be careful." This data is also uploaded to cloud storage and can be reviewed later.

[1603] Furthermore, an emotion-recognition camera captures the driver's face, and the video data is sent to a server. The server's emotion recognition engine analyzes this data to recognize the driver's emotional state (e.g., tension or stress). Based on the recognized emotion, the content of warning messages and corrective advice is adjusted. For example, if the driver is tense, more polite and detailed advice can be provided.

[1604] Specific usage examples

[1605] For example, suppose a vehicle approaches abnormally closely from behind while driving on a highway. In this case, the rear camera captures the image behind the vehicle, and the server's AI engine detects the abnormal approach. The server generates a warning message such as, "There is a vehicle approaching abnormally closely from behind. Change lanes to avoid it," and sends it to the in-car display. At the same time, a camera is also operating to capture the driver's face, and this video data is sent to the server. The server's emotion recognition engine analyzes this data and recognizes that the driver is stressed. Based on this, it checks whether the warning message is considerate of the driver's emotional state and adjusts it as needed. Furthermore, the details of this abnormal driving and the driver's emotional state are stored in the cloud.

[1606] Example of a prompt

[1607] Regarding the rear camera frame: "The system analyzes the rear camera footage to detect abnormally close approaches from behind."

[1608] In the case of a facial emotion recognition camera: "It analyzes the driver's facial image and recognizes emotional states such as tension and stress."

[1609] This allows for appropriate measures to be taken in response to the driver's emotional state, thereby improving safety and reducing stress.

[1610] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1611] Step 1:

[1612] The terminal activates the rear camera and captures the rear view of the vehicle. It receives live video from the rear camera as input, encodes the video data, and sends it to the server. Specifically, it captures the rear camera's video input in real time, compresses the data, and uploads it to the server via the network.

[1613] Step 2:

[1614] The server preprocesses the received video data. It receives encoded video data as input and decodes and preprocesses it into a format that is easy for the AI ​​engine to analyze. Specifically, it decodes the video data and adjusts the frame rate and removes noise as needed.

[1615] Step 3:

[1616] The server analyzes pre-processed video data using an AI engine. It receives pre-processed video data as input and uses an AI model to detect dangerous driving and aggressive driving. Specifically, it inputs the prompt message "Analyze the rear camera footage and detect abnormally close driving from behind" into the generating AI model and obtains the analysis results.

[1617] Step 4:

[1618] The server generates warning messages for detected abnormal operation. It receives analysis results as input and uses templates to create appropriate warning messages. Specifically, depending on the type of abnormal operation, it generates messages such as "An abnormal approach is coming from behind. Please be careful," and sends them to the terminal.

[1619] Step 5:

[1620] The terminal notifies the driver of received warning messages. It receives warning messages from the server as input and displays them on the in-car display or smart glasses. Specifically, it notifies the driver of the warning messages visually and audibly.

[1621] Step 6:

[1622] The device's emotion recognition camera captures the driver's facial image. It receives live video of the driver's face as input, encodes the video data, and sends it to the server. Specifically, the emotion recognition camera captures the driver's facial image in real time, compresses the data, and uploads it to the server.

[1623] Step 7:

[1624] The server preprocesses the received facial video data. It receives encoded facial video data as input and decodes and preprocesses it into a format that is easy for the emotion recognition engine to analyze. Specifically, it decodes the facial video data and adjusts the frame rate and removes noise as needed.

[1625] Step 8:

[1626] The server analyzes pre-processed facial video data using an emotion recognition engine. It receives pre-processed facial video data as input and uses an AI model to recognize the driver's emotional state. Specifically, it inputs the prompt message "Analyze the driver's facial video and recognize emotional states such as tension and stress" into the generating AI model and obtains the analysis results.

[1627] Step 9:

[1628] The server adjusts warning messages and advice based on the driver's emotional state. It receives the result of emotion recognition as input and personalizes warning messages and advice based on that result. For example, if the driver is stressed, the message might be adjusted to something like, "There is a vehicle approaching dangerously close from behind. Please remain calm and take appropriate action."

[1629] Step 10:

[1630] The server stores abnormal driving data and driver emotional states in the cloud. It receives abnormal driving and emotional state data as input and uploads it to cloud storage. Specifically, it converts the data into an appropriate format and sends it to the cloud storage service for storage.

[1631] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1632] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1633] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1634] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1635] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. In the upper and lower directions of the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. Also, the upper side of the concentric circles is where "pleasant" emotions are located, and the lower side is where "unpleasant" emotions are located. In this way, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1636] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1637] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1638] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1639] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1640] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1641] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1642] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1643] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1644] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1645] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1646] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1647] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1648] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1649] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1650] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1651] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[1652] The following is further disclosed regarding the embodiments described above.

[1653] Claims

[1654] (Claim 1)

[1655] A means of capturing rear-view video of a vehicle using a rear camera,

[1656] A method for detecting dangerous driving and aggressive driving by analyzing rear-view video captured by AI in real time,

[1657] A means of sending a warning message to the driver in response to detected abnormal driving behavior and providing advice on countermeasures,

[1658] A means of uploading and saving the detected abnormal operation data to the cloud,

[1659] A voice support system for reporting to external organizations such as the police, based on the driver's instructions,

[1660] A system that includes this.

[1661] (Claim 2)

[1662] The system according to claim 1, further comprising preprocessing means for preprocessing video data for AI analysis.

[1663] (Claim 3)

[1664] The system according to claim 1, which outputs detailed information including location information and vehicle information in voice and text via a notification means.

[1665] "Example 1"

[1666] (Claim 1)

[1667] A means of capturing rearward video of transport equipment using a rear camera,

[1668] A means of detecting dangerous driving or intimidating driving by analyzing rear-view video captured by artificial intelligence in real time,

[1669] A means of sending a warning message to the driver in response to detected abnormal driving behavior and providing advice on countermeasures,

[1670] An auxiliary means for saving driver safety data to a storage device based on the analysis results,

[1671] A means of saving video data to an external storage device using a secure communication protocol,

[1672] A voice support system for notifying an external organization at the driver's instruction,

[1673] A system that includes this.

[1674] (Claim 2)

[1675] The system according to claim 1, further comprising preprocessing means for preprocessing video data for analysis by artificial intelligence.

[1676] (Claim 3)

[1677] The system according to claim 1, which outputs detailed information including location information and transportation equipment information in voice and text via a notification means.

[1678] "Application Example 1"

[1679] (Claim 1)

[1680] A means of capturing rear-view video of a vehicle using a rear camera,

[1681] A method for detecting dangerous driving and aggressive driving by analyzing rear-view video captured by AI in real time,

[1682] A means of sending a warning message to the driver in response to detected abnormal driving behavior and providing advice on countermeasures,

[1683] A means of uploading and saving the detected abnormal operation data to the cloud,

[1684] A voice support system for reporting to external organizations such as the police, based on the driver's instructions,

[1685] A means of detecting approaching abnormal behavior by capturing rearward video using a rear camera mounted on smart glasses,

[1686] A means of providing the user with a visual and audible warning message when approaching abnormal behavior is detected,

[1687] A system that includes this.

[1688] (Claim 2)

[1689] The system according to claim 1, further comprising preprocessing means for preprocessing video data for AI analysis.

[1690] (Claim 3)

[1691] The system according to claim 1, which outputs detailed information including location information and vehicle information in voice and text via a notification means.

[1692] "Example 2 of combining an emotion engine"

[1693] (Claim 1)

[1694] A means of capturing rear-view video of a vehicle using a rear camera,

[1695] A method for detecting dangerous driving and aggressive driving by analyzing rear-view video captured by AI in real time,

[1696] A means of sending a warning message to the driver in response to detected abnormal driving behavior and providing advice on countermeasures,

[1697] A means of uploading and saving the detected abnormal operation data to the cloud,

[1698] A voice support system for reporting to external organizations such as the police, based on the driver's instructions,

[1699] A method for capturing video of the driver's face and performing emotion analysis,

[1700] A means of adjusting warning messages based on the emotions of the driver observed,

[1701] A system that includes this.

[1702] (Claim 2)

[1703] The system according to claim 1, further comprising preprocessing means for preprocessing video data for AI analysis.

[1704] (Claim 3)

[1705] The system according to claim 1, which outputs detailed information including location information and vehicle information in voice and text via a notification means.

[1706]

[1707] "Application example 2 when combining with an emotional engine"

[1708] (Claim 1)

[1709] A means of capturing rear-view video of a vehicle using a rear camera,

[1710] A method for detecting dangerous driving and aggressive driving by analyzing rear-view video captured by AI in real time,

[1711] A means of sending a warning message to the driver in response to detected abnormal driving behavior and providing advice on countermeasures,

[1712] A means of uploading and saving the detected abnormal operation data to the cloud,

[1713] A voice support system for reporting to external organizations such as the police, based on the driver's instructions,

[1714] A method for capturing video of the driver's face and analyzing the driver's emotional state using an emotion recognition engine,

[1715] A means of adjusting the content of warning messages and corrective advice based on the driver's emotional state,

[1716] A system that includes this.

[1717] (Claim 2)

[1718] The system according to claim 1, further comprising preprocessing means for preprocessing video data for AI analysis.

[1719] (Claim 3)

[1720] The system according to claim 1, which outputs detailed information including location information and vehicle information in voice and text via a notification means. [Explanation of Symbols]

[1721] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of capturing rear-view video of a vehicle using a rear camera, A method for detecting dangerous driving and aggressive driving by analyzing rear-view video captured by AI in real time, A means of sending a warning message to the driver in response to detected abnormal driving behavior and providing advice on countermeasures, A means of uploading and saving the detected abnormal operation data to the cloud, A voice support system for reporting to external organizations such as the police, based on the driver's instructions, A system that includes this.

2. The system according to claim 1, further comprising preprocessing means for preprocessing video data for AI analysis.

3. The system according to claim 1, which outputs detailed information including location information and vehicle information in voice and text via a notification means.

Images