system

Abstract

We provide the system. [Solution] A means of recognizing surrounding signs and traffic conditions using an in-vehicle camera, A means of using an artificial intelligence algorithm to provide instructions to the driver based on recognized signs and traffic conditions, A means of collecting vehicle driving behavior data, analyzing that data, and scoring the driver's driving skills, A means for generating data to reflect the scoring results in insurance premiums, Means for sending analysis results and scoring data to a server, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Currently, many drivers are constantly exposed to the risk of accidents caused by overlooking traffic signs and markings, as well as inappropriate driving behaviors such as sudden acceleration and sudden braking. Also, it is difficult for drivers to obtain opportunities to objectively judge and improve their own driving behaviors, which consequently affects the frequency of traffic accidents. Furthermore, despite driving safely, the efforts do not sufficiently translate into economic benefits, resulting in a situation where drivers have few incentives to actively promote safe driving.

Means for Solving the Problems

[0005] This invention provides a system that uses an in-vehicle camera to monitor the road environment in real time and recognizes signs and traffic conditions using an artificial intelligence algorithm. This system provides appropriate instructions to the driver and supports safe driving. Furthermore, by analyzing driving behavior data with AI, quantifying driving skills, and introducing a system where this score is reflected in insurance premiums, the system provides economic benefits to drivers and encourages continued safe driving. In this way, the aim is to support the improvement of driving skills and reduce the risk of accidents.

[0006] An "in-vehicle camera" is a device mounted on a vehicle that captures moving images of the surroundings.

[0007] A "sign" is a traffic sign based on the Road Traffic Act, a visual display intended to provide drivers with necessary information and instructions.

[0008] "Traffic conditions" is a general term for the driving environment that changes due to factors such as vehicles, pedestrians, and traffic lights on the road.

[0009] An "artificial intelligence algorithm" is a learning-based program used in a computer system to perform a specific task.

[0010] A "driver" is an individual who operates a vehicle and is responsible for operating it in accordance with traffic rules.

[0011] "Driving behavior data" refers to information that records the physical behavior of a vehicle while it is being driven, such as its speed, acceleration, braking, and steering.

[0012] "Scoring" is a process of quantifying and evaluating various data on driving behavior according to certain criteria.

[0013] "Insurance premiums" are the money that drivers pay to insurance companies to prepare for accidents and other incidents, and the amount is set based on a risk assessment. [Brief explanation of the drawing]

[0014] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0015] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, a labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, a labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] The present invention is a system installed in a vehicle driven by a driver, and consists of an in-vehicle camera, a processing unit using an artificial intelligence algorithm, a group of sensors for monitoring driving behavior and acquiring data, a communication module, and a display or audio output device for providing information.

[0036] Terminal operation

[0037] The device is installed in the vehicle and first captures real-time video of the road environment using an on-board camera. The captured video is sent to an internal processing unit and analyzed by an artificial intelligence algorithm. This analysis recognizes traffic signs, traffic lights, pedestrians, lanes, and vehicles ahead, enabling the system to provide the driver with necessary information and instructions. For example, if the speed limit is exceeded, a visual warning is displayed on the screen and an audio notification is given.

[0038] Furthermore, the terminal continuously collects various data such as vehicle speed, acceleration, and braking. This data is used to evaluate driving behavior, and items related to safe driving, particularly sudden acceleration and sudden braking, are scored.

[0039] Server operation

[0040] Data transmitted from the terminal is received and stored on the server. The server assigns a unique ID to each driver and manages the data based on this ID. The collected driving data is reanalyzed by a program that runs an artificial intelligence algorithm to accurately score driving skills. The score is calculated based on the driver's past driving data and the frequency of dangerous driving behaviors.

[0041] The server uses the analysis results to provide companies with data for adjusting insurance premiums. This adjustment is made by setting lower premiums for drivers with high scores (i.e., those who drive safely).

[0042] User interaction

[0043] Based on the visual and auditory feedback provided by the system, users can continuously review their driving style. After finishing their drive, they can check the score displayed on the terminal and see how their insurance premiums will change the following month. This motivates users to drive safely.

[0044] A concrete example would be a scenario where, while a driver is driving on a highway, the device recognizes the distance to the vehicle in front and issues a warning to maintain a safe distance. Through such prompts, users can naturally improve their driving habits.

[0045] The following describes the processing flow.

[0046] Step 1:

[0047] The device captures video data from the in-vehicle camera in real time. This data includes road signs, traffic lights, vehicles, pedestrians, and other information.

[0048] Step 2:

[0049] The device analyzes the captured video using an internal artificial intelligence algorithm. The algorithm uses image recognition technology to identify signs and traffic conditions, and provides the user with appropriate driving instructions based on this information.

[0050] Step 3:

[0051] The terminal simultaneously collects various sensor data related to driving behavior, such as vehicle speed, acceleration, and brake usage.

[0052] Step 4:

[0053] The device analyzes collected driving behavior data and calculates a safe driving score. The data analysis includes the frequency of sudden acceleration and braking, as well as whether or not the vehicle is speeding.

[0054] Step 5:

[0055] Based on the analysis results obtained, the device provides visual and auditory feedback to the user to encourage safe driving.

[0056] Step 6:

[0057] The device sends analysis data and scores to the server at regular intervals. The data is linked to a unique user ID.

[0058] Step 7:

[0059] The server receives data sent from the terminal and stores it in the database. Each user's driving history is managed.

[0060] Step 8:

[0061] The server performs additional analysis based on the received data and generates a score that re-evaluates driving behavior. This includes an analysis of driving trends over time.

[0062] Step 9:

[0063] The server provides the generated scores to the insurance company's system, which is then used to adjust the driver's insurance premiums.

[0064] Step 10:

[0065] Users review the feedback and scores displayed on their devices and consider areas for improvement for their next drive.

[0066] (Example 1)

[0067] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0068] In recent years, reducing traffic accidents and ensuring fair insurance premiums have become critical issues. Conventional systems make it difficult to objectively evaluate drivers' driving skills, sometimes resulting in inappropriate premium settings. Furthermore, mechanisms to promote safe driving by providing immediate feedback during driving are insufficient.

[0069] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0070] In this invention, the server includes means for recognizing surrounding traffic information using an in-vehicle image acquisition device, means for using a computational method to provide instructions to the driver based on the recognized traffic information, and means for collecting vehicle movement characteristic data and analyzing the data to evaluate the driver's driving skills. This makes it possible to promote safe driving and set appropriate insurance premiums based on individual driving behavior.

[0071] An "in-vehicle image acquisition device" is a device mounted on a vehicle to capture surrounding traffic information.

[0072] "Traffic information" refers to information that includes data on road conditions, traffic signs, traffic lights, pedestrians, and other vehicles.

[0073] "Computational methods" refer to algorithms and mathematical models used to analyze recognized traffic information and provide instructions to drivers.

[0074] "Movement characteristics data" refers to data related to the movement of a vehicle, such as vehicle speed, acceleration, and braking operation.

[0075] "Driving skills" refer to the ability of a driver to operate a vehicle safely and efficiently.

[0076] "Evaluation" is the process of analyzing a driver's driving behavior numerically or qualitatively based on collected data and making a judgment.

[0077] "Risk management information" refers to the data and indicators used in setting insurance premiums and assessing risks.

[0078] A "data storage device" is a system for recording and storing various types of data.

[0079] This invention is a system for promoting safe driving and enabling adjustment of insurance premiums based on individual driving behavior. The terminal is installed in the vehicle and first uses an in-vehicle image acquisition device to capture traffic information of the road and surrounding environment in real time. This image acquisition device uses high-resolution sensors that can detect traffic signs, signals, pedestrians, and other vehicles.

[0080] The processing unit within the terminal analyzes the acquired information using computational methods executed with frameworks such as TENSORFLOW® and PyTorch. During this process, the system provides the driver with necessary instructions and information in real time. For example, in the event of a sudden lane change, the terminal issues voice and visual warnings.

[0081] Furthermore, the terminal collects data on the vehicle's movement characteristics. This includes data on the vehicle's acceleration, speed, and braking performance, using speed sensors and inertial measurement devices. Based on this data, the driver's driving skills are evaluated, and a safe driving index is calculated according to certain criteria.

[0082] The server receives data transmitted from the terminal and stores it in a database. Simultaneously, the server re-analyzes the stored data and generates driver risk management information. This information is used to adjust insurance premiums.

[0083] Users can review their driving style by receiving immediate feedback from the device. Furthermore, after driving, they can check their evaluated score and any changes in insurance premiums. For example, imagine a scenario where a user is driving on a highway and the device recognizes the distance to the vehicle in front and issues a warning to maintain a safe distance.

[0084] An example of a prompt using a generative AI model is, "Explain what visual and auditory feedback should be provided to the driver to maintain a safe distance from the vehicle ahead while driving on a highway." This allows the AI ​​to be instructed to generate specific feedback. In this way, specific methods are provided to promote safe and appropriate driving for the driver.

[0085] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0086] Step 1:

[0087] The terminal uses an in-vehicle image acquisition device to capture real-time traffic information of roads and the surrounding environment. Inputs include road video data, traffic signs, signals, and pedestrians. This data is initially processed and converted to an appropriate format for subsequent processing. Specifically, a high-resolution camera sensor captures images at a rate of several tens of frames per second.

[0088] Step 2:

[0089] The terminal analyzes the acquired video data using its internal processing unit. Captured image data is used as input, and based on this, the traffic environment is recognized using computational methods. Specifically, an artificial intelligence algorithm is used, and an object recognition model is employed to identify traffic signs, traffic lights, and other objects. As output, information on the identified objects is generated and passed on to the next step.

[0090] Step 3:

[0091] The terminal collects vehicle movement characteristic data and monitors the driver's behavior. Speed ​​sensor and brake operation information are used as input. Data calculations are performed based on the collected data to evaluate the driver's driving skills. In this process, a safe driving index is calculated. Specifically, sensors continuously record the vehicle's acceleration and speed, and the results are analyzed in real time.

[0092] Step 4:

[0093] The terminal provides real-time feedback to the driver based on the analysis results. The inputs used are analyzed object information and movement characteristic data. Based on this, the terminal issues voice and visual warnings and sends appropriate action instructions to the driver. Visual and auditory feedback is generated as output. Specifically, when the safe distance is exceeded, a warning sound or a warning message is emitted on the display.

[0094] Step 5:

[0095] The server receives data transmitted from the terminal and stores it in the database. Driving behavior data and analysis results are sent to the server as input. This data is organized and used for subsequent analysis. Specifically, data is uploaded to the server via a communication module and recorded in the database system.

[0096] Step 6:

[0097] The server uses accumulated data and a generating AI model to re-evaluate the driver's safe driving index. Past and current driving data are taken as input, and a detailed score analysis is performed using the AI ​​model. As output, driver risk management information is generated, contributing to insurance premium adjustments. Specifically, the driver's driving tendencies and risk profile are calculated.

[0098] Step 7:

[0099] After completing a drive, the user reviews the evaluation score and feedback displayed on the device. The input used is the driving data and its analysis. Based on this, the user gains information to improve their driving style. For example, specific advice such as "You tend to brake abruptly" might be displayed on the device.

[0100] (Application Example 1)

[0101] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0102] Drivers are required to pay close attention to their surroundings and traffic conditions when operating a vehicle, but a lack of such attention can lead to accidents. Furthermore, there is a lack of adequate systems for evaluating driving safety and reflecting the results in individual insurance premiums. To solve these problems and promote safe driving, a means of providing drivers with prompt and specific feedback is necessary.

[0103] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0104] In this invention, the server includes means for recognizing surrounding signs and traffic conditions using an in-vehicle camera, means for using an artificial intelligence algorithm to provide instructions to the driver based on the recognized signs and traffic conditions, and means for performing eye tracking and highlighting visual information. This allows the driver to be alerted to important matters in real time, enabling safe driving and appropriate insurance premium setting.

[0105] An "in-vehicle camera" is a camera device installed in a vehicle that captures the surrounding road environment and traffic conditions, and provides video data.

[0106] "Sign and traffic condition recognition" refers to the process of analyzing video data acquired from in-vehicle cameras to identify road signs, traffic lights, vehicles, pedestrians, and other similar elements.

[0107] "Means of using artificial intelligence algorithms" refers to computational processes realized through machine learning and deep learning technologies that are used to make judgments about traffic rules and provide instructions to drivers based on recognized data.

[0108] "Driving behavior data" refers to a set of data related to the driver's actions, such as vehicle speed, acceleration, and braking.

[0109] "Evaluating driving skills" involves analyzing acquired driving behavior data and quantifying and evaluating the degree of safe driving and driving skills.

[0110] "Eye-tracking" is a technology that tracks where a driver is looking and analyzes that data.

[0111] "Highlighting" is a method of making visual information stand out in order to draw the driver's particular attention.

[0112] "A means of issuing warnings using an audio output device" refers to a system that notifies the driver by voice when dangerous driving or situations requiring attention are detected.

[0113] "Information reflected in insurance premiums" refers to data used to adjust individual insurance premiums based on safe driving evaluation results.

[0114] "Means of sending to a storage device" refers to the process of sending analyzed data and evaluation results to a server or cloud storage for storage.

[0115] The system for implementing this invention consists of an in-vehicle camera, an artificial intelligence algorithm, an eye-tracking module, an audio and visual output device, a communication module, and a storage device. The integration of these elements makes it possible to monitor the driver's driving behavior in real time, evaluate driving skills, and promote safe driving.

[0116] The server captures images of the surrounding road environment through an in-vehicle camera and uses that data to recognize traffic signs, signals, pedestrians, and other vehicles. Based on this, it generates necessary instructions for the driver, which are then provided via display and voice. In particular, the use of libraries such as TensorFlow and OpenCV enables highly accurate image analysis.

[0117] The eye-tracking module tracks the driver's field of vision and highlights important information that is often overlooked. This allows for precise guidance of the driver's attention. For example, if the driver's gaze is not focused on a traffic sign ahead, the sign will be highlighted, and an audio alert will be issued.

[0118] Driving behavior data, including vehicle speed, acceleration, and braking information, is evaluated in real time by an artificial intelligence algorithm. A safe driving index is calculated, and these evaluation results are transmitted to storage for adjusting insurance premiums based on the degree of safe driving. This data is stored on a cloud server via a communication module and can be accessed as needed.

[0119] A concrete example would be a situation where, while driving on a highway, the driver misses a speed limit sign ahead. In such a scenario, the system visually highlights the sign and simultaneously provides a voice notification stating, "The speed limit is 80 km / h."

[0120] Examples of prompt statements include the following:

[0121] "When driving on a highway, if your eyes are not focused on road signs or traffic lights, an alert will be displayed. You will also be notified in advance when the traffic light is about to change from green to red."

[0122] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0123] Step 1:

[0124] The terminal uses an in-vehicle camera to capture the surrounding road environment in real time. The input is the video data acquired from the camera, and the output is a stream of this data. The video data is then sent to the next analysis step.

[0125] Step 2:

[0126] The server receives the video data acquired in Step 1 and uses an artificial intelligence algorithm to recognize traffic signs, traffic lights, pedestrians, lanes, vehicles ahead, etc. The input is video data, and the output is a list of the recognized information. TensorFlow and OpenCV libraries are used for this analysis.

[0127] Step 3:

[0128] The device uses an eye-tracking module to acquire the driver's gaze data. The input is the location of the gaze, and the output is the identification of the object the gaze is directed towards. This allows the device to determine the direction in which the driver's attention is focused.

[0129] Step 4:

[0130] The terminal integrates the data from steps 2 and 3 and highlights important signs and signals on the display if they are outside the user's line of sight. The input is recognition information and gaze data, and the output is the highlighting instruction.

[0131] Step 5:

[0132] The terminal will issue a warning via an audio output device if dangerous driving is detected. The input is the result of the driving behavior analysis, and the output is an audio warning. The user will receive an audio alert, such as "The traffic light ahead has turned red."

[0133] Step 6:

[0134] The server receives driving behavior data (speed, acceleration, braking, etc.) and calculates the driver's safe driving index. The input is driving behavior data, and the output is the safe driving index. A machine learning model is used for this evaluation.

[0135] Step 7:

[0136] The server sends the safe driving index calculated in step 6 to a storage device for use in adjusting insurance premiums. The input is the safe driving index, and the output is the updated insurance premium information. This information is stored on a cloud server and can be accessed at a later date.

[0137] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0138] The present invention is a vehicle-mounted system that simultaneously monitors the driver's driving behavior and emotional state, and has a configuration that combines an in-vehicle camera, artificial intelligence algorithm, emotion engine, various sensor groups, communication module, information provision device, and the like.

[0139] Terminal operation

[0140] The device uses an in-car camera to capture real-time video of the road environment and uses an artificial intelligence algorithm to recognize traffic signs, lanes, and vehicles ahead. Simultaneously, another camera and microphone capture the driver's facial expressions and speech, and an emotion engine estimates the user's emotional state. This information is used to evaluate the user's stress level, concentration level, and relaxation level while driving.

[0141] The collected driving and emotional data are analyzed within the device and provided to the user as timely feedback via voice and display. For example, if the device determines that the user is stressed, it will notify the user to reconsider their driving posture. Additionally, a driving skill score is visually displayed, along with the analysis results and emotional state.

[0142] Server operation

[0143] Data processed on the terminal is sent to the server. The server receives this data and stores the driving and emotional data in a database. Driving skills are scored considering the user's past data history and current emotional state, and the results are optimized for insurance premium adjustments.

[0144] The server feeds back the generated scores and emotional data to insurance companies, allowing them to adjust premiums based on this information. Users with high safe driving scores and who maintain a relaxed emotional state are offered benefits such as discounts on their insurance premiums.

[0145] User interaction

[0146] Users can receive real-time feedback while driving, which can help them manage their driving style and emotions. After finishing their drive, they can review the driving score and emotion report displayed on the device to identify specific areas for improvement.

[0147] For example, if the emotion engine detects that the user is becoming fatigued during long-distance driving, the device will display a message prompting them to take a break, supporting safe and comfortable driving. In this way, users can utilize emotional data in conjunction with their driving behavior to achieve safer and more effective driving.

[0148] The following describes the processing flow.

[0149] Step 1:

[0150] The device captures video data from the in-vehicle camera in real time. This data includes traffic signs, traffic lights, lanes, and traffic conditions ahead.

[0151] Step 2:

[0152] The device uses a separate camera and microphone to capture the user's facial expressions and voice, and inputs this data into the emotion engine. The emotion engine analyzes this data to identify the user's emotional state.

[0153] Step 3:

[0154] The terminal analyzes collected traffic data using an artificial intelligence algorithm and provides driving instructions to the user. For example, if it recognizes a speed limit sign, it notifies the user of that information via the display or voice.

[0155] Step 4:

[0156] The device continuously records the user's driving behavior data using speed and acceleration sensors. This includes the frequency of sudden acceleration, sudden braking, and steering maneuvers.

[0157] Step 5:

[0158] The device combines driving behavior and emotional state data to score the user's driving skills. Emotions such as tension and fatigue are taken into consideration, as they may affect the score.

[0159] Step 6:

[0160] The device provides real-time feedback to the user along with the scoring results. If the emotion engine determines that the user is fatigued, the device will verbally remind them to take a break.

[0161] Step 7:

[0162] The terminal periodically sends analyzed driving and emotional data to the server. The transmitted data is assigned a unique user ID and managed on the server.

[0163] Step 8:

[0164] Based on the data received from the terminal, the server analyzes driving behavior and emotional data in more detail and rescores how good the user's driving skills are.

[0165] Step 9:

[0166] The server provides the generated score to the insurance company, and adjusts the user's insurance premium based on that score. In this case, if the user maintains a low-stress and safe driving style, the insurance premium may be discounted.

[0167] Step 10:

[0168] Users review the score and sentiment report displayed on their device and consider how to improve for their next drive. This information is useful in encouraging continued safe driving.

[0169] (Example 2)

[0170] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0171] Conventional in-vehicle systems often evaluate drivers' driving skills and emotional states individually, which presents challenges in improving overall safety and providing flexible responses to users. Furthermore, the lack of mechanisms that directly reflect driving skills and emotional states in insurance premiums is also a problem.

[0172] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0173] In this invention, the server includes means for recognizing surrounding information and traffic conditions using an in-vehicle camera; means for using a machine learning algorithm to provide instructions to the user based on the recognized information and traffic conditions; means for collecting driving behavior data and emotional data, and analyzing the data to evaluate the driver's driving skills; means for generating data to reflect the evaluation results in insurance premiums; means for transmitting the analysis results and evaluation data to a remote storage device; means for providing an emotion analysis engine to predict the user's emotional state and provide appropriate feedback while driving; and means for monitoring the driving state in real time using multiple sensors. This enables comprehensive feedback and insurance premium adjustments based on the driver's skills and emotional state.

[0174] A "vehicle-mounted camera" is a device installed in a vehicle to acquire visual information about the surroundings.

[0175] A "machine learning algorithm" is a program or model that learns from large amounts of data and performs a specific task.

[0176] "Driving behavior data" refers to information collected about a driver's driving style and behavior.

[0177] "Emotional data" refers to information about the driver's psychological state inferred from their facial expressions, voice, and other biosignals.

[0178] "Evaluation" is the process of determining a driver's skills and adaptability based on collected data.

[0179] A "sentiment analysis engine" is a computer program that analyzes collected emotional data to estimate the driver's psychological state.

[0180] A "remote storage device" is an external storage device or database used to store data over a network.

[0181] "Real-time" refers to a situation or method in which data processing or responses occur instantaneously.

[0182] This invention aims to promote safe driving and optimize insurance premiums by monitoring and evaluating the driver's behavior and emotions through a system installed in the vehicle. The system mainly consists of a "terminal" and a "server".

[0183] Terminal operation

[0184] The terminal is installed in the vehicle and uses multiple sensors to collect driver behavior and emotional data in real time. Specifically, it uses an in-vehicle camera to record the road environment and the driver's facial expressions, and a voice detection device to collect the driver's speech. This data is analyzed using machine learning algorithms that run within the terminal. The video data acquired from the camera is used to recognize traffic signs and the behavior of other vehicles, and to provide necessary instructions to the driver. In addition, an emotion analysis engine is used to estimate the driver's psychological state and provide feedback according to the level of stress and fatigue.

[0185] For example, if a driver is fatigued from driving for a long time, the device will display a message prompting them to take a break. This allows the driver to accurately understand their situation and take appropriate action.

[0186] Server operation

[0187] The server is responsible for receiving and storing data transmitted from terminals. The server organizes and stores driving behavior data and emotional data in a database, and uses this data to evaluate the driver's driving skills. The evaluation is based on a generative AI model, comprehensively combining past data history with real-time emotional states.

[0188] The results of the driving skills evaluation are calculated as a driving score and provided to the insurance company from the server. The insurance company uses this information to adjust insurance premiums and applies discounts to drivers with high safe driving scores.

[0189] User interaction

[0190] While driving, users can receive feedback from the device, allowing them to appropriately adjust their driving behavior and emotional state. After driving, they can review the driving score and emotional report displayed on the device to understand specific areas for improvement and the next steps to take.

[0191] For example, a prompt message to be input to the generating AI model would be, "Analyze the driver's driving behavior and emotional state in real time and generate appropriate feedback." This would enable the driver to enjoy a safe and comfortable driving experience.

[0192] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0193] Step 1:

[0194] The terminal collects driver behavioral and emotional data using an in-vehicle camera and voice detection device. The inputs include video and audio data. Specifically, the camera captures road conditions and traffic signs, while the voice detection device captures the driver's voice and the in-vehicle acoustic environment. This data is then sent to the next analysis step.

[0195] Step 2:

[0196] The device applies machine learning algorithms to analyze the collected video and audio data. In this step, it recognizes traffic conditions and estimates the driver's emotional state from the data received as input. Specifically, it recognizes traffic signs and lane information from the video data, and estimates the current emotional state from the audio data and facial expressions using an emotion analysis engine. The output of this process is then compiled into feedback information for driver assistance.

[0197] Step 3:

[0198] The terminal provides real-time feedback to the driver based on the analysis results. The input here is the feedback information obtained in step 2. Specifically, it displays a driving skill score on the screen and provides advice via a voice assistant regarding adjustments to driving posture and the need for breaks. This output is immediately reflected in the driver's experience.

[0199] Step 4:

[0200] The server receives driving behavior data and emotional data transmitted from terminals and stores this data in its database. The input consists of analyzed information about the driver's behavior and emotions. The server organizes and stores this data to use as the basis for the next evaluation process.

[0201] Step 5:

[0202] The server evaluates the driver's driving skills using data stored in the database. The inputs used are stored driving behavior data and emotional data. A generative AI model is used to evaluate each driver based on their past driving records and emotional state, generating a driving skill score. This score is output and used as data for adjusting insurance premiums.

[0203] Step 6:

[0204] After driving, users review their driving score and emotional report displayed on their device. Input here includes evaluation results and feedback information. Based on this data, users identify specific areas for improvement in their driving skills and emotional control. This leads to safer and more comfortable driving.

[0205] (Application Example 2)

[0206] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0207] Conventional vehicle driving monitoring systems only monitor the driver's driving skills and surrounding traffic conditions, making it difficult to provide feedback that takes into account the driver's emotional state and stress levels. As a result, potential hazards may be overlooked, and improvements in safe driving may not be sufficiently achieved. Furthermore, there have been insufficient means to provide relaxation advice based on the driver's emotional state or to optimize insurance premiums.

[0208] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0209] In this invention, the server includes means for using an emotion analysis device to acquire the driver's facial expressions and voice and analyze their emotional state; means for suggesting relaxation methods based on the detected emotional state; and means for analyzing the acquired video data and emotional data and providing real-time feedback to the driver. This enables driving assistance that takes the driver's emotional state into account, providing a safer and more comfortable driving experience.

[0210] An "in-vehicle video acquisition device" is a device mounted on a vehicle to capture images of surrounding signs and traffic conditions.

[0211] An "artificial intelligence processing unit" is a device that executes algorithms to provide instructions to the driver based on recognized signs and traffic conditions.

[0212] "Operational data" refers to data about various operations and conditions recorded while a vehicle is in operation.

[0213] An "emotion analysis device" is a device that analyzes a driver's facial expressions and voice to estimate their emotional state.

[0214] "Means of suggesting relaxation methods" refers to methods of providing stress reduction in accordance with the detected emotional state of the driver.

[0215] An "information processing device" is a device or system for storing or transmitting analysis results and evaluation data.

[0216] The system implementing this invention aims to support safe and comfortable driving by comprehensively analyzing the driver's driving skills and emotional state. The system mainly consists of an in-vehicle video acquisition device, an artificial intelligence processing device, an emotion analysis device, and an information processing device.

[0217] The terminal uses an in-vehicle video acquisition system to recognize the surrounding traffic environment and signs in real time. This video data is analyzed via an artificial intelligence processing unit, which provides appropriate instructions based on the driver's driving behavior and traffic conditions. Furthermore, an emotion analysis device acquires and analyzes the driver's facial expressions and voice to estimate their current emotional state. Based on the estimated emotional state, feedback suggesting relaxation methods is provided via the terminal's display or audio output.

[0218] The server stores analysis results and driving skill evaluation data in an information processing device, and statistically analyzes the driver's past driving data and emotional state. The analysis results are used to support safe and effective driving, especially when the driver is experiencing excessive stress or fatigue, by suggesting appropriate rest times.

[0219] For example, if the emotion analysis device detects driver fatigue during a long-distance drive, the system may display a message such as, "We recommend taking a break at the next service area." In this way, the driving experience is improved.

[0220] Examples of prompt messages are as follows:

[0221] "Analyze the video and audio captured by the in-car camera and use AI to determine the driver's emotional state. If signs of fatigue or stress are detected, design code that displays a message advising the driver to stop driving and suggests appropriate relaxation methods."

[0222] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0223] Step 1:

[0224] The terminal uses an in-vehicle video acquisition device to capture real-time video of surrounding signs and traffic conditions. This video data is used as input, and an artificial intelligence processing unit performs image analysis to identify the location of signs and vehicles, as well as traffic conditions. The output is the recognized sign information and traffic condition data.

[0225] Step 2:

[0226] The terminal uses an emotion analysis device to simultaneously acquire images of the driver's facial expressions and audio. Using this data as input, an emotion analysis algorithm estimates the driver's emotional state. The analysis results are emotional indicators such as stress, fatigue, and relaxation levels, which are then output.

[0227] Step 3:

[0228] The server receives signage information, traffic data, and sentiment indicators transmitted from the terminal. Using this data as input, it comprehensively evaluates the driver's driving behavior and calculates a driving skill score. The evaluation process also includes comparison with past driving data and sentiment history. The output is feedback based on the latest driving skill score and sentiment state.

[0229] Step 4:

[0230] The terminal receives evaluation results from the server and provides feedback to the driver. Specifically, it displays messages on the screen suggesting ways to relax or take a break, and provides voice notifications if necessary. At this time, it again takes in emotional indicators as input and customizes the feedback content. The output is a reminder to the driver and encouragement of appropriate driving behavior.

[0231] Step 5:

[0232] Based on feedback from the device, users take actions such as improving their driving behavior or taking breaks. This feedback can be used to drive more safely and comfortably. Specifically, this includes reviewing driving posture and taking breaks at indicated times. Based on user responses, more data is accumulated and used for future analysis.

[0233] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0234] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0235] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0236] [Second Embodiment]

[0237] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0238] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0239] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0240] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0241] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0242] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0243] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0244] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0245] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0246] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0247] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0248] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0249] The present invention is a system installed in a vehicle driven by a driver, and consists of an in-vehicle camera, a processing unit using an artificial intelligence algorithm, a group of sensors for monitoring driving behavior and acquiring data, a communication module, and a display or audio output device for providing information.

[0250] Terminal operation

[0251] The device is installed in the vehicle and first captures real-time video of the road environment using an on-board camera. The captured video is sent to an internal processing unit and analyzed by an artificial intelligence algorithm. This analysis recognizes traffic signs, traffic lights, pedestrians, lanes, and vehicles ahead, enabling the system to provide the driver with necessary information and instructions. For example, if the speed limit is exceeded, a visual warning is displayed on the screen and an audio notification is given.

[0252] Furthermore, the terminal continuously collects various data such as vehicle speed, acceleration, and braking. This data is used to evaluate driving behavior, and items related to safe driving, particularly sudden acceleration and sudden braking, are scored.

[0253] Server operation

[0254] Data transmitted from the terminal is received and stored on the server. The server assigns a unique ID to each driver and manages the data based on this ID. The collected driving data is reanalyzed by a program that runs an artificial intelligence algorithm to accurately score driving skills. The score is calculated based on the driver's past driving data and the frequency of dangerous driving behaviors.

[0255] The server uses the analysis results to provide companies with data for adjusting insurance premiums. This adjustment is made by setting lower premiums for drivers with high scores (i.e., those who drive safely).

[0256] User interaction

[0257] Based on the visual and auditory feedback provided by the system, users can continuously review their driving style. After finishing their drive, they can check the score displayed on the terminal and see how their insurance premiums will change the following month. This motivates users to drive safely.

[0258] A concrete example would be a scenario where, while a driver is driving on a highway, the device recognizes the distance to the vehicle in front and issues a warning to maintain a safe distance. Through such prompts, users can naturally improve their driving habits.

[0259] The following describes the processing flow.

[0260] Step 1:

[0261] The device captures video data from the in-vehicle camera in real time. This data includes road signs, traffic lights, vehicles, pedestrians, and other information.

[0262] Step 2:

[0263] The device analyzes the captured video using an internal artificial intelligence algorithm. The algorithm uses image recognition technology to identify signs and traffic conditions, and provides the user with appropriate driving instructions based on this information.

[0264] Step 3:

[0265] The terminal simultaneously collects various sensor data related to driving behavior, such as vehicle speed, acceleration, and brake usage.

[0266] Step 4:

[0267] The device analyzes collected driving behavior data and calculates a safe driving score. The data analysis includes the frequency of sudden acceleration and braking, as well as whether or not the vehicle is speeding.

[0268] Step 5:

[0269] Based on the analysis results obtained, the device provides visual and auditory feedback to the user to encourage safe driving.

[0270] Step 6:

[0271] The device sends analysis data and scores to the server at regular intervals. The data is linked to a unique user ID.

[0272] Step 7:

[0273] The server receives data sent from the terminal and stores it in the database. Each user's driving history is managed.

[0274] Step 8:

[0275] The server performs additional analysis based on the received data and generates a score that re-evaluates driving behavior. This includes an analysis of driving trends over time.

[0276] Step 9:

[0277] The server provides the generated scores to the insurance company's system, which is then used to adjust the driver's insurance premiums.

[0278] Step 10:

[0279] Users review the feedback and scores displayed on their devices and consider areas for improvement for their next drive.

[0280] (Example 1)

[0281] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0282] In recent years, reducing traffic accidents and ensuring fair insurance premiums have become critical issues. Conventional systems make it difficult to objectively evaluate drivers' driving skills, sometimes resulting in inappropriate premium settings. Furthermore, mechanisms to promote safe driving by providing immediate feedback during driving are insufficient.

[0283] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0284] In this invention, the server includes means for recognizing surrounding traffic information using an in-vehicle image acquisition device, means for using a calculation method for providing instructions to the driver based on the recognized traffic information, and means for collecting vehicle movement characteristic data and analyzing the data to evaluate the driver's driving skills. Thereby, it becomes possible to promote safe driving and set appropriate insurance premiums based on individual driving behaviors.

[0285] The "in-vehicle image acquisition device" is a device mounted on a vehicle for photographing surrounding traffic information.

[0286] "Traffic information" is information including data on road environment, traffic signs, traffic lights, pedestrians, and other vehicles.

[0287] The "calculation method" refers to an algorithm or mathematical model used to analyze the recognized traffic information and provide instructions to the driver.

[0288] "Movement characteristic data" is data related to the movement of a vehicle, such as the vehicle's speed, acceleration, and brake operation.

[0289] "Driving skills" refer to the skills regarding how safely and efficiently a driver operates a vehicle.

[0290] "Evaluation" is a process of numerically or qualitatively analyzing and making a judgment on a driver's driving behavior based on the collected data.

[0291] "Risk management information" refers to data and indicators used for setting insurance premiums and evaluating risks.

[0292] An "accumulation device" is a system for recording and storing various data.

[0293] This invention is a system for promoting safe driving and enabling adjustment of insurance premiums based on individual driving behavior. The terminal is installed in the vehicle and first uses an in-vehicle image acquisition device to capture traffic information of the road and surrounding environment in real time. This image acquisition device uses high-resolution sensors that can detect traffic signs, signals, pedestrians, and other vehicles.

[0294] The processing unit within the terminal analyzes the acquired information using computational methods executed with frameworks such as TensorFlow and PyTorch. During this process, the system provides the driver with necessary instructions and information in real time. For example, in the event of a sudden lane change, the terminal issues voice and visual warnings.

[0295] Furthermore, the terminal collects data on the vehicle's movement characteristics. This includes data on the vehicle's acceleration, speed, and braking performance, using speed sensors and inertial measurement devices. Based on this data, the driver's driving skills are evaluated, and a safe driving index is calculated according to certain criteria.

[0296] The server receives data transmitted from the terminal and stores it in a database. Simultaneously, the server re-analyzes the stored data and generates driver risk management information. This information is used to adjust insurance premiums.

[0297] Users can review their driving style by receiving immediate feedback from the device. Furthermore, after driving, they can check their evaluated score and any changes in insurance premiums. For example, imagine a scenario where a user is driving on a highway and the device recognizes the distance to the vehicle in front and issues a warning to maintain a safe distance.

[0298] An example of a prompt using a generative AI model is, "Explain what visual and auditory feedback should be provided to the driver to maintain a safe distance from the vehicle ahead while driving on a highway." This allows the AI ​​to be instructed to generate specific feedback. In this way, specific methods are provided to promote safe and appropriate driving for the driver.

[0299] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0300] Step 1:

[0301] The terminal uses an in-vehicle image acquisition device to capture real-time traffic information of roads and the surrounding environment. Inputs include road video data, traffic signs, signals, and pedestrians. This data is initially processed and converted to an appropriate format for subsequent processing. Specifically, a high-resolution camera sensor captures images at a rate of several tens of frames per second.

[0302] Step 2:

[0303] The terminal analyzes the acquired video data using its internal processing unit. Captured image data is used as input, and based on this, the traffic environment is recognized using computational methods. Specifically, an artificial intelligence algorithm is used, and an object recognition model is employed to identify traffic signs, traffic lights, and other objects. As output, information on the identified objects is generated and passed on to the next step.

[0304] Step 3:

[0305] The terminal collects the vehicle's movement characteristic data and monitors the driver's behavior. As inputs, a speed sensor and brake operation information are used. Based on the collected data, data calculations are performed and the driver's driving skills are evaluated. In this process, a safe driving index is calculated. As a specific operation, the sensor continuously records the vehicle's acceleration and speed, and the results are analyzed in real time.

[0306] Step 4:

[0307] Based on the analysis results, the terminal provides real-time feedback to the driver. As inputs, the analyzed object information and movement characteristic data are used. Based on this, the terminal issues warnings audibly and visually and sends appropriate action instructions to the driver. As outputs, visual and auditory feedback are generated. Specifically, when the safe distance is exceeded, a warning sound and a warning message on the display are issued.

[0308] Step 5:

[0309] The server receives the data sent from the terminal and stores it in the database. As inputs, the driving behavior data and analysis results are sent to the server. These data are sorted and used for subsequent analysis. As a specific operation, the data is uploaded to the server via the communication module and recorded in the database system.

[0310] Step 6:

[0311] Based on the accumulated data, the server re-evaluates the driver's safe driving index by utilizing the generated AI model. As inputs, past driving data and current data are incorporated, and detailed score analysis is performed using the AI model. As outputs, the driver's risk management information is generated, contributing to the adjustment of insurance premiums. Specifically, the driver's driving tendency and risk profile are calculated.

[0312] Step 7:

[0313] After completing a drive, the user reviews the evaluation score and feedback displayed on the device. The input used is the driving data and its analysis. Based on this, the user gains information to improve their driving style. For example, specific advice such as "You tend to brake abruptly" might be displayed on the device.

[0314] (Application Example 1)

[0315] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0316] Drivers are required to pay close attention to their surroundings and traffic conditions when operating a vehicle, but a lack of such attention can lead to accidents. Furthermore, there is a lack of adequate systems for evaluating driving safety and reflecting the results in individual insurance premiums. To solve these problems and promote safe driving, a means of providing drivers with prompt and specific feedback is necessary.

[0317] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0318] In this invention, the server includes means for recognizing surrounding signs and traffic conditions using an in-vehicle camera, means for using an artificial intelligence algorithm to provide instructions to the driver based on the recognized signs and traffic conditions, and means for performing eye tracking and highlighting visual information. This allows the driver to be alerted to important matters in real time, enabling safe driving and appropriate insurance premium setting.

[0319] An "in-vehicle camera" is a camera device installed in a vehicle that captures the surrounding road environment and traffic conditions, and provides video data.

[0320] "Sign and traffic condition recognition" refers to the process of analyzing video data acquired from in-vehicle cameras to identify road signs, traffic lights, vehicles, pedestrians, and other similar elements.

[0321] "Means of using artificial intelligence algorithms" refers to computational processes realized through machine learning and deep learning technologies that are used to make judgments about traffic rules and provide instructions to drivers based on recognized data.

[0322] "Driving behavior data" refers to a set of data related to the driver's actions, such as vehicle speed, acceleration, and braking.

[0323] "Evaluating driving skills" involves analyzing acquired driving behavior data and quantifying and evaluating the degree of safe driving and driving skills.

[0324] "Eye-tracking" is a technology that tracks where a driver is looking and analyzes that data.

[0325] "Highlighting" is a method of making visual information stand out in order to draw the driver's particular attention.

[0326] "A means of issuing warnings using an audio output device" refers to a system that notifies the driver by voice when dangerous driving or situations requiring attention are detected.

[0327] "Information reflected in insurance premiums" refers to data used to adjust individual insurance premiums based on safe driving evaluation results.

[0328] "Means of sending to a storage device" refers to the process of sending analyzed data and evaluation results to a server or cloud storage for storage.

[0329] The system for implementing this invention consists of an in-vehicle camera, an artificial intelligence algorithm, an eye-tracking module, an audio and visual output device, a communication module, and a storage device. The integration of these elements makes it possible to monitor the driver's driving behavior in real time, evaluate driving skills, and promote safe driving.

[0330] The server captures images of the surrounding road environment through an in-vehicle camera and uses that data to recognize traffic signs, signals, pedestrians, and other vehicles. Based on this, it generates necessary instructions for the driver, which are then provided via display and voice. In particular, the use of libraries such as TensorFlow and OpenCV enables highly accurate image analysis.

[0331] The eye-tracking module tracks the driver's field of vision and highlights important information that is often overlooked. This allows for precise guidance of the driver's attention. For example, if the driver's gaze is not focused on a traffic sign ahead, the sign will be highlighted, and an audio alert will be issued.

[0332] Driving behavior data, including vehicle speed, acceleration, and braking information, is evaluated in real time by an artificial intelligence algorithm. A safe driving index is calculated, and these evaluation results are transmitted to storage for adjusting insurance premiums based on the degree of safe driving. This data is stored on a cloud server via a communication module and can be accessed as needed.

[0333] A concrete example would be a situation where, while driving on a highway, the driver misses a speed limit sign ahead. In such a scenario, the system visually highlights the sign and simultaneously provides a voice notification stating, "The speed limit is 80 km / h."

[0334] Examples of prompt statements include the following:

[0335] "When driving on a highway, if your eyes are not focused on road signs or traffic lights, an alert will be displayed. You will also be notified in advance when the traffic light is about to change from green to red."

[0336] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0337] Step 1:

[0338] The terminal uses an in-vehicle camera to capture the surrounding road environment in real time. The input is the video data acquired from the camera, and the output is a stream of this data. The video data is then sent to the next analysis step.

[0339] Step 2:

[0340] The server receives the video data acquired in Step 1 and uses an artificial intelligence algorithm to recognize traffic signs, traffic lights, pedestrians, lanes, vehicles ahead, etc. The input is video data, and the output is a list of the recognized information. TensorFlow and OpenCV libraries are used for this analysis.

[0341] Step 3:

[0342] The device uses an eye-tracking module to acquire the driver's gaze data. The input is the location of the gaze, and the output is the identification of the object the gaze is directed towards. This allows the device to determine the direction in which the driver's attention is focused.

[0343] Step 4:

[0344] The terminal integrates the data from steps 2 and 3 and highlights important signs and signals on the display if they are outside the user's line of sight. The input is recognition information and gaze data, and the output is the highlighting instruction.

[0345] Step 5:

[0346] The terminal will issue a warning via an audio output device if dangerous driving is detected. The input is the result of the driving behavior analysis, and the output is an audio warning. The user will receive an audio alert, such as "The traffic light ahead has turned red."

[0347] Step 6:

[0348] The server receives driving behavior data (speed, acceleration, braking, etc.) and calculates the driver's safe driving index. The input is driving behavior data, and the output is the safe driving index. A machine learning model is used for this evaluation.

[0349] Step 7:

[0350] The server sends the safe driving index calculated in step 6 to a storage device for use in adjusting insurance premiums. The input is the safe driving index, and the output is the updated insurance premium information. This information is stored on a cloud server and can be accessed at a later date.

[0351] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0352] The present invention is a vehicle-mounted system that simultaneously monitors the driver's driving behavior and emotional state, and has a configuration that combines an in-vehicle camera, artificial intelligence algorithm, emotion engine, various sensor groups, communication module, information provision device, and the like.

[0353] Terminal operation

[0354] The device uses an in-car camera to capture real-time video of the road environment and uses an artificial intelligence algorithm to recognize traffic signs, lanes, and vehicles ahead. Simultaneously, another camera and microphone capture the driver's facial expressions and speech, and an emotion engine estimates the user's emotional state. This information is used to evaluate the user's stress level, concentration level, and relaxation level while driving.

[0355] The collected driving and emotional data are analyzed within the device and provided to the user as timely feedback via voice and display. For example, if the device determines that the user is stressed, it will notify the user to reconsider their driving posture. Additionally, a driving skill score is visually displayed, along with the analysis results and emotional state.

[0356] Server operation

[0357] Data processed on the terminal is sent to the server. The server receives this data and stores the driving and emotional data in a database. Driving skills are scored considering the user's past data history and current emotional state, and the results are optimized for insurance premium adjustments.

[0358] The server feeds back the generated scores and emotional data to insurance companies, allowing them to adjust premiums based on this information. Users with high safe driving scores and who maintain a relaxed emotional state are offered benefits such as discounts on their insurance premiums.

[0359] User interaction

[0360] Users can receive real-time feedback while driving, which can help them manage their driving style and emotions. After finishing their drive, they can review the driving score and emotion report displayed on the device to identify specific areas for improvement.

[0361] For example, if the emotion engine detects that the user is becoming fatigued during long-distance driving, the device will display a message prompting them to take a break, supporting safe and comfortable driving. In this way, users can utilize emotional data in conjunction with their driving behavior to achieve safer and more effective driving.

[0362] The following describes the processing flow.

[0363] Step 1:

[0364] The device captures video data from the in-vehicle camera in real time. This data includes traffic signs, traffic lights, lanes, and traffic conditions ahead.

[0365] Step 2:

[0366] The device uses a separate camera and microphone to capture the user's facial expressions and voice, and inputs this data into the emotion engine. The emotion engine analyzes this data to identify the user's emotional state.

[0367] Step 3:

[0368] The terminal analyzes collected traffic data using an artificial intelligence algorithm and provides driving instructions to the user. For example, if it recognizes a speed limit sign, it notifies the user of that information via the display or voice.

[0369] Step 4:

[0370] The device continuously records the user's driving behavior data using speed and acceleration sensors. This includes the frequency of sudden acceleration, sudden braking, and steering maneuvers.

[0371] Step 5:

[0372] The device combines driving behavior and emotional state data to score the user's driving skills. Emotions such as tension and fatigue are taken into consideration, as they may affect the score.

[0373] Step 6:

[0374] The device provides real-time feedback to the user along with the scoring results. If the emotion engine determines that the user is fatigued, the device will verbally remind them to take a break.

[0375] Step 7:

[0376] The terminal periodically sends analyzed driving and emotional data to the server. The transmitted data is assigned a unique user ID and managed on the server.

[0377] Step 8:

[0378] Based on the data received from the terminal, the server analyzes driving behavior and emotional data in more detail and rescores how good the user's driving skills are.

[0379] Step 9:

[0380] The server provides the generated score to the insurance company, and adjusts the user's insurance premium based on that score. In this case, if the user maintains a low-stress and safe driving style, the insurance premium may be discounted.

[0381] Step 10:

[0382] Users review the score and sentiment report displayed on their device and consider how to improve for their next drive. This information is useful in encouraging continued safe driving.

[0383] (Example 2)

[0384] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0385] Conventional in-vehicle systems often evaluate drivers' driving skills and emotional states individually, which presents challenges in improving overall safety and providing flexible responses to users. Furthermore, the lack of mechanisms that directly reflect driving skills and emotional states in insurance premiums is also a problem.

[0386] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0387] In this invention, the server includes means for recognizing surrounding information and traffic conditions using an in-vehicle camera; means for using a machine learning algorithm to provide instructions to the user based on the recognized information and traffic conditions; means for collecting driving behavior data and emotional data, and analyzing the data to evaluate the driver's driving skills; means for generating data to reflect the evaluation results in insurance premiums; means for transmitting the analysis results and evaluation data to a remote storage device; means for providing an emotion analysis engine to predict the user's emotional state and provide appropriate feedback while driving; and means for monitoring the driving state in real time using multiple sensors. This enables comprehensive feedback and insurance premium adjustments based on the driver's skills and emotional state.

[0388] A "vehicle-mounted camera" is a device installed in a vehicle to acquire visual information about the surroundings.

[0389] A "machine learning algorithm" is a program or model that learns from large amounts of data and performs a specific task.

[0390] "Driving behavior data" refers to information collected about a driver's driving style and behavior.

[0391] "Emotional data" refers to information about the driver's psychological state inferred from their facial expressions, voice, and other biosignals.

[0392] "Evaluation" is the process of determining a driver's skills and adaptability based on collected data.

[0393] A "sentiment analysis engine" is a computer program that analyzes collected emotional data to estimate the driver's psychological state.

[0394] A "remote storage device" is an external storage device or database used to store data over a network.

[0395] "Real-time" refers to a situation or method in which data processing or responses occur instantaneously.

[0396] This invention aims to promote safe driving and optimize insurance premiums by monitoring and evaluating the driver's behavior and emotions through a system installed in the vehicle. The system mainly consists of a "terminal" and a "server".

[0397] Terminal operation

[0398] The terminal is installed in the vehicle and uses multiple sensors to collect driver behavior and emotional data in real time. Specifically, it uses an in-vehicle camera to record the road environment and the driver's facial expressions, and a voice detection device to collect the driver's speech. This data is analyzed using machine learning algorithms that run within the terminal. The video data acquired from the camera is used to recognize traffic signs and the behavior of other vehicles, and to provide necessary instructions to the driver. In addition, an emotion analysis engine is used to estimate the driver's psychological state and provide feedback according to the level of stress and fatigue.

[0399] For example, if a driver is fatigued from driving for a long time, the device will display a message prompting them to take a break. This allows the driver to accurately understand their situation and take appropriate action.

[0400] Server operation

[0401] The server is responsible for receiving and storing data transmitted from terminals. The server organizes and stores driving behavior data and emotional data in a database, and uses this data to evaluate the driver's driving skills. The evaluation is based on a generative AI model, comprehensively combining past data history with real-time emotional states.

[0402] The results of the driving skills evaluation are calculated as a driving score and provided to the insurance company from the server. The insurance company uses this information to adjust insurance premiums and applies discounts to drivers with high safe driving scores.

[0403] User interaction

[0404] While driving, users can receive feedback from the device, allowing them to appropriately adjust their driving behavior and emotional state. After driving, they can review the driving score and emotional report displayed on the device to understand specific areas for improvement and the next steps to take.

[0405] For example, a prompt message to be input to the generating AI model would be, "Analyze the driver's driving behavior and emotional state in real time and generate appropriate feedback." This would enable the driver to enjoy a safe and comfortable driving experience.

[0406] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0407] Step 1:

[0408] The terminal collects driver behavioral and emotional data using an in-vehicle camera and voice detection device. The inputs include video and audio data. Specifically, the camera captures road conditions and traffic signs, while the voice detection device captures the driver's voice and the in-vehicle acoustic environment. This data is then sent to the next analysis step.

[0409] Step 2:

[0410] The device applies machine learning algorithms to analyze the collected video and audio data. In this step, it recognizes traffic conditions and estimates the driver's emotional state from the data received as input. Specifically, it recognizes traffic signs and lane information from the video data, and estimates the current emotional state from the audio data and facial expressions using an emotion analysis engine. The output of this process is then compiled into feedback information for driver assistance.

[0411] Step 3:

[0412] The terminal provides real-time feedback to the driver based on the analysis results. The input here is the feedback information obtained in step 2. Specifically, it displays a driving skill score on the screen and provides advice via a voice assistant regarding adjustments to driving posture and the need for breaks. This output is immediately reflected in the driver's experience.

[0413] Step 4:

[0414] The server receives driving behavior data and emotional data transmitted from terminals and stores this data in its database. The input consists of analyzed information about the driver's behavior and emotions. The server organizes and stores this data to use as the basis for the next evaluation process.

[0415] Step 5:

[0416] The server evaluates the driver's driving skills using data stored in the database. The inputs used are stored driving behavior data and emotional data. A generative AI model is used to evaluate each driver based on their past driving records and emotional state, generating a driving skill score. This score is output and used as data for adjusting insurance premiums.

[0417] Step 6:

[0418] After driving, users review their driving score and emotional report displayed on their device. Input here includes evaluation results and feedback information. Based on this data, users identify specific areas for improvement in their driving skills and emotional control. This leads to safer and more comfortable driving.

[0419] (Application Example 2)

[0420] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0421] Conventional vehicle driving monitoring systems only monitor the driver's driving skills and surrounding traffic conditions, making it difficult to provide feedback that takes into account the driver's emotional state and stress levels. As a result, potential hazards may be overlooked, and improvements in safe driving may not be sufficiently achieved. Furthermore, there have been insufficient means to provide relaxation advice based on the driver's emotional state or to optimize insurance premiums.

[0422] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0423] In this invention, the server includes means for using an emotion analysis device to acquire the driver's facial expressions and voice and analyze their emotional state; means for suggesting relaxation methods based on the detected emotional state; and means for analyzing the acquired video data and emotional data and providing real-time feedback to the driver. This enables driving assistance that takes the driver's emotional state into account, providing a safer and more comfortable driving experience.

[0424] An "in-vehicle video acquisition device" is a device mounted on a vehicle to capture images of surrounding signs and traffic conditions.

[0425] An "artificial intelligence processing unit" is a device that executes algorithms to provide instructions to the driver based on recognized signs and traffic conditions.

[0426] "Operational data" refers to data about various operations and conditions recorded while a vehicle is in operation.

[0427] An "emotion analysis device" is a device that analyzes a driver's facial expressions and voice to estimate their emotional state.

[0428] "Means of suggesting relaxation methods" refers to methods of providing stress reduction in accordance with the detected emotional state of the driver.

[0429] An "information processing device" is a device or system for storing or transmitting analysis results and evaluation data.

[0430] The system implementing this invention aims to support safe and comfortable driving by comprehensively analyzing the driver's driving skills and emotional state. The system mainly consists of an in-vehicle video acquisition device, an artificial intelligence processing device, an emotion analysis device, and an information processing device.

[0431] The terminal uses an in-vehicle video acquisition system to recognize the surrounding traffic environment and signs in real time. This video data is analyzed via an artificial intelligence processing unit, which provides appropriate instructions based on the driver's driving behavior and traffic conditions. Furthermore, an emotion analysis device acquires and analyzes the driver's facial expressions and voice to estimate their current emotional state. Based on the estimated emotional state, feedback suggesting relaxation methods is provided via the terminal's display or audio output.

[0432] The server stores analysis results and driving skill evaluation data in an information processing device, and statistically analyzes the driver's past driving data and emotional state. The analysis results are used to support safe and effective driving, especially when the driver is experiencing excessive stress or fatigue, by suggesting appropriate rest times.

[0433] For example, if the emotion analysis device detects driver fatigue during a long-distance drive, the system may display a message such as, "We recommend taking a break at the next service area." In this way, the driving experience is improved.

[0434] Examples of prompt messages are as follows:

[0435] "Analyze the video and audio captured by the in-car camera and use AI to determine the driver's emotional state. If signs of fatigue or stress are detected, design code that displays a message advising the driver to stop driving and suggests appropriate relaxation methods."

[0436] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0437] Step 1:

[0438] The terminal uses an in-vehicle video acquisition device to capture real-time video of surrounding signs and traffic conditions. This video data is used as input, and an artificial intelligence processing unit performs image analysis to identify the location of signs and vehicles, as well as traffic conditions. The output is the recognized sign information and traffic condition data.

[0439] Step 2:

[0440] The terminal uses an emotion analysis device to simultaneously acquire images of the driver's facial expressions and audio. Using this data as input, an emotion analysis algorithm estimates the driver's emotional state. The analysis results are emotional indicators such as stress, fatigue, and relaxation levels, which are then output.

[0441] Step 3:

[0442] The server receives signage information, traffic data, and sentiment indicators transmitted from the terminal. Using this data as input, it comprehensively evaluates the driver's driving behavior and calculates a driving skill score. The evaluation process also includes comparison with past driving data and sentiment history. The output is feedback based on the latest driving skill score and sentiment state.

[0443] Step 4:

[0444] The terminal receives evaluation results from the server and provides feedback to the driver. Specifically, it displays messages on the screen suggesting ways to relax or take a break, and provides voice notifications if necessary. At this time, it again takes in emotional indicators as input and customizes the feedback content. The output is a reminder to the driver and encouragement of appropriate driving behavior.

[0445] Step 5:

[0446] Based on feedback from the device, users take actions such as improving their driving behavior or taking breaks. This feedback can be used to drive more safely and comfortably. Specifically, this includes reviewing driving posture and taking breaks at indicated times. Based on user responses, more data is accumulated and used for future analysis.

[0447] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0448] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0449] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0450] [Third Embodiment]

[0451] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0452] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0453] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0454] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0455] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0456] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0457] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0458] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0459] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0460] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0461] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0462] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0463] The present invention is a system installed in a vehicle driven by a driver, and consists of an in-vehicle camera, a processing unit using an artificial intelligence algorithm, a group of sensors for monitoring driving behavior and acquiring data, a communication module, and a display or audio output device for providing information.

[0464] Terminal operation

[0465] The device is installed in the vehicle and first captures real-time video of the road environment using an on-board camera. The captured video is sent to an internal processing unit and analyzed by an artificial intelligence algorithm. This analysis recognizes traffic signs, traffic lights, pedestrians, lanes, and vehicles ahead, enabling the system to provide the driver with necessary information and instructions. For example, if the speed limit is exceeded, a visual warning is displayed on the screen and an audio notification is given.

[0466] Furthermore, the terminal continuously collects various data such as vehicle speed, acceleration, and braking. This data is used to evaluate driving behavior, and items related to safe driving, particularly sudden acceleration and sudden braking, are scored.

[0467] Server operation

[0468] Data transmitted from the terminal is received and stored on the server. The server assigns a unique ID to each driver and manages the data based on this ID. The collected driving data is reanalyzed by a program that runs an artificial intelligence algorithm to accurately score driving skills. The score is calculated based on the driver's past driving data and the frequency of dangerous driving behaviors.

[0469] The server uses the analysis results to provide companies with data for adjusting insurance premiums. This adjustment is made by setting lower premiums for drivers with high scores (i.e., those who drive safely).

[0470] User interaction

[0471] Based on the visual and auditory feedback provided by the system, users can continuously review their driving style. After finishing their drive, they can check the score displayed on the terminal and see how their insurance premiums will change the following month. This motivates users to drive safely.

[0472] A concrete example would be a scenario where, while a driver is driving on a highway, the device recognizes the distance to the vehicle in front and issues a warning to maintain a safe distance. Through such prompts, users can naturally improve their driving habits.

[0473] The following describes the processing flow.

[0474] Step 1:

[0475] The device captures video data from the in-vehicle camera in real time. This data includes road signs, traffic lights, vehicles, pedestrians, and other information.

[0476] Step 2:

[0477] The device analyzes the captured video using an internal artificial intelligence algorithm. The algorithm uses image recognition technology to identify signs and traffic conditions, and provides the user with appropriate driving instructions based on this information.

[0478] Step 3:

[0479] The terminal simultaneously collects various sensor data related to driving behavior, such as vehicle speed, acceleration, and brake usage.

[0480] Step 4:

[0481] The device analyzes collected driving behavior data and calculates a safe driving score. The data analysis includes the frequency of sudden acceleration and braking, as well as whether or not the vehicle is speeding.

[0482] Step 5:

[0483] Based on the analysis results obtained, the device provides visual and auditory feedback to the user to encourage safe driving.

[0484] Step 6:

[0485] The device sends analysis data and scores to the server at regular intervals. The data is linked to a unique user ID.

[0486] Step 7:

[0487] The server receives data sent from the terminal and stores it in the database. Each user's driving history is managed.

[0488] Step 8:

[0489] The server performs additional analysis based on the received data and generates a score that re-evaluates driving behavior. This includes an analysis of driving trends over time.

[0490] Step 9:

[0491] The server provides the generated scores to the insurance company's system, which is then used to adjust the driver's insurance premiums.

[0492] Step 10:

[0493] Users review the feedback and scores displayed on their devices and consider areas for improvement for their next drive.

[0494] (Example 1)

[0495] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0496] In recent years, reducing traffic accidents and ensuring fair insurance premiums have become critical issues. Conventional systems make it difficult to objectively evaluate drivers' driving skills, sometimes resulting in inappropriate premium settings. Furthermore, mechanisms to promote safe driving by providing immediate feedback during driving are insufficient.

[0497] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0498] In this invention, the server includes means for recognizing surrounding traffic information using an in-vehicle image acquisition device, means for using a computational method to provide instructions to the driver based on the recognized traffic information, and means for collecting vehicle movement characteristic data and analyzing the data to evaluate the driver's driving skills. This makes it possible to promote safe driving and set appropriate insurance premiums based on individual driving behavior.

[0499] An "in-vehicle image acquisition device" is a device mounted on a vehicle to capture surrounding traffic information.

[0500] "Traffic information" refers to information that includes data on road conditions, traffic signs, traffic lights, pedestrians, and other vehicles.

[0501] "Computational methods" refer to algorithms and mathematical models used to analyze recognized traffic information and provide instructions to drivers.

[0502] "Movement characteristics data" refers to data related to the movement of a vehicle, such as vehicle speed, acceleration, and braking operation.

[0503] "Driving skills" refer to the ability of a driver to operate a vehicle safely and efficiently.

[0504] "Evaluation" is the process of analyzing a driver's driving behavior numerically or qualitatively based on collected data and making a judgment.

[0505] "Risk management information" refers to the data and indicators used in setting insurance premiums and assessing risks.

[0506] A "data storage device" is a system for recording and storing various types of data.

[0507] This invention is a system for promoting safe driving and enabling adjustment of insurance premiums based on individual driving behavior. The terminal is installed in the vehicle and first uses an in-vehicle image acquisition device to capture traffic information of the road and surrounding environment in real time. This image acquisition device uses high-resolution sensors that can detect traffic signs, signals, pedestrians, and other vehicles.

[0508] The processing unit within the terminal analyzes the acquired information using computational methods executed with frameworks such as TensorFlow and PyTorch. During this process, the system provides the driver with necessary instructions and information in real time. For example, in the event of a sudden lane change, the terminal issues voice and visual warnings.

[0509] Furthermore, the terminal collects data on the vehicle's movement characteristics. This includes data on the vehicle's acceleration, speed, and braking performance, using speed sensors and inertial measurement devices. Based on this data, the driver's driving skills are evaluated, and a safe driving index is calculated according to certain criteria.

[0510] The server receives data transmitted from the terminal and stores it in a database. Simultaneously, the server re-analyzes the stored data and generates driver risk management information. This information is used to adjust insurance premiums.

[0511] Users can review their driving style by receiving immediate feedback from the device. Furthermore, after driving, they can check their evaluated score and any changes in insurance premiums. For example, imagine a scenario where a user is driving on a highway and the device recognizes the distance to the vehicle in front and issues a warning to maintain a safe distance.

[0512] An example of a prompt using a generative AI model is, "Explain what visual and auditory feedback should be provided to the driver to maintain a safe distance from the vehicle ahead while driving on a highway." This allows the AI ​​to be instructed to generate specific feedback. In this way, specific methods are provided to promote safe and appropriate driving for the driver.

[0513] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0514] Step 1:

[0515] The terminal uses an in-vehicle image acquisition device to capture real-time traffic information of roads and the surrounding environment. Inputs include road video data, traffic signs, signals, and pedestrians. This data is initially processed and converted to an appropriate format for subsequent processing. Specifically, a high-resolution camera sensor captures images at a rate of several tens of frames per second.

[0516] Step 2:

[0517] The terminal analyzes the acquired video data using its internal processing unit. Captured image data is used as input, and based on this, the traffic environment is recognized using computational methods. Specifically, an artificial intelligence algorithm is used, and an object recognition model is employed to identify traffic signs, traffic lights, and other objects. As output, information on the identified objects is generated and passed on to the next step.

[0518] Step 3:

[0519] The terminal collects vehicle movement characteristic data and monitors the driver's behavior. Speed ​​sensor and brake operation information are used as input. Data calculations are performed based on the collected data to evaluate the driver's driving skills. In this process, a safe driving index is calculated. Specifically, sensors continuously record the vehicle's acceleration and speed, and the results are analyzed in real time.

[0520] Step 4:

[0521] The terminal provides real-time feedback to the driver based on the analysis results. The inputs used are analyzed object information and movement characteristic data. Based on this, the terminal issues voice and visual warnings and sends appropriate action instructions to the driver. Visual and auditory feedback is generated as output. Specifically, when the safe distance is exceeded, a warning sound or a warning message is emitted on the display.

[0522] Step 5:

[0523] The server receives data transmitted from the terminal and stores it in the database. Driving behavior data and analysis results are sent to the server as input. This data is organized and used for subsequent analysis. Specifically, data is uploaded to the server via a communication module and recorded in the database system.

[0524] Step 6:

[0525] The server uses accumulated data and a generating AI model to re-evaluate the driver's safe driving index. Past and current driving data are taken as input, and a detailed score analysis is performed using the AI ​​model. As output, driver risk management information is generated, contributing to insurance premium adjustments. Specifically, the driver's driving tendencies and risk profile are calculated.

[0526] Step 7:

[0527] After completing a drive, the user reviews the evaluation score and feedback displayed on the device. The input used is the driving data and its analysis. Based on this, the user gains information to improve their driving style. For example, specific advice such as "You tend to brake abruptly" might be displayed on the device.

[0528] (Application Example 1)

[0529] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0530] Drivers are required to pay close attention to their surroundings and traffic conditions when operating a vehicle, but a lack of such attention can lead to accidents. Furthermore, there is a lack of adequate systems for evaluating driving safety and reflecting the results in individual insurance premiums. To solve these problems and promote safe driving, a means of providing drivers with prompt and specific feedback is necessary.

[0531] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0532] In this invention, the server includes means for recognizing surrounding signs and traffic conditions using an in-vehicle camera, means for using an artificial intelligence algorithm to provide instructions to the driver based on the recognized signs and traffic conditions, and means for performing eye tracking and highlighting visual information. This allows the driver to be alerted to important matters in real time, enabling safe driving and appropriate insurance premium setting.

[0533] An "in-vehicle camera" is a camera device installed in a vehicle that captures the surrounding road environment and traffic conditions, and provides video data.

[0534] "Sign and traffic condition recognition" refers to the process of analyzing video data acquired from in-vehicle cameras to identify road signs, traffic lights, vehicles, pedestrians, and other similar elements.

[0535] "Means of using artificial intelligence algorithms" refers to computational processes realized through machine learning and deep learning technologies that are used to make judgments about traffic rules and provide instructions to drivers based on recognized data.

[0536] "Driving behavior data" refers to a set of data related to the driver's actions, such as vehicle speed, acceleration, and braking.

[0537] "Evaluating driving skills" involves analyzing acquired driving behavior data and quantifying and evaluating the degree of safe driving and driving skills.

[0538] "Eye-tracking" is a technology that tracks where a driver is looking and analyzes that data.

[0539] "Highlighting" is a method of making visual information stand out in order to draw the driver's particular attention.

[0540] "A means of issuing warnings using an audio output device" refers to a system that notifies the driver by voice when dangerous driving or situations requiring attention are detected.

[0541] "Information reflected in insurance premiums" refers to data used to adjust individual insurance premiums based on safe driving evaluation results.

[0542] "Means of sending to a storage device" refers to the process of sending analyzed data and evaluation results to a server or cloud storage for storage.

[0543] The system for implementing this invention consists of an in-vehicle camera, an artificial intelligence algorithm, an eye-tracking module, an audio and visual output device, a communication module, and a storage device. The integration of these elements makes it possible to monitor the driver's driving behavior in real time, evaluate driving skills, and promote safe driving.

[0544] The server captures images of the surrounding road environment through an in-vehicle camera and uses that data to recognize traffic signs, signals, pedestrians, and other vehicles. Based on this, it generates necessary instructions for the driver, which are then provided via display and voice. In particular, the use of libraries such as TensorFlow and OpenCV enables highly accurate image analysis.

[0545] The eye-tracking module tracks the driver's field of vision and highlights important information that is often overlooked. This allows for precise guidance of the driver's attention. For example, if the driver's gaze is not focused on a traffic sign ahead, the sign will be highlighted, and an audio alert will be issued.

[0546] Driving behavior data, including vehicle speed, acceleration, and braking information, is evaluated in real time by an artificial intelligence algorithm. A safe driving index is calculated, and these evaluation results are transmitted to storage for adjusting insurance premiums based on the degree of safe driving. This data is stored on a cloud server via a communication module and can be accessed as needed.

[0547] A concrete example would be a situation where, while driving on a highway, the driver misses a speed limit sign ahead. In such a scenario, the system visually highlights the sign and simultaneously provides a voice notification stating, "The speed limit is 80 km / h."

[0548] Examples of prompt statements include the following:

[0549] "When driving on a highway, if your eyes are not focused on road signs or traffic lights, an alert will be displayed. You will also be notified in advance when the traffic light is about to change from green to red."

[0550] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0551] Step 1:

[0552] The terminal uses an in-vehicle camera to capture the surrounding road environment in real time. The input is the video data acquired from the camera, and the output is a stream of this data. The video data is then sent to the next analysis step.

[0553] Step 2:

[0554] The server receives the video data acquired in Step 1 and uses an artificial intelligence algorithm to recognize traffic signs, traffic lights, pedestrians, lanes, vehicles ahead, etc. The input is video data, and the output is a list of the recognized information. TensorFlow and OpenCV libraries are used for this analysis.

[0555] Step 3:

[0556] The device uses an eye-tracking module to acquire the driver's gaze data. The input is the location of the gaze, and the output is the identification of the object the gaze is directed towards. This allows the device to determine the direction in which the driver's attention is focused.

[0557] Step 4:

[0558] The terminal integrates the data from steps 2 and 3 and highlights important signs and signals on the display if they are outside the user's line of sight. The input is recognition information and gaze data, and the output is the highlighting instruction.

[0559] Step 5:

[0560] The terminal will issue a warning via an audio output device if dangerous driving is detected. The input is the result of the driving behavior analysis, and the output is an audio warning. The user will receive an audio alert, such as "The traffic light ahead has turned red."

[0561] Step 6:

[0562] The server receives driving behavior data (speed, acceleration, braking, etc.) and calculates the driver's safe driving index. The input is driving behavior data, and the output is the safe driving index. A machine learning model is used for this evaluation.

[0563] Step 7:

[0564] The server sends the safe driving index calculated in step 6 to a storage device for use in adjusting insurance premiums. The input is the safe driving index, and the output is the updated insurance premium information. This information is stored on a cloud server and can be accessed at a later date.

[0565] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0566] The present invention is a vehicle-mounted system that simultaneously monitors the driver's driving behavior and emotional state, and has a configuration that combines an in-vehicle camera, artificial intelligence algorithm, emotion engine, various sensor groups, communication module, information provision device, and the like.

[0567] Terminal operation

[0568] The device uses an in-car camera to capture real-time video of the road environment and uses an artificial intelligence algorithm to recognize traffic signs, lanes, and vehicles ahead. Simultaneously, another camera and microphone capture the driver's facial expressions and speech, and an emotion engine estimates the user's emotional state. This information is used to evaluate the user's stress level, concentration level, and relaxation level while driving.

[0569] The collected driving and emotional data are analyzed within the device and provided to the user as timely feedback via voice and display. For example, if the device determines that the user is stressed, it will notify the user to reconsider their driving posture. Additionally, a driving skill score is visually displayed, along with the analysis results and emotional state.

[0570] Server operation

[0571] Data processed on the terminal is sent to the server. The server receives this data and stores the driving and emotional data in a database. Driving skills are scored considering the user's past data history and current emotional state, and the results are optimized for insurance premium adjustments.

[0572] The server feeds back the generated scores and emotional data to insurance companies, allowing them to adjust premiums based on this information. Users with high safe driving scores and who maintain a relaxed emotional state are offered benefits such as discounts on their insurance premiums.

[0573] User interaction

[0574] Users can receive real-time feedback while driving, which can help them manage their driving style and emotions. After finishing their drive, they can review the driving score and emotion report displayed on the device to identify specific areas for improvement.

[0575] For example, if the emotion engine detects that the user is becoming fatigued during long-distance driving, the device will display a message prompting them to take a break, supporting safe and comfortable driving. In this way, users can utilize emotional data in conjunction with their driving behavior to achieve safer and more effective driving.

[0576] The following describes the processing flow.

[0577] Step 1:

[0578] The device captures video data from the in-vehicle camera in real time. This data includes traffic signs, traffic lights, lanes, and traffic conditions ahead.

[0579] Step 2:

[0580] The device uses a separate camera and microphone to capture the user's facial expressions and voice, and inputs this data into the emotion engine. The emotion engine analyzes this data to identify the user's emotional state.

[0581] Step 3:

[0582] The terminal analyzes collected traffic data using an artificial intelligence algorithm and provides driving instructions to the user. For example, if it recognizes a speed limit sign, it notifies the user of that information via the display or voice.

[0583] Step 4:

[0584] The device continuously records the user's driving behavior data using speed and acceleration sensors. This includes the frequency of sudden acceleration, sudden braking, and steering maneuvers.

[0585] Step 5:

[0586] The device combines driving behavior and emotional state data to score the user's driving skills. Emotions such as tension and fatigue are taken into consideration, as they may affect the score.

[0587] Step 6:

[0588] The device provides real-time feedback to the user along with the scoring results. If the emotion engine determines that the user is fatigued, the device will verbally remind them to take a break.

[0589] Step 7:

[0590] The terminal periodically sends analyzed driving and emotional data to the server. The transmitted data is assigned a unique user ID and managed on the server.

[0591] Step 8:

[0592] Based on the data received from the terminal, the server analyzes driving behavior and emotional data in more detail and rescores how good the user's driving skills are.

[0593] Step 9:

[0594] The server provides the generated score to the insurance company, and adjusts the user's insurance premium based on that score. In this case, if the user maintains a low-stress and safe driving style, the insurance premium may be discounted.

[0595] Step 10:

[0596] Users review the score and sentiment report displayed on their device and consider how to improve for their next drive. This information is useful in encouraging continued safe driving.

[0597] (Example 2)

[0598] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0599] Conventional in-vehicle systems often evaluate drivers' driving skills and emotional states individually, which presents challenges in improving overall safety and providing flexible responses to users. Furthermore, the lack of mechanisms that directly reflect driving skills and emotional states in insurance premiums is also a problem.

[0600] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0601] In this invention, the server includes means for recognizing surrounding information and traffic conditions using an in-vehicle camera; means for using a machine learning algorithm to provide instructions to the user based on the recognized information and traffic conditions; means for collecting driving behavior data and emotional data, and analyzing the data to evaluate the driver's driving skills; means for generating data to reflect the evaluation results in insurance premiums; means for transmitting the analysis results and evaluation data to a remote storage device; means for providing an emotion analysis engine to predict the user's emotional state and provide appropriate feedback while driving; and means for monitoring the driving state in real time using multiple sensors. This enables comprehensive feedback and insurance premium adjustments based on the driver's skills and emotional state.

[0602] A "vehicle-mounted camera" is a device installed in a vehicle to acquire visual information about the surroundings.

[0603] A "machine learning algorithm" is a program or model that learns from large amounts of data and performs a specific task.

[0604] "Driving behavior data" refers to information collected about a driver's driving style and behavior.

[0605] "Emotional data" refers to information about the driver's psychological state inferred from their facial expressions, voice, and other biosignals.

[0606] "Evaluation" is the process of determining a driver's skills and adaptability based on collected data.

[0607] A "sentiment analysis engine" is a computer program that analyzes collected emotional data to estimate the driver's psychological state.

[0608] A "remote storage device" is an external storage device or database used to store data over a network.

[0609] "Real-time" refers to a situation or method in which data processing or responses occur instantaneously.

[0610] This invention aims to promote safe driving and optimize insurance premiums by monitoring and evaluating the driver's behavior and emotions through a system installed in the vehicle. The system mainly consists of a "terminal" and a "server".

[0611] Terminal operation

[0612] The terminal is installed in the vehicle and uses multiple sensors to collect driver behavior and emotional data in real time. Specifically, it uses an in-vehicle camera to record the road environment and the driver's facial expressions, and a voice detection device to collect the driver's speech. This data is analyzed using machine learning algorithms that run within the terminal. The video data acquired from the camera is used to recognize traffic signs and the behavior of other vehicles, and to provide necessary instructions to the driver. In addition, an emotion analysis engine is used to estimate the driver's psychological state and provide feedback according to the level of stress and fatigue.

[0613] For example, if a driver is fatigued from driving for a long time, the device will display a message prompting them to take a break. This allows the driver to accurately understand their situation and take appropriate action.

[0614] Server operation

[0615] The server is responsible for receiving and storing data transmitted from terminals. The server organizes and stores driving behavior data and emotional data in a database, and uses this data to evaluate the driver's driving skills. The evaluation is based on a generative AI model, comprehensively combining past data history with real-time emotional states.

[0616] The results of the driving skills evaluation are calculated as a driving score and provided to the insurance company from the server. The insurance company uses this information to adjust insurance premiums and applies discounts to drivers with high safe driving scores.

[0617] User interaction

[0618] While driving, users can receive feedback from the device, allowing them to appropriately adjust their driving behavior and emotional state. After driving, they can review the driving score and emotional report displayed on the device to understand specific areas for improvement and the next steps to take.

[0619] For example, a prompt message to be input to the generating AI model would be, "Analyze the driver's driving behavior and emotional state in real time and generate appropriate feedback." This would enable the driver to enjoy a safe and comfortable driving experience.

[0620] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0621] Step 1:

[0622] The terminal collects driver behavioral and emotional data using an in-vehicle camera and voice detection device. The inputs include video and audio data. Specifically, the camera captures road conditions and traffic signs, while the voice detection device captures the driver's voice and the in-vehicle acoustic environment. This data is then sent to the next analysis step.

[0623] Step 2:

[0624] The device applies machine learning algorithms to analyze the collected video and audio data. In this step, it recognizes traffic conditions and estimates the driver's emotional state from the data received as input. Specifically, it recognizes traffic signs and lane information from the video data, and estimates the current emotional state from the audio data and facial expressions using an emotion analysis engine. The output of this process is then compiled into feedback information for driver assistance.

[0625] Step 3:

[0626] The terminal provides real-time feedback to the driver based on the analysis results. The input here is the feedback information obtained in step 2. Specifically, it displays a driving skill score on the screen and provides advice via a voice assistant regarding adjustments to driving posture and the need for breaks. This output is immediately reflected in the driver's experience.

[0627] Step 4:

[0628] The server receives driving behavior data and emotional data transmitted from terminals and stores this data in its database. The input consists of analyzed information about the driver's behavior and emotions. The server organizes and stores this data to use as the basis for the next evaluation process.

[0629] Step 5:

[0630] The server evaluates the driver's driving skills using data stored in the database. The inputs used are stored driving behavior data and emotional data. A generative AI model is used to evaluate each driver based on their past driving records and emotional state, generating a driving skill score. This score is output and used as data for adjusting insurance premiums.

[0631] Step 6:

[0632] After driving, users review their driving score and emotional report displayed on their device. Input here includes evaluation results and feedback information. Based on this data, users identify specific areas for improvement in their driving skills and emotional control. This leads to safer and more comfortable driving.

[0633] (Application Example 2)

[0634] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0635] Conventional vehicle driving monitoring systems only monitor the driver's driving skills and surrounding traffic conditions, making it difficult to provide feedback that takes into account the driver's emotional state and stress levels. As a result, potential hazards may be overlooked, and improvements in safe driving may not be sufficiently achieved. Furthermore, there have been insufficient means to provide relaxation advice based on the driver's emotional state or to optimize insurance premiums.

[0636] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0637] In this invention, the server includes means for using an emotion analysis device to acquire the driver's facial expressions and voice and analyze their emotional state; means for suggesting relaxation methods based on the detected emotional state; and means for analyzing the acquired video data and emotional data and providing real-time feedback to the driver. This enables driving assistance that takes the driver's emotional state into account, providing a safer and more comfortable driving experience.

[0638] An "in-vehicle video acquisition device" is a device mounted on a vehicle to capture images of surrounding signs and traffic conditions.

[0639] An "artificial intelligence processing unit" is a device that executes algorithms to provide instructions to the driver based on recognized signs and traffic conditions.

[0640] "Operational data" refers to data about various operations and conditions recorded while a vehicle is in operation.

[0641] An "emotion analysis device" is a device that analyzes a driver's facial expressions and voice to estimate their emotional state.

[0642] "Means of suggesting relaxation methods" refers to methods of providing stress reduction in accordance with the detected emotional state of the driver.

[0643] An "information processing device" is a device or system for storing or transmitting analysis results and evaluation data.

[0644] The system implementing this invention aims to support safe and comfortable driving by comprehensively analyzing the driver's driving skills and emotional state. The system mainly consists of an in-vehicle video acquisition device, an artificial intelligence processing device, an emotion analysis device, and an information processing device.

[0645] The terminal uses an in-vehicle video acquisition system to recognize the surrounding traffic environment and signs in real time. This video data is analyzed via an artificial intelligence processing unit, which provides appropriate instructions based on the driver's driving behavior and traffic conditions. Furthermore, an emotion analysis device acquires and analyzes the driver's facial expressions and voice to estimate their current emotional state. Based on the estimated emotional state, feedback suggesting relaxation methods is provided via the terminal's display or audio output.

[0646] The server stores analysis results and driving skill evaluation data in an information processing device, and statistically analyzes the driver's past driving data and emotional state. The analysis results are used to support safe and effective driving, especially when the driver is experiencing excessive stress or fatigue, by suggesting appropriate rest times.

[0647] For example, if the emotion analysis device detects driver fatigue during a long-distance drive, the system may display a message such as, "We recommend taking a break at the next service area." In this way, the driving experience is improved.

[0648] Examples of prompt messages are as follows:

[0649] "Analyze the video and audio captured by the in-car camera and use AI to determine the driver's emotional state. If signs of fatigue or stress are detected, design code that displays a message advising the driver to stop driving and suggests appropriate relaxation methods."

[0650] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0651] Step 1:

[0652] The terminal uses an in-vehicle video acquisition device to capture real-time video of surrounding signs and traffic conditions. This video data is used as input, and an artificial intelligence processing unit performs image analysis to identify the location of signs and vehicles, as well as traffic conditions. The output is the recognized sign information and traffic condition data.

[0653] Step 2:

[0654] The terminal uses an emotion analysis device to simultaneously acquire images of the driver's facial expressions and audio. Using this data as input, an emotion analysis algorithm estimates the driver's emotional state. The analysis results are emotional indicators such as stress, fatigue, and relaxation levels, which are then output.

[0655] Step 3:

[0656] The server receives signage information, traffic data, and sentiment indicators transmitted from the terminal. Using this data as input, it comprehensively evaluates the driver's driving behavior and calculates a driving skill score. The evaluation process also includes comparison with past driving data and sentiment history. The output is feedback based on the latest driving skill score and sentiment state.

[0657] Step 4:

[0658] The terminal receives evaluation results from the server and provides feedback to the driver. Specifically, it displays messages on the screen suggesting ways to relax or take a break, and provides voice notifications if necessary. At this time, it again takes in emotional indicators as input and customizes the feedback content. The output is a reminder to the driver and encouragement of appropriate driving behavior.

[0659] Step 5:

[0660] Based on feedback from the device, users take actions such as improving their driving behavior or taking breaks. This feedback can be used to drive more safely and comfortably. Specifically, this includes reviewing driving posture and taking breaks at indicated times. Based on user responses, more data is accumulated and used for future analysis.

[0661] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0662] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0663] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0664] [Fourth Embodiment]

[0665] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0666] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0667] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0668] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0669] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0670] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0671] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0672] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0673] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0674] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0675] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0676] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0677] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0678] The present invention is a system installed in a vehicle driven by a driver, and consists of an in-vehicle camera, a processing unit using an artificial intelligence algorithm, a group of sensors for monitoring driving behavior and acquiring data, a communication module, and a display or audio output device for providing information.

[0679] Terminal operation

[0680] The device is installed in the vehicle and first captures real-time video of the road environment using an on-board camera. The captured video is sent to an internal processing unit and analyzed by an artificial intelligence algorithm. This analysis recognizes traffic signs, traffic lights, pedestrians, lanes, and vehicles ahead, enabling the system to provide the driver with necessary information and instructions. For example, if the speed limit is exceeded, a visual warning is displayed on the screen and an audio notification is given.

[0681] Furthermore, the terminal continuously collects various data such as vehicle speed, acceleration, and braking. This data is used to evaluate driving behavior, and items related to safe driving, particularly sudden acceleration and sudden braking, are scored.

[0682] Server operation

[0683] Data transmitted from the terminal is received and stored on the server. The server assigns a unique ID to each driver and manages the data based on this ID. The collected driving data is reanalyzed by a program that runs an artificial intelligence algorithm to accurately score driving skills. The score is calculated based on the driver's past driving data and the frequency of dangerous driving behaviors.

[0684] The server uses the analysis results to provide companies with data for adjusting insurance premiums. This adjustment is made by setting lower premiums for drivers with high scores (i.e., those who drive safely).

[0685] User interaction

[0686] Based on the visual and auditory feedback provided by the system, users can continuously review their driving style. After finishing their drive, they can check the score displayed on the terminal and see how their insurance premiums will change the following month. This motivates users to drive safely.

[0687] A concrete example would be a scenario where, while a driver is driving on a highway, the device recognizes the distance to the vehicle in front and issues a warning to maintain a safe distance. Through such prompts, users can naturally improve their driving habits.

[0688] The following describes the processing flow.

[0689] Step 1:

[0690] The device captures video data from the in-vehicle camera in real time. This data includes road signs, traffic lights, vehicles, pedestrians, and other information.

[0691] Step 2:

[0692] The device analyzes the captured video using an internal artificial intelligence algorithm. The algorithm uses image recognition technology to identify signs and traffic conditions, and provides the user with appropriate driving instructions based on this information.

[0693] Step 3:

[0694] The terminal simultaneously collects various sensor data related to driving behavior, such as vehicle speed, acceleration, and brake usage.

[0695] Step 4:

[0696] The device analyzes collected driving behavior data and calculates a safe driving score. The data analysis includes the frequency of sudden acceleration and braking, as well as whether or not the vehicle is speeding.

[0697] Step 5:

[0698] Based on the analysis results obtained, the device provides visual and auditory feedback to the user to encourage safe driving.

[0699] Step 6:

[0700] The device sends analysis data and scores to the server at regular intervals. The data is linked to a unique user ID.

[0701] Step 7:

[0702] The server receives data sent from the terminal and stores it in the database. Each user's driving history is managed.

[0703] Step 8:

[0704] The server performs additional analysis based on the received data and generates a score that re-evaluates driving behavior. This includes an analysis of driving trends over time.

[0705] Step 9:

[0706] The server provides the generated scores to the insurance company's system, which is then used to adjust the driver's insurance premiums.

[0707] Step 10:

[0708] Users review the feedback and scores displayed on their devices and consider areas for improvement for their next drive.

[0709] (Example 1)

[0710] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0711] In recent years, reducing traffic accidents and ensuring fair insurance premiums have become critical issues. Conventional systems make it difficult to objectively evaluate drivers' driving skills, sometimes resulting in inappropriate premium settings. Furthermore, mechanisms to promote safe driving by providing immediate feedback during driving are insufficient.

[0712] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0713] In this invention, the server includes means for recognizing surrounding traffic information using an in-vehicle image acquisition device, means for using a computational method to provide instructions to the driver based on the recognized traffic information, and means for collecting vehicle movement characteristic data and analyzing the data to evaluate the driver's driving skills. This makes it possible to promote safe driving and set appropriate insurance premiums based on individual driving behavior.

[0714] An "in-vehicle image acquisition device" is a device mounted on a vehicle to capture surrounding traffic information.

[0715] "Traffic information" refers to information that includes data on road conditions, traffic signs, traffic lights, pedestrians, and other vehicles.

[0716] "Computational methods" refer to algorithms and mathematical models used to analyze recognized traffic information and provide instructions to drivers.

[0717] "Movement characteristics data" refers to data related to the movement of a vehicle, such as vehicle speed, acceleration, and braking operation.

[0718] "Driving skills" refer to the ability of a driver to operate a vehicle safely and efficiently.

[0719] "Evaluation" is the process of analyzing a driver's driving behavior numerically or qualitatively based on collected data and making a judgment.

[0720] "Risk management information" refers to the data and indicators used in setting insurance premiums and assessing risks.

[0721] A "data storage device" is a system for recording and storing various types of data.

[0722] This invention is a system for promoting safe driving and enabling adjustment of insurance premiums based on individual driving behavior. The terminal is installed in the vehicle and first uses an in-vehicle image acquisition device to capture traffic information of the road and surrounding environment in real time. This image acquisition device uses high-resolution sensors that can detect traffic signs, signals, pedestrians, and other vehicles.

[0723] The processing unit within the terminal analyzes the acquired information using computational methods executed with frameworks such as TensorFlow and PyTorch. During this process, the system provides the driver with necessary instructions and information in real time. For example, in the event of a sudden lane change, the terminal issues voice and visual warnings.

[0724] Furthermore, the terminal collects data on the vehicle's movement characteristics. This includes data on the vehicle's acceleration, speed, and braking performance, using speed sensors and inertial measurement devices. Based on this data, the driver's driving skills are evaluated, and a safe driving index is calculated according to certain criteria.

[0725] The server receives data transmitted from the terminal and stores it in a database. Simultaneously, the server re-analyzes the stored data and generates driver risk management information. This information is used to adjust insurance premiums.

[0726] Users can review their driving style by receiving immediate feedback from the device. Furthermore, after driving, they can check their evaluated score and any changes in insurance premiums. For example, imagine a scenario where a user is driving on a highway and the device recognizes the distance to the vehicle in front and issues a warning to maintain a safe distance.

[0727] An example of a prompt using a generative AI model is, "Explain what visual and auditory feedback should be provided to the driver to maintain a safe distance from the vehicle ahead while driving on a highway." This allows the AI ​​to be instructed to generate specific feedback. In this way, specific methods are provided to promote safe and appropriate driving for the driver.

[0728] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0729] Step 1:

[0730] The terminal uses an in-vehicle image acquisition device to capture real-time traffic information of roads and the surrounding environment. Inputs include road video data, traffic signs, signals, and pedestrians. This data is initially processed and converted to an appropriate format for subsequent processing. Specifically, a high-resolution camera sensor captures images at a rate of several tens of frames per second.

[0731] Step 2:

[0732] The terminal analyzes the acquired video data using its internal processing unit. Captured image data is used as input, and based on this, the traffic environment is recognized using computational methods. Specifically, an artificial intelligence algorithm is used, and an object recognition model is employed to identify traffic signs, traffic lights, and other objects. As output, information on the identified objects is generated and passed on to the next step.

[0733] Step 3:

[0734] The terminal collects vehicle movement characteristic data and monitors the driver's behavior. Speed ​​sensor and brake operation information are used as input. Data calculations are performed based on the collected data to evaluate the driver's driving skills. In this process, a safe driving index is calculated. Specifically, sensors continuously record the vehicle's acceleration and speed, and the results are analyzed in real time.

[0735] Step 4:

[0736] The terminal provides real-time feedback to the driver based on the analysis results. The inputs used are analyzed object information and movement characteristic data. Based on this, the terminal issues voice and visual warnings and sends appropriate action instructions to the driver. Visual and auditory feedback is generated as output. Specifically, when the safe distance is exceeded, a warning sound or a warning message is emitted on the display.

[0737] Step 5:

[0738] The server receives data transmitted from the terminal and stores it in the database. Driving behavior data and analysis results are sent to the server as input. This data is organized and used for subsequent analysis. Specifically, data is uploaded to the server via a communication module and recorded in the database system.

[0739] Step 6:

[0740] The server uses accumulated data and a generating AI model to re-evaluate the driver's safe driving index. Past and current driving data are taken as input, and a detailed score analysis is performed using the AI ​​model. As output, driver risk management information is generated, contributing to insurance premium adjustments. Specifically, the driver's driving tendencies and risk profile are calculated.

[0741] Step 7:

[0742] After completing a drive, the user reviews the evaluation score and feedback displayed on the device. The input used is the driving data and its analysis. Based on this, the user gains information to improve their driving style. For example, specific advice such as "You tend to brake abruptly" might be displayed on the device.

[0743] (Application Example 1)

[0744] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0745] Drivers are required to pay close attention to their surroundings and traffic conditions when operating a vehicle, but a lack of such attention can lead to accidents. Furthermore, there is a lack of adequate systems for evaluating driving safety and reflecting the results in individual insurance premiums. To solve these problems and promote safe driving, a means of providing drivers with prompt and specific feedback is necessary.

[0746] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0747] In this invention, the server includes means for recognizing surrounding signs and traffic conditions using an in-vehicle camera, means for using an artificial intelligence algorithm to provide instructions to the driver based on the recognized signs and traffic conditions, and means for performing eye tracking and highlighting visual information. This allows the driver to be alerted to important matters in real time, enabling safe driving and appropriate insurance premium setting.

[0748] An "in-vehicle camera" is a camera device installed in a vehicle that captures the surrounding road environment and traffic conditions, and provides video data.

[0749] "Sign and traffic condition recognition" refers to the process of analyzing video data acquired from in-vehicle cameras to identify road signs, traffic lights, vehicles, pedestrians, and other similar elements.

[0750] "Means of using artificial intelligence algorithms" refers to computational processes realized through machine learning and deep learning technologies that are used to make judgments about traffic rules and provide instructions to drivers based on recognized data.

[0751] "Driving behavior data" refers to a set of data related to the driver's actions, such as vehicle speed, acceleration, and braking.

[0752] "Evaluating driving skills" involves analyzing acquired driving behavior data and quantifying and evaluating the degree of safe driving and driving skills.

[0753] "Eye-tracking" is a technology that tracks where a driver is looking and analyzes that data.

[0754] "Highlighting" is a method of making visual information stand out in order to draw the driver's particular attention.

[0755] "A means of issuing warnings using an audio output device" refers to a system that notifies the driver by voice when dangerous driving or situations requiring attention are detected.

[0756] "Information reflected in insurance premiums" refers to data used to adjust individual insurance premiums based on safe driving evaluation results.

[0757] "Means of sending to a storage device" refers to the process of sending analyzed data and evaluation results to a server or cloud storage for storage.

[0758] The system for implementing this invention consists of an in-vehicle camera, an artificial intelligence algorithm, an eye-tracking module, an audio and visual output device, a communication module, and a storage device. The integration of these elements makes it possible to monitor the driver's driving behavior in real time, evaluate driving skills, and promote safe driving.

[0759] The server captures images of the surrounding road environment through an in-vehicle camera and uses that data to recognize traffic signs, signals, pedestrians, and other vehicles. Based on this, it generates necessary instructions for the driver, which are then provided via display and voice. In particular, the use of libraries such as TensorFlow and OpenCV enables highly accurate image analysis.

[0760] The eye-tracking module tracks the driver's field of vision and highlights important information that is often overlooked. This allows for precise guidance of the driver's attention. For example, if the driver's gaze is not focused on a traffic sign ahead, the sign will be highlighted, and an audio alert will be issued.

[0761] Driving behavior data, including vehicle speed, acceleration, and braking information, is evaluated in real time by an artificial intelligence algorithm. A safe driving index is calculated, and these evaluation results are transmitted to storage for adjusting insurance premiums based on the degree of safe driving. This data is stored on a cloud server via a communication module and can be accessed as needed.

[0762] A concrete example would be a situation where, while driving on a highway, the driver misses a speed limit sign ahead. In such a scenario, the system visually highlights the sign and simultaneously provides a voice notification stating, "The speed limit is 80 km / h."

[0763] Examples of prompt statements include the following:

[0764] "When driving on a highway, if your eyes are not focused on road signs or traffic lights, an alert will be displayed. You will also be notified in advance when the traffic light is about to change from green to red."

[0765] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0766] Step 1:

[0767] The terminal uses an in-vehicle camera to capture the surrounding road environment in real time. The input is the video data acquired from the camera, and the output is a stream of this data. The video data is then sent to the next analysis step.

[0768] Step 2:

[0769] The server receives the video data acquired in Step 1 and uses an artificial intelligence algorithm to recognize traffic signs, traffic lights, pedestrians, lanes, vehicles ahead, etc. The input is video data, and the output is a list of the recognized information. TensorFlow and OpenCV libraries are used for this analysis.

[0770] Step 3:

[0771] The device uses an eye-tracking module to acquire the driver's gaze data. The input is the location of the gaze, and the output is the identification of the object the gaze is directed towards. This allows the device to determine the direction in which the driver's attention is focused.

[0772] Step 4:

[0773] The terminal integrates the data from steps 2 and 3 and highlights important signs and signals on the display if they are outside the user's line of sight. The input is recognition information and gaze data, and the output is the highlighting instruction.

[0774] Step 5:

[0775] The terminal will issue a warning via an audio output device if dangerous driving is detected. The input is the result of the driving behavior analysis, and the output is an audio warning. The user will receive an audio alert, such as "The traffic light ahead has turned red."

[0776] Step 6:

[0777] The server receives driving behavior data (speed, acceleration, braking, etc.) and calculates the driver's safe driving index. The input is driving behavior data, and the output is the safe driving index. A machine learning model is used for this evaluation.

[0778] Step 7:

[0779] The server sends the safe driving index calculated in step 6 to a storage device for use in adjusting insurance premiums. The input is the safe driving index, and the output is the updated insurance premium information. This information is stored on a cloud server and can be accessed at a later date.

[0780] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0781] The present invention is a vehicle-mounted system that simultaneously monitors the driver's driving behavior and emotional state, and has a configuration that combines an in-vehicle camera, artificial intelligence algorithm, emotion engine, various sensor groups, communication module, information provision device, and the like.

[0782] Terminal operation

[0783] The device uses an in-car camera to capture real-time video of the road environment and uses an artificial intelligence algorithm to recognize traffic signs, lanes, and vehicles ahead. Simultaneously, another camera and microphone capture the driver's facial expressions and speech, and an emotion engine estimates the user's emotional state. This information is used to evaluate the user's stress level, concentration level, and relaxation level while driving.

[0784] The collected driving and emotional data are analyzed within the device and provided to the user as timely feedback via voice and display. For example, if the device determines that the user is stressed, it will notify the user to reconsider their driving posture. Additionally, a driving skill score is visually displayed, along with the analysis results and emotional state.

[0785] Server operation

[0786] Data processed on the terminal is sent to the server. The server receives this data and stores the driving and emotional data in a database. Driving skills are scored considering the user's past data history and current emotional state, and the results are optimized for insurance premium adjustments.

[0787] The server feeds back the generated scores and emotional data to insurance companies, allowing them to adjust premiums based on this information. Users with high safe driving scores and who maintain a relaxed emotional state are offered benefits such as discounts on their insurance premiums.

[0788] User interaction

[0789] Users can receive real-time feedback while driving, which can help them manage their driving style and emotions. After finishing their drive, they can review the driving score and emotion report displayed on the device to identify specific areas for improvement.

[0790] For example, if the emotion engine detects that the user is becoming fatigued during long-distance driving, the device will display a message prompting them to take a break, supporting safe and comfortable driving. In this way, users can utilize emotional data in conjunction with their driving behavior to achieve safer and more effective driving.

[0791] The following describes the processing flow.

[0792] Step 1:

[0793] The device captures video data from the in-vehicle camera in real time. This data includes traffic signs, traffic lights, lanes, and traffic conditions ahead.

[0794] Step 2:

[0795] The device uses a separate camera and microphone to capture the user's facial expressions and voice, and inputs this data into the emotion engine. The emotion engine analyzes this data to identify the user's emotional state.

[0796] Step 3:

[0797] The terminal analyzes collected traffic data using an artificial intelligence algorithm and provides driving instructions to the user. For example, if it recognizes a speed limit sign, it notifies the user of that information via the display or voice.

[0798] Step 4:

[0799] The device continuously records the user's driving behavior data using speed and acceleration sensors. This includes the frequency of sudden acceleration, sudden braking, and steering maneuvers.

[0800] Step 5:

[0801] The device combines driving behavior and emotional state data to score the user's driving skills. Emotions such as tension and fatigue are taken into consideration, as they may affect the score.

[0802] Step 6:

[0803] The device provides real-time feedback to the user along with the scoring results. If the emotion engine determines that the user is fatigued, the device will verbally remind them to take a break.

[0804] Step 7:

[0805] The terminal periodically sends analyzed driving and emotional data to the server. The transmitted data is assigned a unique user ID and managed on the server.

[0806] Step 8:

[0807] Based on the data received from the terminal, the server analyzes driving behavior and emotional data in more detail and rescores how good the user's driving skills are.

[0808] Step 9:

[0809] The server provides the generated score to the insurance company, and adjusts the user's insurance premium based on that score. In this case, if the user maintains a low-stress and safe driving style, the insurance premium may be discounted.

[0810] Step 10:

[0811] Users review the score and sentiment report displayed on their device and consider how to improve for their next drive. This information is useful in encouraging continued safe driving.

[0812] (Example 2)

[0813] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0814] Conventional in-vehicle systems often evaluate drivers' driving skills and emotional states individually, which presents challenges in improving overall safety and providing flexible responses to users. Furthermore, the lack of mechanisms that directly reflect driving skills and emotional states in insurance premiums is also a problem.

[0815] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0816] In this invention, the server includes means for recognizing surrounding information and traffic conditions using an in-vehicle camera; means for using a machine learning algorithm to provide instructions to the user based on the recognized information and traffic conditions; means for collecting driving behavior data and emotional data, and analyzing the data to evaluate the driver's driving skills; means for generating data to reflect the evaluation results in insurance premiums; means for transmitting the analysis results and evaluation data to a remote storage device; means for providing an emotion analysis engine to predict the user's emotional state and provide appropriate feedback while driving; and means for monitoring the driving state in real time using multiple sensors. This enables comprehensive feedback and insurance premium adjustments based on the driver's skills and emotional state.

[0817] A "vehicle-mounted camera" is a device installed in a vehicle to acquire visual information about the surroundings.

[0818] A "machine learning algorithm" is a program or model that learns from large amounts of data and performs a specific task.

[0819] "Driving behavior data" refers to information collected about a driver's driving style and behavior.

[0820] "Emotional data" refers to information about the driver's psychological state inferred from their facial expressions, voice, and other biosignals.

[0821] "Evaluation" is the process of determining a driver's skills and adaptability based on collected data.

[0822] A "sentiment analysis engine" is a computer program that analyzes collected emotional data to estimate the driver's psychological state.

[0823] A "remote storage device" is an external storage device or database used to store data over a network.

[0824] "Real-time" refers to a situation or method in which data processing or responses occur instantaneously.

[0825] This invention aims to promote safe driving and optimize insurance premiums by monitoring and evaluating the driver's behavior and emotions through a system installed in the vehicle. The system mainly consists of a "terminal" and a "server".

[0826] Terminal operation

[0827] The terminal is installed in the vehicle and uses multiple sensors to collect driver behavior and emotional data in real time. Specifically, it uses an in-vehicle camera to record the road environment and the driver's facial expressions, and a voice detection device to collect the driver's speech. This data is analyzed using machine learning algorithms that run within the terminal. The video data acquired from the camera is used to recognize traffic signs and the behavior of other vehicles, and to provide necessary instructions to the driver. In addition, an emotion analysis engine is used to estimate the driver's psychological state and provide feedback according to the level of stress and fatigue.

[0828] For example, if a driver is fatigued from driving for a long time, the device will display a message prompting them to take a break. This allows the driver to accurately understand their situation and take appropriate action.

[0829] Server operation

[0830] The server is responsible for receiving and storing data transmitted from terminals. The server organizes and stores driving behavior data and emotional data in a database, and uses this data to evaluate the driver's driving skills. The evaluation is based on a generative AI model, comprehensively combining past data history with real-time emotional states.

[0831] The results of the driving skills evaluation are calculated as a driving score and provided to the insurance company from the server. The insurance company uses this information to adjust insurance premiums and applies discounts to drivers with high safe driving scores.

[0832] User interaction

[0833] While driving, users can receive feedback from the device, allowing them to appropriately adjust their driving behavior and emotional state. After driving, they can review the driving score and emotional report displayed on the device to understand specific areas for improvement and the next steps to take.

[0834] For example, a prompt message to be input to the generating AI model would be, "Analyze the driver's driving behavior and emotional state in real time and generate appropriate feedback." This would enable the driver to enjoy a safe and comfortable driving experience.

[0835] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0836] Step 1:

[0837] The terminal collects driver behavioral and emotional data using an in-vehicle camera and voice detection device. The inputs include video and audio data. Specifically, the camera captures road conditions and traffic signs, while the voice detection device captures the driver's voice and the in-vehicle acoustic environment. This data is then sent to the next analysis step.

[0838] Step 2:

[0839] The device applies machine learning algorithms to analyze the collected video and audio data. In this step, it recognizes traffic conditions and estimates the driver's emotional state from the data received as input. Specifically, it recognizes traffic signs and lane information from the video data, and estimates the current emotional state from the audio data and facial expressions using an emotion analysis engine. The output of this process is then compiled into feedback information for driver assistance.

[0840] Step 3:

[0841] The terminal provides real-time feedback to the driver based on the analysis results. The input here is the feedback information obtained in step 2. Specifically, it displays a driving skill score on the screen and provides advice via a voice assistant regarding adjustments to driving posture and the need for breaks. This output is immediately reflected in the driver's experience.

[0842] Step 4:

[0843] The server receives driving behavior data and emotional data transmitted from terminals and stores this data in its database. The input consists of analyzed information about the driver's behavior and emotions. The server organizes and stores this data to use as the basis for the next evaluation process.

[0844] Step 5:

[0845] The server evaluates the driver's driving skills using data stored in the database. The inputs used are stored driving behavior data and emotional data. A generative AI model is used to evaluate each driver based on their past driving records and emotional state, generating a driving skill score. This score is output and used as data for adjusting insurance premiums.

[0846] Step 6:

[0847] After driving, users review their driving score and emotional report displayed on their device. Input here includes evaluation results and feedback information. Based on this data, users identify specific areas for improvement in their driving skills and emotional control. This leads to safer and more comfortable driving.

[0848] (Application Example 2)

[0849] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0850] Conventional vehicle driving monitoring systems only monitor the driver's driving skills and surrounding traffic conditions, making it difficult to provide feedback that takes into account the driver's emotional state and stress levels. As a result, potential hazards may be overlooked, and improvements in safe driving may not be sufficiently achieved. Furthermore, there have been insufficient means to provide relaxation advice based on the driver's emotional state or to optimize insurance premiums.

[0851] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0852] In this invention, the server includes means for using an emotion analysis device to acquire the driver's facial expressions and voice and analyze their emotional state; means for suggesting relaxation methods based on the detected emotional state; and means for analyzing the acquired video data and emotional data and providing real-time feedback to the driver. This enables driving assistance that takes the driver's emotional state into account, providing a safer and more comfortable driving experience.

[0853] An "in-vehicle video acquisition device" is a device mounted on a vehicle to capture images of surrounding signs and traffic conditions.

[0854] An "artificial intelligence processing unit" is a device that executes algorithms to provide instructions to the driver based on recognized signs and traffic conditions.

[0855] "Operational data" refers to data about various operations and conditions recorded while a vehicle is in operation.

[0856] An "emotion analysis device" is a device that analyzes a driver's facial expressions and voice to estimate their emotional state.

[0857] "Means of suggesting relaxation methods" refers to methods of providing stress reduction in accordance with the detected emotional state of the driver.

[0858] An "information processing device" is a device or system for storing or transmitting analysis results and evaluation data.

[0859] The system implementing this invention aims to support safe and comfortable driving by comprehensively analyzing the driver's driving skills and emotional state. The system mainly consists of an in-vehicle video acquisition device, an artificial intelligence processing device, an emotion analysis device, and an information processing device.

[0860] The terminal uses an in-vehicle video acquisition system to recognize the surrounding traffic environment and signs in real time. This video data is analyzed via an artificial intelligence processing unit, which provides appropriate instructions based on the driver's driving behavior and traffic conditions. Furthermore, an emotion analysis device acquires and analyzes the driver's facial expressions and voice to estimate their current emotional state. Based on the estimated emotional state, feedback suggesting relaxation methods is provided via the terminal's display or audio output.

[0861] The server stores analysis results and driving skill evaluation data in an information processing device, and statistically analyzes the driver's past driving data and emotional state. The analysis results are used to support safe and effective driving, especially when the driver is experiencing excessive stress or fatigue, by suggesting appropriate rest times.

[0862] For example, if the emotion analysis device detects driver fatigue during a long-distance drive, the system may display a message such as, "We recommend taking a break at the next service area." In this way, the driving experience is improved.

[0863] Examples of prompt messages are as follows:

[0864] "Analyze the video and audio captured by the in-car camera and use AI to determine the driver's emotional state. If signs of fatigue or stress are detected, design code that displays a message advising the driver to stop driving and suggests appropriate relaxation methods."

[0865] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0866] Step 1:

[0867] The terminal uses an in-vehicle video acquisition device to capture real-time video of surrounding signs and traffic conditions. This video data is used as input, and an artificial intelligence processing unit performs image analysis to identify the location of signs and vehicles, as well as traffic conditions. The output is the recognized sign information and traffic condition data.

[0868] Step 2:

[0869] The terminal uses an emotion analysis device to simultaneously acquire images of the driver's facial expressions and audio. Using this data as input, an emotion analysis algorithm estimates the driver's emotional state. The analysis results are emotional indicators such as stress, fatigue, and relaxation levels, which are then output.

[0870] Step 3:

[0871] The server receives signage information, traffic data, and sentiment indicators transmitted from the terminal. Using this data as input, it comprehensively evaluates the driver's driving behavior and calculates a driving skill score. The evaluation process also includes comparison with past driving data and sentiment history. The output is feedback based on the latest driving skill score and sentiment state.

[0872] Step 4:

[0873] The terminal receives evaluation results from the server and provides feedback to the driver. Specifically, it displays messages on the screen suggesting ways to relax or take a break, and provides voice notifications if necessary. At this time, it again takes in emotional indicators as input and customizes the feedback content. The output is a reminder to the driver and encouragement of appropriate driving behavior.

[0874] Step 5:

[0875] Based on feedback from the device, users take actions such as improving their driving behavior or taking breaks. This feedback can be used to drive more safely and comfortably. Specifically, this includes reviewing driving posture and taking breaks at indicated times. Based on user responses, more data is accumulated and used for future analysis.

[0876] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0877] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0878] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0879] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0880] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0881] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0882] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0883] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0884] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0885] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0886] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0887] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0888] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0889] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0890] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0891] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0892] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0893] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0894] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0895] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0896] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0897] The following is further disclosed regarding the embodiments described above.

[0898] (Claim 1)

[0899] A means of recognizing surrounding signs and traffic conditions using an in-vehicle camera,

[0900] A means of using an artificial intelligence algorithm to provide instructions to the driver based on recognized signs and traffic conditions,

[0901] A means of collecting vehicle driving behavior data, analyzing that data, and scoring the driver's driving skills,

[0902] A means for generating data to reflect the scoring results in insurance premiums,

[0903] Means for sending analysis results and scoring data to a server,

[0904] A system that includes this.

[0905] (Claim 2)

[0906] The system according to claim 1, which analyzes video data collected by an in-vehicle camera and provides real-time feedback to the driver.

[0907] (Claim 3)

[0908] The system according to claim 1, which calculates a safe driving score based on driving behavior data and adjusts insurance premiums according to the score.

[0909] "Example 1"

[0910] (Claim 1)

[0911] A means for recognizing surrounding traffic information using an in-vehicle image acquisition device,

[0912] A means of using a computational method to provide instructions to the driver based on recognized traffic information,

[0913] A means for collecting vehicle movement characteristic data, analyzing that data, and evaluating the driver's driving skills,

[0914] A means for generating data to reflect the evaluation results in risk management information,

[0915] Means for transmitting analysis results and evaluation data to a storage device,

[0916] A system that includes this.

[0917] (Claim 2)

[0918] The system according to claim 1, which analyzes video data collected by an in-vehicle image acquisition device and provides immediate feedback to the driver.

[0919] (Claim 3)

[0920] The system according to claim 1, which calculates a safe handling index based on movement characteristics data and adjusts risk management information according to the index.

[0921] "Application Example 1"

[0922] (Claim 1)

[0923] A means of recognizing surrounding signs and traffic conditions using an in-vehicle camera,

[0924] A means of using an artificial intelligence algorithm to provide instructions to the driver based on recognized signs and traffic conditions,

[0925] A means for collecting vehicle driving behavior data, analyzing that data, and evaluating the driver's driving skills,

[0926] A means for generating information to reflect the evaluation results in insurance premiums,

[0927] Means for transmitting analysis results and evaluation data to a storage device,

[0928] A means of performing eye tracking and highlighting visual information,

[0929] A means of issuing a warning using an audio output device in the event of dangerous driving,

[0930] A system that includes this.

[0931] (Claim 2)

[0932] The system according to claim 1, which analyzes video data collected by an in-vehicle camera and provides real-time feedback to the driver.

[0933] (Claim 3)

[0934] The system according to claim 1, which calculates a safe driving index based on driving behavior data and adjusts insurance premiums according to the index.

[0935] "Example 2 of combining an emotion engine"

[0936] (Claim 1)

[0937] A means of recognizing surrounding information and traffic conditions using an in-vehicle camera,

[0938] A means of using machine learning algorithms to provide instructions to users based on recognized information and traffic conditions,

[0939] A means for collecting driving behavior data and emotional data, and analyzing that data to evaluate the driver's driving skills,

[0940] A means of generating data to reflect the evaluation results in insurance premiums,

[0941] Means for transmitting analysis results and evaluation data to a remote storage device,

[0942] A means equipped with an emotion analysis engine to predict the user's emotional state and provide appropriate feedback during driving,

[0943] A means of monitoring the operating status in real time using multiple sensors,

[0944] A system that includes this.

[0945] (Claim 2)

[0946] The system according to claim 1, which analyzes data collected by an in-vehicle camera and a voice detection device and provides real-time feedback to the user.

[0947] (Claim 3)

[0948] The system according to claim 1, which calculates a driving skill evaluation based on driving behavior data and emotional state, and adjusts insurance premiums according to the evaluation.

[0949] "Application example 2 when combining with an emotional engine"

[0950] (Claim 1)

[0951] A means for recognizing surrounding signs and traffic environment using an in-vehicle video acquisition device,

[0952] Means for using an artificial intelligence processing device to provide instructions to the driver based on recognized signs and traffic environment,

[0953] A means of collecting operational data, analyzing that data, and evaluating the driver's driving skills,

[0954] A means for generating information to reflect the evaluation results in the fee calculation,

[0955] Means for transmitting analysis results and evaluation data to an information processing device,

[0956] A means of using an emotion analysis device to acquire the driver's facial expressions and voice and analyze their emotional state,

[0957] A means of suggesting relaxation methods based on the detected emotional state,

[0958] A system that includes this.

[0959] (Claim 2)

[0960] The system according to claim 1, which analyzes acquired video data and emotion data and provides real-time feedback to the driver.

[0961] (Claim 3)

[0962] The system according to claim 1, which calculates a safe driving evaluation based on operational data and emotional state data, and adjusts the fare according to the evaluation. [Explanation of Symbols]

[0963] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of recognizing surrounding signs and traffic conditions using an in-vehicle camera, A means of using an artificial intelligence algorithm to provide instructions to the driver based on recognized signs and traffic conditions, A means of collecting vehicle driving behavior data, analyzing that data, and scoring the driver's driving skills, A means for generating data to reflect the scoring results in insurance premiums, Means for sending analysis results and scoring data to a server, A system that includes this.

2. The system according to claim 1, which analyzes video data collected by an in-vehicle camera and provides real-time feedback to the driver.

3. The system according to claim 1, which calculates a safe driving score based on driving behavior data and adjusts insurance premiums according to the score.

Images