system

Abstract

We provide the system. [Solution] A means of recognizing road signs by analyzing images acquired in real time from an imaging device mounted on a vehicle, A means for generating and providing voice notifications to the operator based on recognized signs, A means for accumulating individual flight data and analyzing flight tendencies based on that data, A means for generating improvement guidelines for the pilot based on the analyzed piloting tendencies, A means of optimizing insurance premiums based on piloting skills by linking piloting data with an external device, A means of providing visual notification via a display device installed inside the vehicle, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] With the increase in the number of elderly drivers and novice drivers, the risk of traffic accidents is increasing. In contrast, in conventional driving support systems, the recognition of road signs and the analysis of individual driving tendencies are insufficient, and the support for improving the driver's skills and safe driving is limited. In addition, since the mechanism for reflecting driving behavior in insurance premiums is also insufficient, more precise and personalized driving support is required.

Means for Solving the Problems

[0005] This invention provides a system that uses a camera installed in a vehicle to recognize road signs in real time and provides feedback to the driver via voice. This system analyzes accumulated driving data and generates improvement suggestions based on driving trends, thereby improving driving skills. Furthermore, by linking driving data with an external system, it enables the optimization of insurance premiums according to the driver's skill level. This allows for more individually optimized support and promotes safer driving.

[0006] A "camera" is a device installed in a vehicle to acquire images.

[0007] "Road signs" are signs using symbols and figures that are used on roads to indicate traffic rules and information.

[0008] "Voice feedback" is a function that communicates recognized information and analysis results to the user via voice.

[0009] "Driving data" refers to data collected while driving, such as speed, location information, and behavioral history.

[0010] "Driving tendencies" refer to specific driving patterns and habits of a driver, extracted from accumulated driving data.

[0011] "Improvement suggestions" are advice and suggestions for improving driving skills, generated based on driving tendencies.

[0012] An "external system" is a system connected to the vehicle, separate from the in-vehicle systems, in order to utilize driving data.

[0013] "Insurance premium optimization" is the process of adjusting insurance premiums to an appropriate amount based on the driver's skills and behavior. [Brief explanation of the drawing]

[0014] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Mode for Carrying Out the Invention

[0015] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, the labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, the labeled RAM (Random Access Memory) is a memory where information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, the labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0020] In the following embodiments, the labeled communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] This invention relates to a system that provides driving assistance using road images acquired by a camera installed in a vehicle. First, a terminal continuously acquires images using the in-vehicle camera and uses AI to recognize road signs in real time from those images. Based on the recognized signs, the terminal generates voice feedback and provides it directly to the user. This voice feedback serves as a notification to encourage attention while driving.

[0036] Furthermore, the device transmits data acquired during driving, namely speed, location information, and traffic sign recognition results, to the server. The server stores this data and analyzes it using AI to understand the user's driving tendencies. Based on the analysis results, the server generates personalized driving improvement suggestions and provides feedback to the user via smartphone.

[0037] For example, when driving on a highway, the device recognizes speed limit signs and provides a voice warning, "The speed limit is 80 km / h." Furthermore, if the server detects frequent speeding based on past data, it provides specific advice to the user via the smartphone app, such as, "Adhering to the speed limit will improve fuel efficiency."

[0038] Furthermore, this driving data is linked with insurance companies, and the system also includes a mechanism to adjust users' insurance premiums based on their driving skills. As a result, users are motivated to drive safely and can also enjoy the benefit of lower insurance premiums.

[0039] As described above, the system of the present invention reduces the risk of accidents and promotes safe driving by providing individually optimized driving support to all drivers, including the elderly and novice drivers.

[0040] The following describes the processing flow.

[0041] Step 1:

[0042] The terminal uses a camera installed in the vehicle to acquire road images in real time. The camera continuously captures the surrounding environment while driving and generates video data.

[0043] Step 2:

[0044] The AI ​​installed in the device analyzes the acquired video data and recognizes road signs within the video. The AI ​​uses an image recognition algorithm to identify speed limit signs, stop signs, and other similar signs.

[0045] Step 3:

[0046] The device generates voice feedback based on recognized signs. Using speech synthesis technology, it creates messages such as "The speed limit is XX km / h" or "Please stop at the next intersection."

[0047] Step 4:

[0048] The device plays the generated audio feedback through the car's speakers, notifying the user. This allows the user to receive signage information in real time.

[0049] Step 5:

[0050] The terminal collects driving data, including speed and location information, and sends it to the server. This data is periodically transferred to the server using a secure communication protocol.

[0051] Step 6:

[0052] The server stores the received driving data in a database and performs AI-based analysis. The analysis extracts the driver's driving patterns and tendencies and generates safe driving indicators.

[0053] Step 7:

[0054] Based on the analysis results, the server generates specific suggestions for improving driving performance. These suggestions are compiled as advice and points of caution that take driving trends into account.

[0055] Step 8:

[0056] The server sends the generated driving improvement suggestions to the user's smartphone app. The user can then review the suggestions through the app and adjust their driving behavior accordingly.

[0057] Step 9:

[0058] If the user accepts the improvement suggestion, new operating data is generated based on the implementation results, and the process returns to step 5 for data transmission and analysis.

[0059] Step 10:

[0060] Driving data is regularly shared with insurance companies, and insurance premiums are optimized based on driving skills. The server also manages this data exchange, contributing to the adjustment of users' insurance premiums.

[0061] (Example 1)

[0062] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0063] In recent years, as safety in vehicle operation has become increasingly important to society, there is a growing demand for appropriate driving assistance and cost-effective driving instruction tailored to individual drivers. However, conventional technologies have had difficulties in recognizing road markings in real time to support driving, or in providing effective improvement suggestions based on drivers' driving tendencies. Furthermore, there has been a lack of mechanisms to effectively utilize individual driving data to optimize insurance premiums. Technologies that can solve these problems are needed.

[0064] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0065] In this invention, the server includes a processing unit that processes moving images obtained from a video acquisition device mounted on the vehicle in real time and identifies road markings; an output unit that generates audio output information based on the identified markings and notifies the driver; and an analysis unit that collects individual driving data and analyzes driving trends based on the data. As a result, the driver can recognize sign information in real time and receive appropriate warnings according to the driving situation. Furthermore, it is possible to promote safe driving and reduce the financial burden through driving improvement suggestions and insurance premium optimization based on the collected data.

[0066] A "vehicle-mounted video acquisition device" is a device installed inside or outside a vehicle to capture images of road conditions and the surrounding environment while driving.

[0067] "Real-time processing of video" means analyzing captured video data instantly without delay and extracting the necessary information.

[0068] "Identifying road markings" means identifying signs and traffic rule indicators installed on roads from acquired video footage, and clarifying their type and content.

[0069] "Generating voice output information" means creating data to provide drivers with voice notifications and guidance based on the analysis results.

[0070] "Notifying the driver" means communicating information or warnings to the vehicle's driver through audio or visual means.

[0071] "Collecting driving data" means recording and accumulating various data related to the operation of a vehicle (e.g., speed, position, acceleration).

[0072] "Analyzing driving trends" means analyzing accumulated driving data to understand and evaluate the driver's driving habits and characteristics.

[0073] "Adjusting insurance premiums" means evaluating the risk of individual drivers based on collected driving data and appropriately calculating insurance premiums in insurance contracts.

[0074] The embodiments for carrying out the present invention are described below.

[0075] The system of the present invention acquires road images in real time from a video acquisition device mounted on a vehicle to provide driving assistance. The terminal acquires road images in high resolution through a camera mounted on the vehicle. These acquired images are analyzed in real time using a generative AI model. Here, widely used models such as the YOLO (You Only Look Once) series and OpenCV's DNN module are used. These models have the ability to identify road signs quickly and with high accuracy.

[0076] The device generates audio output information based on identified road signs. This process utilizes speech synthesis engines such as Google® Text-to-Speech and Amazon Polly to convert visual information into audio. This allows drivers to receive road sign information in real time via audio, enabling them to concentrate on driving without diverting their gaze.

[0077] Furthermore, the device transmits speed, location information, and other driving data collected during driving to a server. The server stores this data in the cloud and is responsible for analyzing driving trends. By using AI analysis tools such as Amazon SageMaker, it is possible to understand the driver's driving habits and generate appropriate improvement suggestions.

[0078] The server generates specific improvement suggestions for the driver based on the analysis results and provides feedback to the driver via a portable information terminal such as a smartphone. The suggestions should be specific and actionable, and may include visual infographics and text explanations to aid the driver's understanding.

[0079] A further feature of this system is that the collected driving data is linked with the insurance company's system, and insurance premiums are calculated based on the driver's driving skills. This motivates drivers to drive safely while also providing economic benefits such as reduced insurance premiums.

[0080] To give specific examples, while driving on a highway, the device might send a voice notification saying, "The speed limit is 80 km / h," or a server might display advice on the smartphone saying, "By being mindful of the speed limit, you can improve fuel efficiency."

[0081] An example of a prompt to input into the generating AI model is, "Please describe the procedure for analyzing camera footage, recognizing road signs, and generating an audio notification."

[0082] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0083] Step 1:

[0084] The terminal continuously acquires road images using a camera mounted on the vehicle. The input is raw image data from the camera, and this data is temporarily stored in high resolution. Specifically, the camera captures several frames of images per second and saves them to a buffer, preparing for image analysis.

[0085] Step 2:

[0086] The device inputs the acquired video into a generating AI model to recognize road signs in real time. The input in this step is the image data acquired in step 1, and the frames containing this data are passed to the AI ​​model. Models such as YOLO perform feature extraction and classification to identify the location and type of the sign. The output is information about the recognized sign, including its location coordinates and type. Specifically, a rectangle is drawn around the recognized sign, and the type of sign is displayed as text next to it.

[0087] Step 3:

[0088] The device generates voice feedback based on recognized traffic sign information. The input is the traffic sign information, which is the output of step 2, and is converted into voice data using Google Text-to-Speech or Amazon Polly. The output is either a voice file or a real-time voice stream. Specifically, the generated voice is played through the car's audio system, providing the driver with an audible message such as "The speed limit is 80 km / h."

[0089] Step 4:

[0090] The terminal sends acquired driving data (speed, location information, traffic sign recognition results, etc.) to the server. The input is the aggregate of data acquired in steps 1 and 2, which is sent to the server in an encrypted format using the HTTPS protocol. The output is the dataset sent to the server. Specifically, data packets are periodically sent from the user's terminal to the cloud server, and the data storage process on the server side proceeds.

[0091] Step 5:

[0092] The server stores and analyzes the received data. The input is the driving data transmitted in step 4, which is used for statistical processing and machine learning analysis to understand the user's driving patterns. The output is the analysis results, which include driving trends and areas for improvement. Specifically, the processing is executed within a cloud computing environment, and the results are stored in a database.

[0093] Step 6:

[0094] The server generates driving improvement suggestions based on the analysis results and provides feedback to the user's smartphone. The input is the analysis results from step 5, which serve as the basis for generating advice on specific driving behaviors. The output is a notification message displayed in the smartphone application. Specifically, the app displays a pop-up advising the user, such as "Maintaining a consistent speed limit will improve fuel efficiency," prompting them to view it.

[0095] Step 7:

[0096] The server transmits driving data to the insurance company's system, and premium adjustments are made. The input is the driving data and analysis results obtained in steps 4 and 5, and a risk assessment is performed according to the insurance company's standards. The output is the updated insurance contract terms or premium. Specifically, a process proceeds in which a new premium discount based on the user's driving evaluation is calculated.

[0097] (Application Example 1)

[0098] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0099] In recent years, many car accidents have been caused by driver inattention or overlooking road signs. Furthermore, drivers often lack real-time feedback on their driving tendencies, hindering their ability to improve their safe driving habits. Therefore, there is a need to develop a system that improves the accuracy of road sign recognition, provides drivers with real-time and effective feedback, analyzes driving techniques based on individual driving data, and offers guidance for improvement.

[0100] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0101] In this invention, the server includes means for analyzing video acquired from an imaging device mounted on the vehicle, means for generating voice notifications based on recognized signs, and means for accumulating and analyzing driving data. This enables the provision of real-time and effective driving support to the driver, thereby promoting safe driving.

[0102] An "imaging device" is a device installed in a vehicle to acquire video information while it is in motion, and includes devices such as cameras.

[0103] "Means of analysis" refers to methods and devices for processing acquired video information to recognize road signs and analyze driving tendencies.

[0104] "Means for generating voice notifications" refers to a system for providing voice information to drivers based on recognized signs and other information.

[0105] "Driving data" refers to information related to driving conditions, such as speed, location information, and traffic sign recognition results obtained during driving.

[0106] "Means of accumulation and analysis" refers to methods and systems for saving collected driving data and using it to understand and analyze driving trends.

[0107] "Means of providing visual notification" refers to methods of providing visual information to the driver using display devices installed inside the vehicle.

[0108] A "portable information terminal" is a small digital device used to provide drivers with guidance on how to improve their driving, and includes smartphones and similar devices.

[0109] This invention is a system that provides driver assistance within a vehicle. The server acquires video from an imaging device mounted on the vehicle and analyzes this video in real time to recognize road signs. The hardware used includes an in-vehicle camera and edge computing devices, and the software employs machine learning libraries (e.g., TENSORFLOW®, PyTorch).

[0110] The terminal generates voice notifications based on analyzed sign information and provides them to the driver using speech synthesis technology (e.g., Google Text-to-Speech API). Visual notifications are also provided using in-vehicle displays. This allows drivers to obtain important driving information in real time.

[0111] Furthermore, the terminal stores driving data including speed, location information, and recognition results. This data is transferred to a cloud server and analyzed using data analysis tools (e.g., Apache® Hadoop). Based on the analysis results, the server identifies individual driving trends and generates improvement guidelines.

[0112] The improvement guidelines will be provided to drivers via portable information terminals, specifically smartphones. This will allow drivers to understand their own driving tendencies and make improvements as needed. Specific examples include situations where the system recognizes speed limit signs while driving on a highway and notifies the driver by voice, "The speed limit is 80 km / h."

[0113] Examples of prompts include: "Please describe in detail how to help design a system that analyzes in-vehicle camera video data in real time and provides drivers with audio and visual notifications regarding speed limits and road signs."

[0114] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0115] Step 1:

[0116] The terminal acquires video from the in-vehicle camera. The video data is passed to the terminal as input, and the terminal prepares this data for processing. Specifically, it converts the video to an appropriate resolution and performs noise reduction to make it suitable for analysis.

[0117] Step 2:

[0118] The device analyzes video in real time and recognizes road signs. The input is previously processed video data, and the device uses an AI model (e.g., a model using TensorFlow or PyTorch) to identify the signs. Data calculations include feature extraction and pattern matching. The output is the recognized sign information.

[0119] Step 3:

[0120] The device generates an audio notification based on recognized traffic sign information. The input is the traffic sign information obtained in step 2, and the device uses a speech synthesis API (e.g., Google Text-to-Speech) to generate an audio notification optimized for the driver. The output is an audio file.

[0121] Step 4:

[0122] The terminal plays audio files to provide information to the user. At this time, the audio is output using the vehicle's speakers. Furthermore, visual information is simultaneously provided using the vehicle's display device. This allows the user to be notified of important points to remember while driving in real time.

[0123] Step 5:

[0124] The terminal collects speed, location information, and traffic sign recognition results during driving, and stores them as driving data. This data is then sent to a cloud server. Measurement data is used as input, which the terminal organizes and converts into a transmittable format. As a result, transmittable data packets are output.

[0125] Step 6:

[0126] The server analyzes the received driving data to understand driving trends. The input is data sent from the terminal, and analysis is performed using data analysis tools (e.g., Apache Hadoop). As a result of the analysis, insights into the user's driving trends are obtained.

[0127] Step 7:

[0128] Based on the analysis results, the server generates driving improvement guidelines for the user. The input is the analysis results of driving trends, and the server uses the data to generate individual suggestions. The generated suggestions are output and notified to the user via a portable information terminal.

[0129] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0130] This invention is a system that combines a camera installed in a vehicle with an emotion engine to assist the user's driving. The emotion engine uses image analysis and voice analysis technologies to recognize the emotional state of the user from the video and audio acquired by the terminal.

[0131] First, the device acquires road footage via an in-car camera and simultaneously captures the user's facial expressions. Using AI, it recognizes road signs and analyzes the user's emotional state. During this process, it determines whether the user is irritated, relaxed, or tired.

[0132] Next, the device adjusts the voice feedback based on the recognized sign information and emotional information. For example, if the user is feeling anxious, it selectively provides encouraging messages such as "Please drive calmly" or confidence-boosting feedback.

[0133] The terminal then transmits this driving and emotional data to a server. The server stores this data and analyzes driving tendencies while considering the emotional state. Based on the analysis results, it generates suggestions for improving driving. These suggestions include personalized advice tailored to the user's emotions, such as "Drive without rushing."

[0134] Users can receive these suggestions and analysis results via their smartphones. Furthermore, emotional data is integrated with other driving data, and in collaboration with insurance companies, driving tendencies based on the user's emotional state are also used to optimize insurance premiums. Higher emotional stability may lead to lower insurance premiums.

[0135] For example, if a user shows signs of frustration while driving, the system will provide specific feedback such as, "Take a deep breath and relax." If the driving remains calm afterward, the server evaluates the result and provides positive feedback through the app. In this way, the present invention provides driving assistance that incorporates the user's emotions, realizing a safer and more secure driving experience.

[0136] The following describes the processing flow.

[0137] Step 1:

[0138] The device uses an in-car camera to simultaneously capture video of the road and the user's face. The road video is used to monitor driving conditions, and the facial video is used for analyzing the user's emotions.

[0139] Step 2:

[0140] The AI ​​installed in the device analyzes video data in real time, recognizing road signs and analyzing the user's emotional state. Using image recognition technology, it determines the type of emotion (e.g., irritation, concentration, relaxation) from the user's facial expressions.

[0141] Step 3:

[0142] The device generates necessary voice feedback for the driver based on recognized road signs, while simultaneously adjusting the feedback content based on emotion analysis results. For example, if the user is feeling stressed, it will generate a gentle and reassuring voice message such as, "Please drive safely."

[0143] Step 4:

[0144] The device plays the generated audio feedback through the car's speakers, providing it to the user. The user can then adjust their driving based on this feedback.

[0145] Step 5:

[0146] The device transmits driving data and emotional data to the server. This data includes speed, location, traffic sign recognition results, and emotional state.

[0147] Step 6:

[0148] The server stores the received data in a database and uses AI to analyze detailed driving trends. Here, emotional fluctuations are also treated as patterns and linked to driving tendencies.

[0149] Step 7:

[0150] The server generates suggestions for improving driving performance, including advice tailored to your emotional state. It offers ways to reduce stress and points to be mindful of.

[0151] Step 8:

[0152] The server sends the generated improvement suggestions to the user's smartphone app. The user can review the advice through the app and consider adjusting their driving to improve their emotional stability.

[0153] Step 9:

[0154] Driving information, including user emotional data, is used in collaboration with insurance companies to adjust insurance premiums. Driving with stable emotions is valued, contributing to the optimization of insurance premiums.

[0155] (Example 2)

[0156] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0157] Modern vehicle driving requires not only the ability to recognize road signs but also the ability to appropriately recognize the driver's emotional state and provide corresponding feedback. However, existing systems are insufficient in supporting drivers while considering their mental state, making it difficult to provide feedback based on emotional state or analyze driving tendencies. Furthermore, the use of data to optimize insurance premiums based on driving skills and emotional state is limited.

[0158] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0159] In this invention, the server includes means for analyzing video acquired from an imaging device installed in the vehicle in real time and recognizing road signs, means for recognizing emotional states using an AI model generated from the acquired video and audio data, and means for accumulating individual driving information and emotional state data and analyzing driving tendencies based on that data. This enables real-time support and safety improvements in response to the driver's emotional state, and also realizes personalized optimization of insurance premiums based on driving characteristics.

[0160] An "imaging device" is a device installed in a vehicle to capture images of the surroundings.

[0161] "Analysis" is the process of examining acquired data in detail and extracting specific information.

[0162] "Road sign recognition" refers to identifying specific shapes or patterns from acquired video data and determining that they are a type of road sign.

[0163] A "generative AI model" is an artificial intelligence algorithm or system that learns from large amounts of data and is used to perform a specific task.

[0164] "Emotional state" refers to the mental and emotional condition of the driver, as inferred from their facial expressions, tone of voice, and other factors.

[0165] "Feedback" refers to the process of providing information to the driver, either verbally or visually, based on acquired information and analysis results, in order to assist them.

[0166] "Driving information" refers to all data related to the operation of a vehicle, including information such as speed, location, and route.

[0167] An "external system" is an external information processing device or network that exists separately from the in-vehicle system and is connected for the purpose of sharing and analyzing data.

[0168] "Insurance premium optimization" means evaluating the driver's characteristics and risks based on acquired data, and calculating an appropriate insurance premium accordingly.

[0169] A "mobile device" is an electronic device that a user can carry with them, and includes smartphones, tablet devices, and other similar devices.

[0170] This invention is a system that combines an imaging device mounted on a vehicle with a generative AI model for recognizing emotional states to provide safer and more personalized driving assistance.

[0171] The server manages the main data processing, while the terminal performs critical real-time processing. The terminal acquires road video and driver audio using imaging and acoustic input devices installed in the vehicle. The video data is analyzed by image analysis software to recognize road signs. Meanwhile, the audio data is processed through acoustic analysis algorithms to identify emotional states. In this step, a generative AI model used to train a specific emotion model plays a crucial role.

[0172] Subsequently, the device generates and provides voice feedback to the user based on the recognized sign information and emotional state. This feedback is customized to take the user's emotional state into account and provide appropriate driving assistance.

[0173] The server stores driving information and emotional state data transmitted from the terminal and performs analysis based on this data. This analysis reveals driving trends and allows for the provision of more detailed improvement suggestions to the user. As a result, improved driver safety and driving performance are expected.

[0174] For example, if a user shows signs of frustration while driving, the device provides appropriate feedback such as, "Take a deep breath and relax." If the driving then proceeds smoothly, the server evaluates this and provides positive feedback to the user.

[0175] An example of a prompt is, "Design a program that analyzes the driver's emotional state from video and audio data acquired by an in-vehicle camera and generates real-time feedback." This prompt suggests the purpose of the invention and its specific implementation.

[0176] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0177] Step 1:

[0178] The terminal acquires road video and driver audio data from the vehicle's mounted imaging and audio input devices. Inputs include real-time video frames and audio signals. This data is recorded in an appropriate format as it is used directly in the next processing step.

[0179] Step 2:

[0180] The device analyzes acquired video data using image analysis software to recognize road signs. Video data is used as input. The image analysis algorithm identifies signs through feature extraction and pattern matching, and outputs the recognized sign information.

[0181] Step 3:

[0182] The device analyzes the input voice data using a generating AI model to recognize the driver's emotional state. Voice data is input, and processes for extracting acoustic features and estimating the emotional state are performed. The output is a label indicating the emotional state.

[0183] Step 4:

[0184] The device generates voice feedback based on recognized road sign information and emotional state. The sign information and emotional state labels are used as input, and a text generation engine generates an appropriate voice message. The output is voice feedback.

[0185] Step 5:

[0186] The device provides the user with generated voice feedback. The generated voice message is then played through the car's speakers. This allows the user to receive real-time instructions and advice.

[0187] Step 6:

[0188] The terminal transmits driving information and emotional state data, which are the results of all processing steps, to the server. The transmitted data is then used for subsequent analysis and evaluation.

[0189] Step 7:

[0190] The server stores the received data and analyzes driving trends. Past driving information and sentiment data are stored as input, and a report on driving trends based on a statistical model is generated as output.

[0191] Step 8:

[0192] Based on the analysis results, the server generates specific driving improvement suggestions for the user. Using the driving trend report as input, it outputs specific improvement points and advice, which are then provided to the user as feedback for the next driving session.

[0193] (Application Example 2)

[0194] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0195] In situations where a driver's emotional fluctuations during operation can affect driving performance, this project aims to improve safety by understanding the driver's emotional state in real time and providing appropriate feedback. Furthermore, it aims to contribute to reducing the burden on drivers by providing a new method for optimizing insurance coverage using emotional information.

[0196] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0197] In this invention, the server includes means for analyzing video acquired from an imaging device installed in the vehicle in real time and recognizing road signals; means for accumulating driving-related data and emotional information and analyzing driving tendencies based on that data; and means for generating improvement suggestions for the driver based on the analyzed driving tendencies and emotional information, and further presenting them visually and audibly. This enables safe driving assistance that takes into account the driver's emotional state and optimization of insurance premiums.

[0198] An "imaging device" is a device attached to a vehicle to acquire image information of the surroundings.

[0199] "Signals" refers to visual information that affects driving, including traffic signs and signals on the road.

[0200] The term "driver" refers to the person operating a vehicle, and their psychological state and driving skills affect the safety of the ride.

[0201] "Emotional information" refers to data representing the psychological state obtained from the pilot's facial expressions and voice.

[0202] "Real-time analysis" refers to a process where information is processed instantly the moment it is acquired, and results are derived immediately.

[0203] "Feedback" refers to information and instructions provided to the pilot based on analysis results, with the role of improving or supporting their actions.

[0204] "Driving-related data" refers to all information related to the operation of the vehicle, including information on speed, location, and driving style.

[0205] "Driving tendencies" refer to information that indicates the driver's driving style and patterns, extracted from accumulated data.

[0206] "Means of visual and auditory presentation" refers to methods of conveying information through displays and speakers in a way that the operator can immediately understand.

[0207] "Insurance premium optimization" refers to a method of calculating insurance premiums rationally by taking into account the driver's driving history and emotional stability.

[0208] The system implementing this invention is a driving assistance system that utilizes emotion recognition technology. This system consists of various devices installed in the vehicle and an external information processing system, and analyzes the driver's emotional state and driving data in real time to support safe driving.

[0209] First, the server uses an imaging device installed in the vehicle to capture images of the driver's facial expressions and the surrounding road environment. This video data is processed in real time using image recognition libraries such as OpenCV to recognize traffic signals and other visual road information. Simultaneously, emotional information is obtained from the driver's facial expressions. Emotion recognition uses machine learning models built with TensorFlow or PyTorch.

[0210] Next, the device generates feedback based on this information. It generates voice messages using Amazon Polly or the Google Text-to-Speech API, and also provides visual feedback to the driver via a display device. This allows the driver to receive appropriate driving assistance in real time, taking into account their emotional state.

[0211] The data is sent to a cloud-based external information processing system, where it is stored and analyzed by a server. During this process, driving trends are analyzed, and a data model for optimizing insurance premiums is generated using machine learning. In this process, a generative AI model is used to construct prompt messages based on past data. An example of a prompt message is, "Generate driving assistance messages that take into account the user's current emotional state. For example, suggest a message to encourage relaxation if the user is irritated."

[0212] To give a specific example, if excellent driving results in a stable emotional state even after long periods of driving, the server will generate and provide positive feedback as a reward to the driver. This encourages the driver to constantly be aware of their own driving and emotional stability, resulting in a safer and more comfortable driving experience.

[0213] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0214] Step 1:

[0215] The server acquires video data of the driver's facial expressions and the surrounding road environment through an imaging device installed in the vehicle. The input is video data from the imaging device, and image analysis is performed based on this data. The software used here is an image processing library such as OpenCV.

[0216] Step 2:

[0217] The server uses the acquired video data to perform image analysis and recognize traffic signals and other visual road information on the road. This process outputs the recognized visual information. A traffic signal recognition algorithm is used for data calculation.

[0218] Step 3:

[0219] The server inputs the pilot's facial expression data into an emotion recognition program, which then analyzes the emotional information. The input is the pilot's facial expression data, and the output is data indicating the emotional state. The calculations are performed using a machine learning model based on TensorFlow.

[0220] Step 4:

[0221] The device generates feedback based on signal recognition results and sentiment information. This feedback generation program uses Amazon Polly or the Google Text-to-Speech API to create audio data. Visual information is displayed on the screen. The input is signal recognition results and sentiment information, and the output is feedback to the operator.

[0222] Step 5:

[0223] The server uses a generated AI model to send piloting data and emotional information to a cloud system, where it is stored in a database. The inputs here are piloting data and emotional information, and the output is stored data on the cloud. Data storage processing is performed by the database management system.

[0224] Step 6:

[0225] The server analyzes accumulated data to determine driving trends. Based on the analysis results, it attempts to optimize insurance premiums. The input consists of driving history data and emotional stability data, and the output is an optimized insurance premium proposal. The analysis is performed using machine learning algorithms.

[0226] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0227] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0228] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0229] [Second Embodiment]

[0230] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0231] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0232] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0233] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0234] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0235] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0236] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0237] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0238] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0239] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0240] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0241] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0242] This invention relates to a system that provides driving assistance using road images acquired by a camera installed in a vehicle. First, a terminal continuously acquires images using the in-vehicle camera and uses AI to recognize road signs in real time from those images. Based on the recognized signs, the terminal generates voice feedback and provides it directly to the user. This voice feedback serves as a notification to encourage attention while driving.

[0243] Furthermore, the device transmits data acquired during driving, namely speed, location information, and traffic sign recognition results, to the server. The server stores this data and analyzes it using AI to understand the user's driving tendencies. Based on the analysis results, the server generates personalized driving improvement suggestions and provides feedback to the user via smartphone.

[0244] For example, when driving on a highway, the device recognizes speed limit signs and provides a voice warning, "The speed limit is 80 km / h." Furthermore, if the server detects frequent speeding based on past data, it provides specific advice to the user via the smartphone app, such as, "Adhering to the speed limit will improve fuel efficiency."

[0245] Furthermore, this driving data is linked with insurance companies, and the system also includes a mechanism to adjust users' insurance premiums based on their driving skills. As a result, users are motivated to drive safely and can also enjoy the benefit of lower insurance premiums.

[0246] As described above, the system of the present invention reduces the risk of accidents and promotes safe driving by providing individually optimized driving support to all drivers, including the elderly and novice drivers.

[0247] The following describes the processing flow.

[0248] Step 1:

[0249] The terminal uses a camera installed in the vehicle to acquire road images in real time. The camera continuously captures the surrounding environment while driving and generates video data.

[0250] Step 2:

[0251] The AI ​​installed in the device analyzes the acquired video data and recognizes road signs within the video. The AI ​​uses an image recognition algorithm to identify speed limit signs, stop signs, and other similar signs.

[0252] Step 3:

[0253] The device generates voice feedback based on recognized signs. Using speech synthesis technology, it creates messages such as "The speed limit is XX km / h" or "Please stop at the next intersection."

[0254] Step 4:

[0255] The device plays the generated audio feedback through the car's speakers, notifying the user. This allows the user to receive signage information in real time.

[0256] Step 5:

[0257] The terminal collects driving data, including speed and location information, and sends it to the server. This data is periodically transferred to the server using a secure communication protocol.

[0258] Step 6:

[0259] The server stores the received driving data in a database and performs AI-based analysis. The analysis extracts the driver's driving patterns and tendencies and generates safe driving indicators.

[0260] Step 7:

[0261] Based on the analysis results, the server generates specific suggestions for improving driving performance. These suggestions are compiled as advice and points of caution that take driving trends into account.

[0262] Step 8:

[0263] The server sends the generated driving improvement suggestions to the user's smartphone app. The user can then review the suggestions through the app and adjust their driving behavior accordingly.

[0264] Step 9:

[0265] If the user accepts the improvement suggestion, new operating data is generated based on the implementation results, and the process returns to step 5 for data transmission and analysis.

[0266] Step 10:

[0267] Driving data is regularly shared with insurance companies, and insurance premiums are optimized based on driving skills. The server also manages this data exchange, contributing to the adjustment of users' insurance premiums.

[0268] (Example 1)

[0269] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0270] In recent years, as safety in vehicle operation has become increasingly important to society, there is a growing demand for appropriate driving assistance and cost-effective driving instruction tailored to individual drivers. However, conventional technologies have had difficulties in recognizing road markings in real time to support driving, or in providing effective improvement suggestions based on drivers' driving tendencies. Furthermore, there has been a lack of mechanisms to effectively utilize individual driving data to optimize insurance premiums. Technologies that can solve these problems are needed.

[0271] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0272] In this invention, the server includes a processing unit that processes moving images obtained from a video acquisition device mounted on the vehicle in real time and identifies road markings; an output unit that generates audio output information based on the identified markings and notifies the driver; and an analysis unit that collects individual driving data and analyzes driving trends based on the data. As a result, the driver can recognize sign information in real time and receive appropriate warnings according to the driving situation. Furthermore, it is possible to promote safe driving and reduce the financial burden through driving improvement suggestions and insurance premium optimization based on the collected data.

[0273] A "vehicle-mounted video acquisition device" is a device installed inside or outside a vehicle to capture images of road conditions and the surrounding environment while driving.

[0274] "Real-time processing of video" means analyzing captured video data instantly without delay and extracting the necessary information.

[0275] "Identifying road markings" means identifying signs and traffic rule indicators installed on roads from acquired video footage, and clarifying their type and content.

[0276] "Generating voice output information" means creating data to provide drivers with voice notifications and guidance based on the analysis results.

[0277] "Notifying the driver" means communicating information or warnings to the vehicle's driver through audio or visual means.

[0278] "Collecting driving data" means recording and accumulating various data related to the operation of a vehicle (e.g., speed, position, acceleration).

[0279] "Analyzing driving trends" means analyzing accumulated driving data to understand and evaluate the driver's driving habits and characteristics.

[0280] "Adjusting insurance premiums" means evaluating the risks of individual drivers based on the collected driving data and appropriately calculating the insurance premiums in the insurance contract.

[0281] The embodiments for implementing the present invention will be described below.

[0282] The system of the present invention acquires the road video in real time from the video acquisition device mounted on the vehicle to realize driving assistance. The terminal acquires the road video with high resolution through the camera attached to the vehicle. The acquired video is analyzed in real time by utilizing the generated AI model. Here, widely used models such as the YOLO (You Only Look Once) series and the DNN module of OpenCV are used. These models have the ability to identify road signs quickly and accurately.

[0283] The terminal generates output information by voice based on the identified road signs. In this process, Google Text-to-Speech, Amazon Polly, etc. are used as the text-to-speech engine to vocalize visual information. As a result, the driver can receive road sign information by voice in real time and can concentrate on driving without diverting their line of sight.

[0284] Furthermore, the terminal transmits the speed, position information, and other driving data collected during driving to the server. The server stores these data on the cloud and plays the role of analyzing the driving trend. By using an AI analysis tool such as Amazon SageMaker, the driving habits of the driver can be grasped and appropriate improvement suggestions can be generated.

[0285] The server creates specific improvement suggestions for the driver based on the analysis results and feedbacks them to the driver through a portable information terminal such as a smartphone. The form of the suggestions should be specific and actionable, and visual infographics and text explanations can be included to assist the driver's understanding.

[0286] As a further feature of this system, the collected driving data is also linked to the insurance company's system, and the insurance premium is calculated based on the driver's driving skills. As a result, the driver can be motivated to drive safely and at the same time can receive economic benefits such as a reduction in the insurance premium.

[0287] For a specific example, it is conceivable that while driving on a highway, the terminal emits an audio notification saying "The speed limit is 80 km / h", or the server displays an advice saying "Pay attention to speed compliance to improve fuel efficiency" on the smartphone.

[0288] An example of a prompt sentence to be input into the generative AI model is "Please explain the procedure for analyzing the camera image, recognizing road signs, and generating an audio notification."

[0289] The flow of the specific process in Example 1 will be described with reference to FIG. 11.

[0290] Step 1:

[0291] The terminal continuously acquires an image of the road using the camera mounted on the vehicle. The input is the original image data from the camera, and this data is temporarily stored in a high-resolution state. As a specific operation, the camera captures images at several frames per second and stores them in a buffer to prepare for image analysis.

[0292] Step 2:

[0293] The terminal inputs the acquired video into the generative AI model to recognize road signs in real time. The input in this step is the image data acquired in Step 1, and the frame including this data is passed to the AI model. A model such as YOLO performs feature extraction and classification to identify the position and type of the sign. The output is the information of the recognized sign, including the position coordinates and the type. As a specific operation, a rectangle is drawn for the recognized sign, and the type of the sign is displayed in text beside it.

[0294] Step 3:

[0295] The device generates voice feedback based on recognized traffic sign information. The input is the traffic sign information, which is the output of step 2, and is converted into voice data using Google Text-to-Speech or Amazon Polly. The output is either a voice file or a real-time voice stream. Specifically, the generated voice is played through the car's audio system, providing the driver with an audible message such as "The speed limit is 80 km / h."

[0296] Step 4:

[0297] The terminal sends acquired driving data (speed, location information, traffic sign recognition results, etc.) to the server. The input is the aggregate of data acquired in steps 1 and 2, which is sent to the server in an encrypted format using the HTTPS protocol. The output is the dataset sent to the server. Specifically, data packets are periodically sent from the user's terminal to the cloud server, and the data storage process on the server side proceeds.

[0298] Step 5:

[0299] The server stores and analyzes the received data. The input is the driving data transmitted in step 4, which is used for statistical processing and machine learning analysis to understand the user's driving patterns. The output is the analysis results, which include driving trends and areas for improvement. Specifically, the processing is executed within a cloud computing environment, and the results are stored in a database.

[0300] Step 6:

[0301] Based on the analysis results, the server generates driving improvement suggestions and provides feedback to the user's smartphone. The input is the analysis result of step 5, and based on this, advice on specific driving actions is generated. The output is a notification message displayed in the smartphone application. As a specific operation, the application pops up and displays advice such as "Pay attention to speed compliance to improve fuel efficiency" to prompt the user to view.

[0302] Step 7:

[0303] The server transmits the driving data to the insurance company's system, and the insurance premium is adjusted. The input is the driving data and analysis results obtained in steps 4 and 5, and a risk assessment is performed according to the insurance company's criteria. The output is the updated insurance contract conditions or insurance premium. As a specific operation, a process of calculating a new insurance premium discount based on the user's driving evaluation proceeds.

[0304] (Application Example 1)

[0305] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0306] In recent years, many automobile accidents are sometimes caused by driver inattention or overlooking signs. Also, it is difficult for drivers to obtain feedback on their driving tendencies in real time and use it to improve their awareness of safe driving. Therefore, there is a need to develop a system that improves the recognition accuracy of road signs, provides real-time and effective feedback to drivers, analyzes driving skills based on individual driving data, and presents improvement guidelines.

[0307] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0308] In this invention, the server includes means for analyzing video acquired from an imaging device mounted on the vehicle, means for generating voice notifications based on recognized signs, and means for accumulating and analyzing driving data. This enables the provision of real-time and effective driving support to the driver, thereby promoting safe driving.

[0309] An "imaging device" is a device installed in a vehicle to acquire video information while it is in motion, and includes devices such as cameras.

[0310] "Means of analysis" refers to methods and devices for processing acquired video information to recognize road signs and analyze driving tendencies.

[0311] "Means for generating voice notifications" refers to a system for providing voice information to drivers based on recognized signs and other information.

[0312] "Driving data" refers to information related to driving conditions, such as speed, location information, and traffic sign recognition results obtained during driving.

[0313] "Means of accumulation and analysis" refers to methods and systems for saving collected driving data and using it to understand and analyze driving trends.

[0314] "Means of providing visual notification" refers to methods of providing visual information to the driver using display devices installed inside the vehicle.

[0315] A "portable information terminal" is a small digital device used to provide drivers with guidance on how to improve their driving, and includes smartphones and similar devices.

[0316] This invention is a system that provides driver assistance within a vehicle. The server acquires video from an imaging device mounted on the vehicle and analyzes this video in real time to recognize road signs. The hardware used includes an in-vehicle camera and edge computing devices, and the software employs machine learning libraries (e.g., TensorFlow, PyTorch).

[0317] The terminal generates voice notifications based on analyzed sign information and provides them to the driver using speech synthesis technology (e.g., Google Text-to-Speech API). Visual notifications are also provided using in-vehicle displays. This allows drivers to obtain important driving information in real time.

[0318] Furthermore, the terminal stores driving data including speed, location information, and recognition results. This data is transferred to a cloud server and analyzed using data analysis tools (e.g., Apache Hadoop). Based on the analysis results, the server identifies individual driving trends and generates improvement guidelines.

[0319] The improvement guidelines will be provided to drivers via portable information terminals, specifically smartphones. This will allow drivers to understand their own driving tendencies and make improvements as needed. Specific examples include situations where the system recognizes speed limit signs while driving on a highway and notifies the driver by voice, "The speed limit is 80 km / h."

[0320] Examples of prompts include: "Please describe in detail how to help design a system that analyzes in-vehicle camera video data in real time and provides drivers with audio and visual notifications regarding speed limits and road signs."

[0321] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0322] Step 1:

[0323] The terminal acquires video from the in-vehicle camera. The video data is passed to the terminal as input, and the terminal prepares this data for processing. Specifically, it converts the video to an appropriate resolution and performs noise reduction to make it suitable for analysis.

[0324] Step 2:

[0325] The device analyzes video in real time and recognizes road signs. The input is previously processed video data, and the device uses an AI model (e.g., a model using TensorFlow or PyTorch) to identify the signs. Data calculations include feature extraction and pattern matching. The output is the recognized sign information.

[0326] Step 3:

[0327] The device generates an audio notification based on recognized traffic sign information. The input is the traffic sign information obtained in step 2, and the device uses a speech synthesis API (e.g., Google Text-to-Speech) to generate an audio notification optimized for the driver. The output is an audio file.

[0328] Step 4:

[0329] The terminal plays audio files to provide information to the user. At this time, the audio is output using the vehicle's speakers. Furthermore, visual information is simultaneously provided using the vehicle's display device. This allows the user to be notified of important points to remember while driving in real time.

[0330] Step 5:

[0331] The terminal collects speed, location information, and traffic sign recognition results during driving, and stores them as driving data. This data is then sent to a cloud server. Measurement data is used as input, which the terminal organizes and converts into a transmittable format. As a result, transmittable data packets are output.

[0332] Step 6:

[0333] The server analyzes the received driving data to understand driving trends. The input is data sent from the terminal, and analysis is performed using data analysis tools (e.g., Apache Hadoop). As a result of the analysis, insights into the user's driving trends are obtained.

[0334] Step 7:

[0335] Based on the analysis results, the server generates driving improvement guidelines for the user. The input is the analysis results of driving trends, and the server uses the data to generate individual suggestions. The generated suggestions are output and notified to the user via a portable information terminal.

[0336] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0337] This invention is a system that combines a camera installed in a vehicle with an emotion engine to assist the user's driving. The emotion engine uses image analysis and voice analysis technologies to recognize the emotional state of the user from the video and audio acquired by the terminal.

[0338] First, the device acquires road footage via an in-car camera and simultaneously captures the user's facial expressions. Using AI, it recognizes road signs and analyzes the user's emotional state. During this process, it determines whether the user is irritated, relaxed, or tired.

[0339] Next, the device adjusts the voice feedback based on the recognized sign information and emotional information. For example, if the user is feeling anxious, it selectively provides encouraging messages such as "Please drive calmly" or confidence-boosting feedback.

[0340] The terminal then transmits this driving and emotional data to a server. The server stores this data and analyzes driving tendencies while considering the emotional state. Based on the analysis results, it generates suggestions for improving driving. These suggestions include personalized advice tailored to the user's emotions, such as "Drive without rushing."

[0341] Users can receive these suggestions and analysis results via their smartphones. Furthermore, emotional data is integrated with other driving data, and in collaboration with insurance companies, driving tendencies based on the user's emotional state are also used to optimize insurance premiums. Higher emotional stability may lead to lower insurance premiums.

[0342] For example, if a user shows signs of frustration while driving, the system will provide specific feedback such as, "Take a deep breath and relax." If the driving remains calm afterward, the server evaluates the result and provides positive feedback through the app. In this way, the present invention provides driving assistance that incorporates the user's emotions, realizing a safer and more secure driving experience.

[0343] The following describes the processing flow.

[0344] Step 1:

[0345] The device uses an in-car camera to simultaneously capture video of the road and the user's face. The road video is used to monitor driving conditions, and the facial video is used for analyzing the user's emotions.

[0346] Step 2:

[0347] The AI ​​installed in the device analyzes video data in real time, recognizing road signs and analyzing the user's emotional state. Using image recognition technology, it determines the type of emotion (e.g., irritation, concentration, relaxation) from the user's facial expressions.

[0348] Step 3:

[0349] The device generates necessary voice feedback for the driver based on recognized road signs, while simultaneously adjusting the feedback content based on emotion analysis results. For example, if the user is feeling stressed, it will generate a gentle and reassuring voice message such as, "Please drive safely."

[0350] Step 4:

[0351] The device plays the generated audio feedback through the car's speakers, providing it to the user. The user can then adjust their driving based on this feedback.

[0352] Step 5:

[0353] The device transmits driving data and emotional data to the server. This data includes speed, location, traffic sign recognition results, and emotional state.

[0354] Step 6:

[0355] The server stores the received data in a database and uses AI to analyze detailed driving trends. Here, emotional fluctuations are also treated as patterns and linked to driving tendencies.

[0356] Step 7:

[0357] The server generates suggestions for improving driving performance, including advice tailored to your emotional state. It offers ways to reduce stress and points to be mindful of.

[0358] Step 8:

[0359] The server sends the generated improvement suggestions to the user's smartphone app. The user can review the advice through the app and consider adjusting their driving to improve their emotional stability.

[0360] Step 9:

[0361] Driving information, including user emotional data, is used in collaboration with insurance companies to adjust insurance premiums. Driving with stable emotions is valued, contributing to the optimization of insurance premiums.

[0362] (Example 2)

[0363] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0364] Modern vehicle driving requires not only the ability to recognize road signs but also the ability to appropriately recognize the driver's emotional state and provide corresponding feedback. However, existing systems are insufficient in supporting drivers while considering their mental state, making it difficult to provide feedback based on emotional state or analyze driving tendencies. Furthermore, the use of data to optimize insurance premiums based on driving skills and emotional state is limited.

[0365] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0366] In this invention, the server includes means for analyzing video acquired from an imaging device installed in the vehicle in real time and recognizing road signs, means for recognizing emotional states using an AI model generated from the acquired video and audio data, and means for accumulating individual driving information and emotional state data and analyzing driving tendencies based on that data. This enables real-time support and safety improvements in response to the driver's emotional state, and also realizes personalized optimization of insurance premiums based on driving characteristics.

[0367] An "imaging device" is a device installed in a vehicle to capture images of the surroundings.

[0368] "Analysis" is the process of examining acquired data in detail and extracting specific information.

[0369] "Road sign recognition" refers to identifying specific shapes or patterns from acquired video data and determining that they are a type of road sign.

[0370] A "generative AI model" is an artificial intelligence algorithm or system that learns from large amounts of data and is used to perform a specific task.

[0371] "Emotional state" refers to the mental and emotional condition of the driver, as inferred from their facial expressions, tone of voice, and other factors.

[0372] "Feedback" refers to the process of providing information to the driver, either verbally or visually, based on acquired information and analysis results, in order to assist them.

[0373] "Driving information" refers to all data related to the operation of a vehicle, including information such as speed, location, and route.

[0374] An "external system" is an external information processing device or network that exists separately from the in-vehicle system and is connected for the purpose of sharing and analyzing data.

[0375] "Insurance premium optimization" means evaluating the driver's characteristics and risks based on acquired data, and calculating an appropriate insurance premium accordingly.

[0376] A "mobile device" is an electronic device that a user can carry with them, and includes smartphones, tablet devices, and other similar devices.

[0377] This invention is a system that combines an imaging device mounted on a vehicle with a generative AI model for recognizing emotional states to provide safer and more personalized driving assistance.

[0378] The server manages the main data processing, while the terminal performs critical real-time processing. The terminal acquires road video and driver audio using imaging and acoustic input devices installed in the vehicle. The video data is analyzed by image analysis software to recognize road signs. Meanwhile, the audio data is processed through acoustic analysis algorithms to identify emotional states. In this step, a generative AI model used to train a specific emotion model plays a crucial role.

[0379] Subsequently, the device generates and provides voice feedback to the user based on the recognized sign information and emotional state. This feedback is customized to take the user's emotional state into account and provide appropriate driving assistance.

[0380] The server stores driving information and emotional state data transmitted from the terminal and performs analysis based on this data. This analysis reveals driving trends and allows for the provision of more detailed improvement suggestions to the user. As a result, improved driver safety and driving performance are expected.

[0381] For example, if a user shows signs of frustration while driving, the device provides appropriate feedback such as, "Take a deep breath and relax." If the driving then proceeds smoothly, the server evaluates this and provides positive feedback to the user.

[0382] An example of a prompt is, "Design a program that analyzes the driver's emotional state from video and audio data acquired by an in-vehicle camera and generates real-time feedback." This prompt suggests the purpose of the invention and its specific implementation.

[0383] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0384] Step 1:

[0385] The terminal acquires road video and driver audio data from the vehicle's mounted imaging and audio input devices. Inputs include real-time video frames and audio signals. This data is recorded in an appropriate format as it is used directly in the next processing step.

[0386] Step 2:

[0387] The device analyzes acquired video data using image analysis software to recognize road signs. Video data is used as input. The image analysis algorithm identifies signs through feature extraction and pattern matching, and outputs the recognized sign information.

[0388] Step 3:

[0389] The device analyzes the input voice data using a generating AI model to recognize the driver's emotional state. Voice data is input, and processes for extracting acoustic features and estimating the emotional state are performed. The output is a label indicating the emotional state.

[0390] Step 4:

[0391] The device generates voice feedback based on recognized road sign information and emotional state. The sign information and emotional state labels are used as input, and a text generation engine generates an appropriate voice message. The output is voice feedback.

[0392] Step 5:

[0393] The device provides the user with generated voice feedback. The generated voice message is then played through the car's speakers. This allows the user to receive real-time instructions and advice.

[0394] Step 6:

[0395] The terminal transmits driving information and emotional state data, which are the results of all processing steps, to the server. The transmitted data is then used for subsequent analysis and evaluation.

[0396] Step 7:

[0397] The server stores the received data and analyzes driving trends. Past driving information and sentiment data are stored as input, and a report on driving trends based on a statistical model is generated as output.

[0398] Step 8:

[0399] Based on the analysis results, the server generates specific driving improvement suggestions for the user. Using the driving trend report as input, it outputs specific improvement points and advice, which are then provided to the user as feedback for the next driving session.

[0400] (Application Example 2)

[0401] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0402] In situations where a driver's emotional fluctuations during operation can affect driving performance, this project aims to improve safety by understanding the driver's emotional state in real time and providing appropriate feedback. Furthermore, it aims to contribute to reducing the burden on drivers by providing a new method for optimizing insurance coverage using emotional information.

[0403] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0404] In this invention, the server includes means for analyzing video acquired from an imaging device installed in the vehicle in real time and recognizing road signals; means for accumulating driving-related data and emotional information and analyzing driving tendencies based on that data; and means for generating improvement suggestions for the driver based on the analyzed driving tendencies and emotional information, and further presenting them visually and audibly. This enables safe driving assistance that takes into account the driver's emotional state and optimization of insurance premiums.

[0405] An "imaging device" is a device attached to a vehicle to acquire image information of the surroundings.

[0406] "Signals" refers to visual information that affects driving, including traffic signs and signals on the road.

[0407] The term "driver" refers to the person operating a vehicle, and their psychological state and driving skills affect the safety of the ride.

[0408] "Emotional information" refers to data representing the psychological state obtained from the pilot's facial expressions and voice.

[0409] "Real-time analysis" refers to a process where information is processed instantly the moment it is acquired, and results are derived immediately.

[0410] "Feedback" refers to information and instructions provided to the pilot based on analysis results, with the role of improving or supporting their actions.

[0411] "Driving-related data" refers to all information related to the operation of the vehicle, including information on speed, location, and driving style.

[0412] "Driving tendencies" refer to information that indicates the driver's driving style and patterns, extracted from accumulated data.

[0413] "Means of visual and auditory presentation" refers to methods of conveying information through displays and speakers in a way that the operator can immediately understand.

[0414] "Insurance premium optimization" refers to a method of calculating insurance premiums rationally by taking into account the driver's driving history and emotional stability.

[0415] The system implementing this invention is a driving assistance system that utilizes emotion recognition technology. This system consists of various devices installed in the vehicle and an external information processing system, and analyzes the driver's emotional state and driving data in real time to support safe driving.

[0416] First, the server uses an imaging device installed in the vehicle to capture images of the driver's facial expressions and the surrounding road environment. This video data is processed in real time using image recognition libraries such as OpenCV to recognize traffic signals and other visual road information. Simultaneously, emotional information is obtained from the driver's facial expressions. Emotion recognition uses machine learning models built with TensorFlow or PyTorch.

[0417] Next, the device generates feedback based on this information. It generates voice messages using Amazon Polly or the Google Text-to-Speech API, and also provides visual feedback to the driver via a display device. This allows the driver to receive appropriate driving assistance in real time, taking into account their emotional state.

[0418] The data is sent to a cloud-based external information processing system, where it is stored and analyzed by a server. During this process, driving trends are analyzed, and a data model for optimizing insurance premiums is generated using machine learning. In this process, a generative AI model is used to construct prompt messages based on past data. An example of a prompt message is, "Generate driving assistance messages that take into account the user's current emotional state. For example, suggest a message to encourage relaxation if the user is irritated."

[0419] To give a specific example, if excellent driving results in a stable emotional state even after long periods of driving, the server will generate and provide positive feedback as a reward to the driver. This encourages the driver to constantly be aware of their own driving and emotional stability, resulting in a safer and more comfortable driving experience.

[0420] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0421] Step 1:

[0422] The server acquires video data of the driver's facial expressions and the surrounding road environment through an imaging device installed in the vehicle. The input is video data from the imaging device, and image analysis is performed based on this data. The software used here is an image processing library such as OpenCV.

[0423] Step 2:

[0424] The server uses the acquired video data to perform image analysis and recognize traffic signals and other visual road information on the road. This process outputs the recognized visual information. A traffic signal recognition algorithm is used for data calculation.

[0425] Step 3:

[0426] The server inputs the pilot's facial expression data into an emotion recognition program, which then analyzes the emotional information. The input is the pilot's facial expression data, and the output is data indicating the emotional state. The calculations are performed using a machine learning model based on TensorFlow.

[0427] Step 4:

[0428] The device generates feedback based on signal recognition results and sentiment information. This feedback generation program uses Amazon Polly or the Google Text-to-Speech API to create audio data. Visual information is displayed on the screen. The input is signal recognition results and sentiment information, and the output is feedback to the operator.

[0429] Step 5:

[0430] The server uses a generated AI model to send piloting data and emotional information to a cloud system, where it is stored in a database. The inputs here are piloting data and emotional information, and the output is stored data on the cloud. Data storage processing is performed by the database management system.

[0431] Step 6:

[0432] The server analyzes accumulated data to determine driving trends. Based on the analysis results, it attempts to optimize insurance premiums. The input consists of driving history data and emotional stability data, and the output is an optimized insurance premium proposal. The analysis is performed using machine learning algorithms.

[0433] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0434] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0435] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0436] [Third Embodiment]

[0437] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0438] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0439] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0440] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0441] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0442] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0443] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0444] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0445] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0446] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0447] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0448] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0449] This invention relates to a system that provides driving assistance using road images acquired by a camera installed in a vehicle. First, a terminal continuously acquires images using the in-vehicle camera and uses AI to recognize road signs in real time from those images. Based on the recognized signs, the terminal generates voice feedback and provides it directly to the user. This voice feedback serves as a notification to encourage attention while driving.

[0450] Furthermore, the device transmits data acquired during driving, namely speed, location information, and traffic sign recognition results, to the server. The server stores this data and analyzes it using AI to understand the user's driving tendencies. Based on the analysis results, the server generates personalized driving improvement suggestions and provides feedback to the user via smartphone.

[0451] For example, when driving on a highway, the device recognizes speed limit signs and provides a voice warning, "The speed limit is 80 km / h." Furthermore, if the server detects frequent speeding based on past data, it provides specific advice to the user via the smartphone app, such as, "Adhering to the speed limit will improve fuel efficiency."

[0452] Furthermore, this driving data is linked with insurance companies, and the system also includes a mechanism to adjust users' insurance premiums based on their driving skills. As a result, users are motivated to drive safely and can also enjoy the benefit of lower insurance premiums.

[0453] As described above, the system of the present invention reduces the risk of accidents and promotes safe driving by providing individually optimized driving support to all drivers, including the elderly and novice drivers.

[0454] The following describes the processing flow.

[0455] Step 1:

[0456] The terminal uses a camera installed in the vehicle to acquire road images in real time. The camera continuously captures the surrounding environment while driving and generates video data.

[0457] Step 2:

[0458] The AI ​​installed in the device analyzes the acquired video data and recognizes road signs within the video. The AI ​​uses an image recognition algorithm to identify speed limit signs, stop signs, and other similar signs.

[0459] Step 3:

[0460] The device generates voice feedback based on recognized signs. Using speech synthesis technology, it creates messages such as "The speed limit is XX km / h" or "Please stop at the next intersection."

[0461] Step 4:

[0462] The device plays the generated audio feedback through the car's speakers, notifying the user. This allows the user to receive signage information in real time.

[0463] Step 5:

[0464] The terminal collects driving data, including speed and location information, and sends it to the server. This data is periodically transferred to the server using a secure communication protocol.

[0465] Step 6:

[0466] The server stores the received driving data in a database and performs AI-based analysis. The analysis extracts the driver's driving patterns and tendencies and generates safe driving indicators.

[0467] Step 7:

[0468] Based on the analysis results, the server generates specific suggestions for improving driving performance. These suggestions are compiled as advice and points of caution that take driving trends into account.

[0469] Step 8:

[0470] The server sends the generated driving improvement suggestions to the user's smartphone app. The user can then review the suggestions through the app and adjust their driving behavior accordingly.

[0471] Step 9:

[0472] If the user accepts the improvement suggestion, new operating data is generated based on the implementation results, and the process returns to step 5 for data transmission and analysis.

[0473] Step 10:

[0474] Driving data is regularly shared with insurance companies, and insurance premiums are optimized based on driving skills. The server also manages this data exchange, contributing to the adjustment of users' insurance premiums.

[0475] (Example 1)

[0476] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0477] In recent years, as safety in vehicle operation has become increasingly important to society, there is a growing demand for appropriate driving assistance and cost-effective driving instruction tailored to individual drivers. However, conventional technologies have had difficulties in recognizing road markings in real time to support driving, or in providing effective improvement suggestions based on drivers' driving tendencies. Furthermore, there has been a lack of mechanisms to effectively utilize individual driving data to optimize insurance premiums. Technologies that can solve these problems are needed.

[0478] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0479] In this invention, the server includes a processing unit that processes moving images obtained from a video acquisition device mounted on the vehicle in real time and identifies road markings; an output unit that generates audio output information based on the identified markings and notifies the driver; and an analysis unit that collects individual driving data and analyzes driving trends based on the data. As a result, the driver can recognize sign information in real time and receive appropriate warnings according to the driving situation. Furthermore, it is possible to promote safe driving and reduce the financial burden through driving improvement suggestions and insurance premium optimization based on the collected data.

[0480] A "vehicle-mounted video acquisition device" is a device installed inside or outside a vehicle to capture images of road conditions and the surrounding environment while driving.

[0481] "Real-time processing of video" means analyzing captured video data instantly without delay and extracting the necessary information.

[0482] "Identifying road markings" means identifying signs and traffic rule indicators installed on roads from acquired video footage, and clarifying their type and content.

[0483] "Generating voice output information" means creating data to provide drivers with voice notifications and guidance based on the analysis results.

[0484] "Notifying the driver" means communicating information or warnings to the vehicle's driver through audio or visual means.

[0485] "Collecting driving data" means recording and accumulating various data related to the operation of a vehicle (e.g., speed, position, acceleration).

[0486] "Analyzing driving trends" means analyzing accumulated driving data to understand and evaluate the driver's driving habits and characteristics.

[0487] "Adjusting insurance premiums" means evaluating the risk of individual drivers based on collected driving data and appropriately calculating insurance premiums in insurance contracts.

[0488] The embodiments for carrying out the present invention are described below.

[0489] The system of the present invention acquires road images in real time from a video acquisition device mounted on a vehicle to provide driving assistance. The terminal acquires road images in high resolution through a camera mounted on the vehicle. These acquired images are analyzed in real time using a generative AI model. Here, widely used models such as the YOLO (You Only Look Once) series and OpenCV's DNN module are used. These models have the ability to identify road signs quickly and with high accuracy.

[0490] The device generates audio output information based on identified road signs. This process utilizes speech synthesis engines such as Google Text-to-Speech and Amazon Polly to convert visual information into audio. This allows drivers to receive road sign information in real time via audio, enabling them to concentrate on driving without taking their eyes off the road.

[0491] Furthermore, the device transmits speed, location information, and other driving data collected during driving to a server. The server stores this data in the cloud and is responsible for analyzing driving trends. By using AI analysis tools such as Amazon SageMaker, it is possible to understand the driver's driving habits and generate appropriate improvement suggestions.

[0492] The server generates specific improvement suggestions for the driver based on the analysis results and provides feedback to the driver via a portable information terminal such as a smartphone. The suggestions should be specific and actionable, and may include visual infographics and text explanations to aid the driver's understanding.

[0493] A further feature of this system is that the collected driving data is linked with the insurance company's system, and insurance premiums are calculated based on the driver's driving skills. This motivates drivers to drive safely while also providing economic benefits such as reduced insurance premiums.

[0494] To give specific examples, while driving on a highway, the device might send a voice notification saying, "The speed limit is 80 km / h," or a server might display advice on the smartphone saying, "By being mindful of the speed limit, you can improve fuel efficiency."

[0495] An example of a prompt to input into the generating AI model is, "Please describe the procedure for analyzing camera footage, recognizing road signs, and generating an audio notification."

[0496] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0497] Step 1:

[0498] The terminal continuously acquires road images using a camera mounted on the vehicle. The input is raw image data from the camera, and this data is temporarily stored in high resolution. Specifically, the camera captures several frames of images per second and saves them to a buffer, preparing for image analysis.

[0499] Step 2:

[0500] The device inputs the acquired video into a generating AI model to recognize road signs in real time. The input in this step is the image data acquired in step 1, and the frames containing this data are passed to the AI ​​model. Models such as YOLO perform feature extraction and classification to identify the location and type of the sign. The output is information about the recognized sign, including its location coordinates and type. Specifically, a rectangle is drawn around the recognized sign, and the type of sign is displayed as text next to it.

[0501] Step 3:

[0502] The device generates voice feedback based on recognized traffic sign information. The input is the traffic sign information, which is the output of step 2, and is converted into voice data using Google Text-to-Speech or Amazon Polly. The output is either a voice file or a real-time voice stream. Specifically, the generated voice is played through the car's audio system, providing the driver with an audible message such as "The speed limit is 80 km / h."

[0503] Step 4:

[0504] The terminal sends acquired driving data (speed, location information, traffic sign recognition results, etc.) to the server. The input is the aggregate of data acquired in steps 1 and 2, which is sent to the server in an encrypted format using the HTTPS protocol. The output is the dataset sent to the server. Specifically, data packets are periodically sent from the user's terminal to the cloud server, and the data storage process on the server side proceeds.

[0505] Step 5:

[0506] The server stores and analyzes the received data. The input is the driving data transmitted in step 4, which is used for statistical processing and machine learning analysis to understand the user's driving patterns. The output is the analysis results, which include driving trends and areas for improvement. Specifically, the processing is executed within a cloud computing environment, and the results are stored in a database.

[0507] Step 6:

[0508] The server generates driving improvement suggestions based on the analysis results and provides feedback to the user's smartphone. The input is the analysis results from step 5, which serve as the basis for generating advice on specific driving behaviors. The output is a notification message displayed in the smartphone application. Specifically, the app displays a pop-up advising the user, such as "Maintaining a consistent speed limit will improve fuel efficiency," prompting them to view it.

[0509] Step 7:

[0510] The server transmits driving data to the insurance company's system, and premium adjustments are made. The input is the driving data and analysis results obtained in steps 4 and 5, and a risk assessment is performed according to the insurance company's standards. The output is the updated insurance contract terms or premium. Specifically, a process proceeds in which a new premium discount based on the user's driving evaluation is calculated.

[0511] (Application Example 1)

[0512] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0513] In recent years, many car accidents have been caused by driver inattention or overlooking road signs. Furthermore, drivers often lack real-time feedback on their driving tendencies, hindering their ability to improve their safe driving habits. Therefore, there is a need to develop a system that improves the accuracy of road sign recognition, provides drivers with real-time and effective feedback, analyzes driving techniques based on individual driving data, and offers guidance for improvement.

[0514] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0515] In this invention, the server includes means for analyzing video acquired from an imaging device mounted on the vehicle, means for generating voice notifications based on recognized signs, and means for accumulating and analyzing driving data. This enables the provision of real-time and effective driving support to the driver, thereby promoting safe driving.

[0516] An "imaging device" is a device installed in a vehicle to acquire video information while it is in motion, and includes devices such as cameras.

[0517] "Means of analysis" refers to methods and devices for processing acquired video information to recognize road signs and analyze driving tendencies.

[0518] "Means for generating voice notifications" refers to a system for providing voice information to drivers based on recognized signs and other information.

[0519] "Driving data" refers to information related to driving conditions, such as speed, location information, and traffic sign recognition results obtained during driving.

[0520] "Means of accumulation and analysis" refers to methods and systems for saving collected driving data and using it to understand and analyze driving trends.

[0521] "Means of providing visual notification" refers to methods of providing visual information to the driver using display devices installed inside the vehicle.

[0522] A "portable information terminal" is a small digital device used to provide drivers with guidance on how to improve their driving, and includes smartphones and similar devices.

[0523] This invention is a system that provides driver assistance within a vehicle. The server acquires video from an imaging device mounted on the vehicle and analyzes this video in real time to recognize road signs. The hardware used includes an in-vehicle camera and edge computing devices, and the software employs machine learning libraries (e.g., TensorFlow, PyTorch).

[0524] The terminal generates voice notifications based on analyzed sign information and provides them to the driver using speech synthesis technology (e.g., Google Text-to-Speech API). Visual notifications are also provided using in-vehicle displays. This allows drivers to obtain important driving information in real time.

[0525] Furthermore, the terminal stores driving data including speed, location information, and recognition results. This data is transferred to a cloud server and analyzed using data analysis tools (e.g., Apache Hadoop). Based on the analysis results, the server identifies individual driving trends and generates improvement guidelines.

[0526] The improvement guidelines will be provided to drivers via portable information terminals, specifically smartphones. This will allow drivers to understand their own driving tendencies and make improvements as needed. Specific examples include situations where the system recognizes speed limit signs while driving on a highway and notifies the driver by voice, "The speed limit is 80 km / h."

[0527] Examples of prompts include: "Please describe in detail how to help design a system that analyzes in-vehicle camera video data in real time and provides drivers with audio and visual notifications regarding speed limits and road signs."

[0528] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0529] Step 1:

[0530] The terminal acquires video from the in-vehicle camera. The video data is passed to the terminal as input, and the terminal prepares this data for processing. Specifically, it converts the video to an appropriate resolution and performs noise reduction to make it suitable for analysis.

[0531] Step 2:

[0532] The device analyzes video in real time and recognizes road signs. The input is previously processed video data, and the device uses an AI model (e.g., a model using TensorFlow or PyTorch) to identify the signs. Data calculations include feature extraction and pattern matching. The output is the recognized sign information.

[0533] Step 3:

[0534] The device generates an audio notification based on recognized traffic sign information. The input is the traffic sign information obtained in step 2, and the device uses a speech synthesis API (e.g., Google Text-to-Speech) to generate an audio notification optimized for the driver. The output is an audio file.

[0535] Step 4:

[0536] The terminal plays audio files to provide information to the user. At this time, the audio is output using the vehicle's speakers. Furthermore, visual information is simultaneously provided using the vehicle's display device. This allows the user to be notified of important points to remember while driving in real time.

[0537] Step 5:

[0538] The terminal collects speed, location information, and traffic sign recognition results during driving, and stores them as driving data. This data is then sent to a cloud server. Measurement data is used as input, which the terminal organizes and converts into a transmittable format. As a result, transmittable data packets are output.

[0539] Step 6:

[0540] The server analyzes the received driving data to understand driving trends. The input is data sent from the terminal, and analysis is performed using data analysis tools (e.g., Apache Hadoop). As a result of the analysis, insights into the user's driving trends are obtained.

[0541] Step 7:

[0542] Based on the analysis results, the server generates driving improvement guidelines for the user. The input is the analysis results of driving trends, and the server uses the data to generate individual suggestions. The generated suggestions are output and notified to the user via a portable information terminal.

[0543] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0544] This invention is a system that combines a camera installed in a vehicle with an emotion engine to assist the user's driving. The emotion engine uses image analysis and voice analysis technologies to recognize the emotional state of the user from the video and audio acquired by the terminal.

[0545] First, the device acquires road footage via an in-car camera and simultaneously captures the user's facial expressions. Using AI, it recognizes road signs and analyzes the user's emotional state. During this process, it determines whether the user is irritated, relaxed, or tired.

[0546] Next, the device adjusts the voice feedback based on the recognized sign information and emotional information. For example, if the user is feeling anxious, it selectively provides encouraging messages such as "Please drive calmly" or confidence-boosting feedback.

[0547] The terminal then transmits this driving and emotional data to a server. The server stores this data and analyzes driving tendencies while considering the emotional state. Based on the analysis results, it generates suggestions for improving driving. These suggestions include personalized advice tailored to the user's emotions, such as "Drive without rushing."

[0548] Users can receive these suggestions and analysis results via their smartphones. Furthermore, emotional data is integrated with other driving data, and in collaboration with insurance companies, driving tendencies based on the user's emotional state are also used to optimize insurance premiums. Higher emotional stability may lead to lower insurance premiums.

[0549] For example, if a user shows signs of frustration while driving, the system will provide specific feedback such as, "Take a deep breath and relax." If the driving remains calm afterward, the server evaluates the result and provides positive feedback through the app. In this way, the present invention provides driving assistance that incorporates the user's emotions, realizing a safer and more secure driving experience.

[0550] The following describes the processing flow.

[0551] Step 1:

[0552] The device uses an in-car camera to simultaneously capture video of the road and the user's face. The road video is used to monitor driving conditions, and the facial video is used for analyzing the user's emotions.

[0553] Step 2:

[0554] The AI ​​installed in the device analyzes video data in real time, recognizing road signs and analyzing the user's emotional state. Using image recognition technology, it determines the type of emotion (e.g., irritation, concentration, relaxation) from the user's facial expressions.

[0555] Step 3:

[0556] The device generates necessary voice feedback for the driver based on recognized road signs, while simultaneously adjusting the feedback content based on emotion analysis results. For example, if the user is feeling stressed, it will generate a gentle and reassuring voice message such as, "Please drive safely."

[0557] Step 4:

[0558] The device plays the generated audio feedback through the car's speakers, providing it to the user. The user can then adjust their driving based on this feedback.

[0559] Step 5:

[0560] The device transmits driving data and emotional data to the server. This data includes speed, location, traffic sign recognition results, and emotional state.

[0561] Step 6:

[0562] The server stores the received data in a database and uses AI to analyze detailed driving trends. Here, emotional fluctuations are also treated as patterns and linked to driving tendencies.

[0563] Step 7:

[0564] The server generates suggestions for improving driving performance, including advice tailored to your emotional state. It offers ways to reduce stress and points to be mindful of.

[0565] Step 8:

[0566] The server sends the generated improvement suggestions to the user's smartphone app. The user can review the advice through the app and consider adjusting their driving to improve their emotional stability.

[0567] Step 9:

[0568] Driving information, including user emotional data, is used in collaboration with insurance companies to adjust insurance premiums. Driving with stable emotions is valued, contributing to the optimization of insurance premiums.

[0569] (Example 2)

[0570] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0571] Modern vehicle driving requires not only the ability to recognize road signs but also the ability to appropriately recognize the driver's emotional state and provide corresponding feedback. However, existing systems are insufficient in supporting drivers while considering their mental state, making it difficult to provide feedback based on emotional state or analyze driving tendencies. Furthermore, the use of data to optimize insurance premiums based on driving skills and emotional state is limited.

[0572] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0573] In this invention, the server includes means for analyzing video acquired from an imaging device installed in the vehicle in real time and recognizing road signs, means for recognizing emotional states using an AI model generated from the acquired video and audio data, and means for accumulating individual driving information and emotional state data and analyzing driving tendencies based on that data. This enables real-time support and safety improvements in response to the driver's emotional state, and also realizes personalized optimization of insurance premiums based on driving characteristics.

[0574] An "imaging device" is a device installed in a vehicle to capture images of the surroundings.

[0575] "Analysis" is the process of examining acquired data in detail and extracting specific information.

[0576] "Road sign recognition" refers to identifying specific shapes or patterns from acquired video data and determining that they are a type of road sign.

[0577] A "generative AI model" is an artificial intelligence algorithm or system that learns from large amounts of data and is used to perform a specific task.

[0578] "Emotional state" refers to the mental and emotional condition of the driver, as inferred from their facial expressions, tone of voice, and other factors.

[0579] "Feedback" refers to the process of providing information to the driver, either verbally or visually, based on acquired information and analysis results, in order to assist them.

[0580] "Driving information" refers to all data related to the operation of a vehicle, including information such as speed, location, and route.

[0581] An "external system" is an external information processing device or network that exists separately from the in-vehicle system and is connected for the purpose of sharing and analyzing data.

[0582] "Insurance premium optimization" means evaluating the driver's characteristics and risks based on acquired data, and calculating an appropriate insurance premium accordingly.

[0583] A "mobile device" is an electronic device that a user can carry with them, and includes smartphones, tablet devices, and other similar devices.

[0584] This invention is a system that combines an imaging device mounted on a vehicle with a generative AI model for recognizing emotional states to provide safer and more personalized driving assistance.

[0585] The server manages the main data processing, while the terminal performs critical real-time processing. The terminal acquires road video and driver audio using imaging and acoustic input devices installed in the vehicle. The video data is analyzed by image analysis software to recognize road signs. Meanwhile, the audio data is processed through acoustic analysis algorithms to identify emotional states. In this step, a generative AI model used to train a specific emotion model plays a crucial role.

[0586] Subsequently, the device generates and provides voice feedback to the user based on the recognized sign information and emotional state. This feedback is customized to take the user's emotional state into account and provide appropriate driving assistance.

[0587] The server stores driving information and emotional state data transmitted from the terminal and performs analysis based on this data. This analysis reveals driving trends and allows for the provision of more detailed improvement suggestions to the user. As a result, improved driver safety and driving performance are expected.

[0588] For example, if a user shows signs of frustration while driving, the device provides appropriate feedback such as, "Take a deep breath and relax." If the driving then proceeds smoothly, the server evaluates this and provides positive feedback to the user.

[0589] An example of a prompt is, "Design a program that analyzes the driver's emotional state from video and audio data acquired by an in-vehicle camera and generates real-time feedback." This prompt suggests the purpose of the invention and its specific implementation.

[0590] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0591] Step 1:

[0592] The terminal acquires road video and driver audio data from the vehicle's mounted imaging and audio input devices. Inputs include real-time video frames and audio signals. This data is recorded in an appropriate format as it is used directly in the next processing step.

[0593] Step 2:

[0594] The device analyzes acquired video data using image analysis software to recognize road signs. Video data is used as input. The image analysis algorithm identifies signs through feature extraction and pattern matching, and outputs the recognized sign information.

[0595] Step 3:

[0596] The device analyzes the input voice data using a generating AI model to recognize the driver's emotional state. Voice data is input, and processes for extracting acoustic features and estimating the emotional state are performed. The output is a label indicating the emotional state.

[0597] Step 4:

[0598] The device generates voice feedback based on recognized road sign information and emotional state. The sign information and emotional state labels are used as input, and a text generation engine generates an appropriate voice message. The output is voice feedback.

[0599] Step 5:

[0600] The device provides the user with generated voice feedback. The generated voice message is then played through the car's speakers. This allows the user to receive real-time instructions and advice.

[0601] Step 6:

[0602] The terminal transmits driving information and emotional state data, which are the results of all processing steps, to the server. The transmitted data is then used for subsequent analysis and evaluation.

[0603] Step 7:

[0604] The server stores the received data and analyzes driving trends. Past driving information and sentiment data are stored as input, and a report on driving trends based on a statistical model is generated as output.

[0605] Step 8:

[0606] Based on the analysis results, the server generates specific driving improvement suggestions for the user. Using the driving trend report as input, it outputs specific improvement points and advice, which are then provided to the user as feedback for the next driving session.

[0607] (Application Example 2)

[0608] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0609] In situations where a driver's emotional fluctuations during operation can affect driving performance, this project aims to improve safety by understanding the driver's emotional state in real time and providing appropriate feedback. Furthermore, it aims to contribute to reducing the burden on drivers by providing a new method for optimizing insurance coverage using emotional information.

[0610] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0611] In this invention, the server includes means for analyzing video acquired from an imaging device installed in the vehicle in real time and recognizing road signals; means for accumulating driving-related data and emotional information and analyzing driving tendencies based on that data; and means for generating improvement suggestions for the driver based on the analyzed driving tendencies and emotional information, and further presenting them visually and audibly. This enables safe driving assistance that takes into account the driver's emotional state and optimization of insurance premiums.

[0612] An "imaging device" is a device attached to a vehicle to acquire image information of the surroundings.

[0613] "Signals" refers to visual information that affects driving, including traffic signs and signals on the road.

[0614] The term "driver" refers to the person operating a vehicle, and their psychological state and driving skills affect the safety of the ride.

[0615] "Emotional information" refers to data representing the psychological state obtained from the pilot's facial expressions and voice.

[0616] "Real-time analysis" refers to a process where information is processed instantly the moment it is acquired, and results are derived immediately.

[0617] "Feedback" refers to information and instructions provided to the pilot based on analysis results, with the role of improving or supporting their actions.

[0618] "Driving-related data" refers to all information related to the operation of the vehicle, including information on speed, location, and driving style.

[0619] "Driving tendencies" refer to information that indicates the driver's driving style and patterns, extracted from accumulated data.

[0620] "Means of visual and auditory presentation" refers to methods of conveying information through displays and speakers in a way that the operator can immediately understand.

[0621] "Insurance premium optimization" refers to a method of calculating insurance premiums rationally by taking into account the driver's driving history and emotional stability.

[0622] The system implementing this invention is a driving assistance system that utilizes emotion recognition technology. This system consists of various devices installed in the vehicle and an external information processing system, and analyzes the driver's emotional state and driving data in real time to support safe driving.

[0623] First, the server uses an imaging device installed in the vehicle to capture images of the driver's facial expressions and the surrounding road environment. This video data is processed in real time using image recognition libraries such as OpenCV to recognize traffic signals and other visual road information. Simultaneously, emotional information is obtained from the driver's facial expressions. Emotion recognition uses machine learning models built with TensorFlow or PyTorch.

[0624] Next, the device generates feedback based on this information. It generates voice messages using Amazon Polly or the Google Text-to-Speech API, and also provides visual feedback to the driver via a display device. This allows the driver to receive appropriate driving assistance in real time, taking into account their emotional state.

[0625] The data is sent to a cloud-based external information processing system, where it is stored and analyzed by a server. During this process, driving trends are analyzed, and a data model for optimizing insurance premiums is generated using machine learning. In this process, a generative AI model is used to construct prompt messages based on past data. An example of a prompt message is, "Generate driving assistance messages that take into account the user's current emotional state. For example, suggest a message to encourage relaxation if the user is irritated."

[0626] To give a specific example, if excellent driving results in a stable emotional state even after long periods of driving, the server will generate and provide positive feedback as a reward to the driver. This encourages the driver to constantly be aware of their own driving and emotional stability, resulting in a safer and more comfortable driving experience.

[0627] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0628] Step 1:

[0629] The server acquires video data of the driver's facial expressions and the surrounding road environment through an imaging device installed in the vehicle. The input is video data from the imaging device, and image analysis is performed based on this data. The software used here is an image processing library such as OpenCV.

[0630] Step 2:

[0631] The server uses the acquired video data to perform image analysis and recognize traffic signals and other visual road information on the road. This process outputs the recognized visual information. A traffic signal recognition algorithm is used for data calculation.

[0632] Step 3:

[0633] The server inputs the pilot's facial expression data into an emotion recognition program, which then analyzes the emotional information. The input is the pilot's facial expression data, and the output is data indicating the emotional state. The calculations are performed using a machine learning model based on TensorFlow.

[0634] Step 4:

[0635] The device generates feedback based on signal recognition results and sentiment information. This feedback generation program uses Amazon Polly or the Google Text-to-Speech API to create audio data. Visual information is displayed on the screen. The input is signal recognition results and sentiment information, and the output is feedback to the operator.

[0636] Step 5:

[0637] The server uses a generated AI model to send piloting data and emotional information to a cloud system, where it is stored in a database. The inputs here are piloting data and emotional information, and the output is stored data on the cloud. Data storage processing is performed by the database management system.

[0638] Step 6:

[0639] The server analyzes accumulated data to determine driving trends. Based on the analysis results, it attempts to optimize insurance premiums. The input consists of driving history data and emotional stability data, and the output is an optimized insurance premium proposal. The analysis is performed using machine learning algorithms.

[0640] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0641] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0642] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0643] [Fourth Embodiment]

[0644] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0645] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0646] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0647] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0648] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0649] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0650] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0651] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0652] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0653] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0654] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0655] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0656] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0657] This invention relates to a system that provides driving assistance using road images acquired by a camera installed in a vehicle. First, a terminal continuously acquires images using the in-vehicle camera and uses AI to recognize road signs in real time from those images. Based on the recognized signs, the terminal generates voice feedback and provides it directly to the user. This voice feedback serves as a notification to encourage attention while driving.

[0658] Furthermore, the device transmits data acquired during driving, namely speed, location information, and traffic sign recognition results, to the server. The server stores this data and analyzes it using AI to understand the user's driving tendencies. Based on the analysis results, the server generates personalized driving improvement suggestions and provides feedback to the user via smartphone.

[0659] For example, when driving on a highway, the device recognizes speed limit signs and provides a voice warning, "The speed limit is 80 km / h." Furthermore, if the server detects frequent speeding based on past data, it provides specific advice to the user via the smartphone app, such as, "Adhering to the speed limit will improve fuel efficiency."

[0660] Furthermore, this driving data is linked with insurance companies, and the system also includes a mechanism to adjust users' insurance premiums based on their driving skills. As a result, users are motivated to drive safely and can also enjoy the benefit of lower insurance premiums.

[0661] As described above, the system of the present invention reduces the risk of accidents and promotes safe driving by providing individually optimized driving support to all drivers, including the elderly and novice drivers.

[0662] The following describes the processing flow.

[0663] Step 1:

[0664] The terminal uses a camera installed in the vehicle to acquire road images in real time. The camera continuously captures the surrounding environment while driving and generates video data.

[0665] Step 2:

[0666] The AI ​​installed in the device analyzes the acquired video data and recognizes road signs within the video. The AI ​​uses an image recognition algorithm to identify speed limit signs, stop signs, and other similar signs.

[0667] Step 3:

[0668] The device generates voice feedback based on recognized signs. Using speech synthesis technology, it creates messages such as "The speed limit is XX km / h" or "Please stop at the next intersection."

[0669] Step 4:

[0670] The device plays the generated audio feedback through the car's speakers, notifying the user. This allows the user to receive signage information in real time.

[0671] Step 5:

[0672] The terminal collects driving data, including speed and location information, and sends it to the server. This data is periodically transferred to the server using a secure communication protocol.

[0673] Step 6:

[0674] The server stores the received driving data in a database and performs AI-based analysis. The analysis extracts the driver's driving patterns and tendencies and generates safe driving indicators.

[0675] Step 7:

[0676] Based on the analysis results, the server generates specific suggestions for improving driving performance. These suggestions are compiled as advice and points of caution that take driving trends into account.

[0677] Step 8:

[0678] The server sends the generated driving improvement suggestions to the user's smartphone app. The user can then review the suggestions through the app and adjust their driving behavior accordingly.

[0679] Step 9:

[0680] If the user accepts the improvement suggestion, new operating data is generated based on the implementation results, and the process returns to step 5 for data transmission and analysis.

[0681] Step 10:

[0682] Driving data is regularly shared with insurance companies, and insurance premiums are optimized based on driving skills. The server also manages this data exchange, contributing to the adjustment of users' insurance premiums.

[0683] (Example 1)

[0684] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0685] In recent years, as safety in vehicle operation has become increasingly important to society, there is a growing demand for appropriate driving assistance and cost-effective driving instruction tailored to individual drivers. However, conventional technologies have had difficulties in recognizing road markings in real time to support driving, or in providing effective improvement suggestions based on drivers' driving tendencies. Furthermore, there has been a lack of mechanisms to effectively utilize individual driving data to optimize insurance premiums. Technologies that can solve these problems are needed.

[0686] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0687] In this invention, the server includes a processing unit that processes moving images obtained from a video acquisition device mounted on the vehicle in real time and identifies road markings; an output unit that generates audio output information based on the identified markings and notifies the driver; and an analysis unit that collects individual driving data and analyzes driving trends based on the data. As a result, the driver can recognize sign information in real time and receive appropriate warnings according to the driving situation. Furthermore, it is possible to promote safe driving and reduce the financial burden through driving improvement suggestions and insurance premium optimization based on the collected data.

[0688] A "vehicle-mounted video acquisition device" is a device installed inside or outside a vehicle to capture images of road conditions and the surrounding environment while driving.

[0689] "Real-time processing of video" means analyzing captured video data instantly without delay and extracting the necessary information.

[0690] "Identifying road markings" means identifying signs and traffic rule indicators installed on roads from acquired video footage, and clarifying their type and content.

[0691] "Generating voice output information" means creating data to provide drivers with voice notifications and guidance based on the analysis results.

[0692] "Notifying the driver" means communicating information or warnings to the vehicle's driver through audio or visual means.

[0693] "Collecting driving data" means recording and accumulating various data related to the operation of a vehicle (e.g., speed, position, acceleration).

[0694] "Analyzing driving trends" means analyzing accumulated driving data to understand and evaluate the driver's driving habits and characteristics.

[0695] "Adjusting insurance premiums" means evaluating the risk of individual drivers based on collected driving data and appropriately calculating insurance premiums in insurance contracts.

[0696] The embodiments for carrying out the present invention are described below.

[0697] The system of the present invention acquires road images in real time from a video acquisition device mounted on a vehicle to provide driving assistance. The terminal acquires road images in high resolution through a camera mounted on the vehicle. These acquired images are analyzed in real time using a generative AI model. Here, widely used models such as the YOLO (You Only Look Once) series and OpenCV's DNN module are used. These models have the ability to identify road signs quickly and with high accuracy.

[0698] The device generates audio output information based on identified road signs. This process utilizes speech synthesis engines such as Google Text-to-Speech and Amazon Polly to convert visual information into audio. This allows drivers to receive road sign information in real time via audio, enabling them to concentrate on driving without taking their eyes off the road.

[0699] Furthermore, the device transmits speed, location information, and other driving data collected during driving to a server. The server stores this data in the cloud and is responsible for analyzing driving trends. By using AI analysis tools such as Amazon SageMaker, it is possible to understand the driver's driving habits and generate appropriate improvement suggestions.

[0700] The server generates specific improvement suggestions for the driver based on the analysis results and provides feedback to the driver via a portable information terminal such as a smartphone. The suggestions should be specific and actionable, and may include visual infographics and text explanations to aid the driver's understanding.

[0701] A further feature of this system is that the collected driving data is linked with the insurance company's system, and insurance premiums are calculated based on the driver's driving skills. This motivates drivers to drive safely while also providing economic benefits such as reduced insurance premiums.

[0702] To give specific examples, while driving on a highway, the device might send a voice notification saying, "The speed limit is 80 km / h," or a server might display advice on the smartphone saying, "By being mindful of the speed limit, you can improve fuel efficiency."

[0703] An example of a prompt to input into the generating AI model is, "Please describe the procedure for analyzing camera footage, recognizing road signs, and generating an audio notification."

[0704] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0705] Step 1:

[0706] The terminal continuously acquires road images using a camera mounted on the vehicle. The input is raw image data from the camera, and this data is temporarily stored in high resolution. Specifically, the camera captures several frames of images per second and saves them to a buffer, preparing for image analysis.

[0707] Step 2:

[0708] The device inputs the acquired video into a generating AI model to recognize road signs in real time. The input in this step is the image data acquired in step 1, and the frames containing this data are passed to the AI ​​model. Models such as YOLO perform feature extraction and classification to identify the location and type of the sign. The output is information about the recognized sign, including its location coordinates and type. Specifically, a rectangle is drawn around the recognized sign, and the type of sign is displayed as text next to it.

[0709] Step 3:

[0710] The device generates voice feedback based on recognized traffic sign information. The input is the traffic sign information, which is the output of step 2, and is converted into voice data using Google Text-to-Speech or Amazon Polly. The output is either a voice file or a real-time voice stream. Specifically, the generated voice is played through the car's audio system, providing the driver with an audible message such as "The speed limit is 80 km / h."

[0711] Step 4:

[0712] The terminal sends acquired driving data (speed, location information, traffic sign recognition results, etc.) to the server. The input is the aggregate of data acquired in steps 1 and 2, which is sent to the server in an encrypted format using the HTTPS protocol. The output is the dataset sent to the server. Specifically, data packets are periodically sent from the user's terminal to the cloud server, and the data storage process on the server side proceeds.

[0713] Step 5:

[0714] The server stores and analyzes the received data. The input is the driving data transmitted in step 4, which is used for statistical processing and machine learning analysis to understand the user's driving patterns. The output is the analysis results, which include driving trends and areas for improvement. Specifically, the processing is executed within a cloud computing environment, and the results are stored in a database.

[0715] Step 6:

[0716] The server generates driving improvement suggestions based on the analysis results and provides feedback to the user's smartphone. The input is the analysis results from step 5, which serve as the basis for generating advice on specific driving behaviors. The output is a notification message displayed in the smartphone application. Specifically, the app displays a pop-up advising the user, such as "Maintaining a consistent speed limit will improve fuel efficiency," prompting them to view it.

[0717] Step 7:

[0718] The server transmits driving data to the insurance company's system, and premium adjustments are made. The input is the driving data and analysis results obtained in steps 4 and 5, and a risk assessment is performed according to the insurance company's standards. The output is the updated insurance contract terms or premium. Specifically, a process proceeds in which a new premium discount based on the user's driving evaluation is calculated.

[0719] (Application Example 1)

[0720] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0721] In recent years, many car accidents have been caused by driver inattention or overlooking road signs. Furthermore, drivers often lack real-time feedback on their driving tendencies, hindering their ability to improve their safe driving habits. Therefore, there is a need to develop a system that improves the accuracy of road sign recognition, provides drivers with real-time and effective feedback, analyzes driving techniques based on individual driving data, and offers guidance for improvement.

[0722] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0723] In this invention, the server includes means for analyzing video acquired from an imaging device mounted on the vehicle, means for generating voice notifications based on recognized signs, and means for accumulating and analyzing driving data. This enables the provision of real-time and effective driving support to the driver, thereby promoting safe driving.

[0724] An "imaging device" is a device installed in a vehicle to acquire video information while it is in motion, and includes devices such as cameras.

[0725] "Means of analysis" refers to methods and devices for processing acquired video information to recognize road signs and analyze driving tendencies.

[0726] "Means for generating voice notifications" refers to a system for providing voice information to drivers based on recognized signs and other information.

[0727] "Driving data" refers to information related to driving conditions, such as speed, location information, and traffic sign recognition results obtained during driving.

[0728] "Means of accumulation and analysis" refers to methods and systems for saving collected driving data and using it to understand and analyze driving trends.

[0729] "Means of providing visual notification" refers to methods of providing visual information to the driver using display devices installed inside the vehicle.

[0730] A "portable information terminal" is a small digital device used to provide drivers with guidance on how to improve their driving, and includes smartphones and similar devices.

[0731] This invention is a system that provides driver assistance within a vehicle. The server acquires video from an imaging device mounted on the vehicle and analyzes this video in real time to recognize road signs. The hardware used includes an in-vehicle camera and edge computing devices, and the software employs machine learning libraries (e.g., TensorFlow, PyTorch).

[0732] The terminal generates voice notifications based on analyzed sign information and provides them to the driver using speech synthesis technology (e.g., Google Text-to-Speech API). Visual notifications are also provided using in-vehicle displays. This allows drivers to obtain important driving information in real time.

[0733] Furthermore, the terminal stores driving data including speed, location information, and recognition results. This data is transferred to a cloud server and analyzed using data analysis tools (e.g., Apache Hadoop). Based on the analysis results, the server identifies individual driving trends and generates improvement guidelines.

[0734] The improvement guidelines will be provided to drivers via portable information terminals, specifically smartphones. This will allow drivers to understand their own driving tendencies and make improvements as needed. Specific examples include situations where the system recognizes speed limit signs while driving on a highway and notifies the driver by voice, "The speed limit is 80 km / h."

[0735] Examples of prompts include: "Please describe in detail how to help design a system that analyzes in-vehicle camera video data in real time and provides drivers with audio and visual notifications regarding speed limits and road signs."

[0736] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0737] Step 1:

[0738] The terminal acquires video from the in-vehicle camera. The video data is passed to the terminal as input, and the terminal prepares this data for processing. Specifically, it converts the video to an appropriate resolution and performs noise reduction to make it suitable for analysis.

[0739] Step 2:

[0740] The device analyzes video in real time and recognizes road signs. The input is previously processed video data, and the device uses an AI model (e.g., a model using TensorFlow or PyTorch) to identify the signs. Data calculations include feature extraction and pattern matching. The output is the recognized sign information.

[0741] Step 3:

[0742] The device generates an audio notification based on recognized traffic sign information. The input is the traffic sign information obtained in step 2, and the device uses a speech synthesis API (e.g., Google Text-to-Speech) to generate an audio notification optimized for the driver. The output is an audio file.

[0743] Step 4:

[0744] The terminal plays audio files to provide information to the user. At this time, the audio is output using the vehicle's speakers. Furthermore, visual information is simultaneously provided using the vehicle's display device. This allows the user to be notified of important points to remember while driving in real time.

[0745] Step 5:

[0746] The terminal collects speed, location information, and traffic sign recognition results during driving, and stores them as driving data. This data is then sent to a cloud server. Measurement data is used as input, which the terminal organizes and converts into a transmittable format. As a result, transmittable data packets are output.

[0747] Step 6:

[0748] The server analyzes the received driving data to understand driving trends. The input is data sent from the terminal, and analysis is performed using data analysis tools (e.g., Apache Hadoop). As a result of the analysis, insights into the user's driving trends are obtained.

[0749] Step 7:

[0750] Based on the analysis results, the server generates driving improvement guidelines for the user. The input is the analysis results of driving trends, and the server uses the data to generate individual suggestions. The generated suggestions are output and notified to the user via a portable information terminal.

[0751] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0752] This invention is a system that combines a camera installed in a vehicle with an emotion engine to assist the user's driving. The emotion engine uses image analysis and voice analysis technologies to recognize the emotional state of the user from the video and audio acquired by the terminal.

[0753] First, the device acquires road footage via an in-car camera and simultaneously captures the user's facial expressions. Using AI, it recognizes road signs and analyzes the user's emotional state. During this process, it determines whether the user is irritated, relaxed, or tired.

[0754] Next, the device adjusts the voice feedback based on the recognized sign information and emotional information. For example, if the user is feeling anxious, it selectively provides encouraging messages such as "Please drive calmly" or confidence-boosting feedback.

[0755] The terminal then transmits this driving and emotional data to a server. The server stores this data and analyzes driving tendencies while considering the emotional state. Based on the analysis results, it generates suggestions for improving driving. These suggestions include personalized advice tailored to the user's emotions, such as "Drive without rushing."

[0756] Users can receive these suggestions and analysis results via their smartphones. Furthermore, emotional data is integrated with other driving data, and in collaboration with insurance companies, driving tendencies based on the user's emotional state are also used to optimize insurance premiums. Higher emotional stability may lead to lower insurance premiums.

[0757] For example, if a user shows signs of frustration while driving, the system will provide specific feedback such as, "Take a deep breath and relax." If the driving remains calm afterward, the server evaluates the result and provides positive feedback through the app. In this way, the present invention provides driving assistance that incorporates the user's emotions, realizing a safer and more secure driving experience.

[0758] The following describes the processing flow.

[0759] Step 1:

[0760] The device uses an in-car camera to simultaneously capture video of the road and the user's face. The road video is used to monitor driving conditions, and the facial video is used for analyzing the user's emotions.

[0761] Step 2:

[0762] The AI ​​installed in the device analyzes video data in real time, recognizing road signs and analyzing the user's emotional state. Using image recognition technology, it determines the type of emotion (e.g., irritation, concentration, relaxation) from the user's facial expressions.

[0763] Step 3:

[0764] The device generates necessary voice feedback for the driver based on recognized road signs, while simultaneously adjusting the feedback content based on emotion analysis results. For example, if the user is feeling stressed, it will generate a gentle and reassuring voice message such as, "Please drive safely."

[0765] Step 4:

[0766] The device plays the generated audio feedback through the car's speakers, providing it to the user. The user can then adjust their driving based on this feedback.

[0767] Step 5:

[0768] The device transmits driving data and emotional data to the server. This data includes speed, location, traffic sign recognition results, and emotional state.

[0769] Step 6:

[0770] The server stores the received data in a database and uses AI to analyze detailed driving trends. Here, emotional fluctuations are also treated as patterns and linked to driving tendencies.

[0771] Step 7:

[0772] The server generates suggestions for improving driving performance, including advice tailored to your emotional state. It offers ways to reduce stress and points to be mindful of.

[0773] Step 8:

[0774] The server sends the generated improvement suggestions to the user's smartphone app. The user can review the advice through the app and consider adjusting their driving to improve their emotional stability.

[0775] Step 9:

[0776] Driving information, including user emotional data, is used in collaboration with insurance companies to adjust insurance premiums. Driving with stable emotions is valued, contributing to the optimization of insurance premiums.

[0777] (Example 2)

[0778] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0779] Modern vehicle driving requires not only the ability to recognize road signs but also the ability to appropriately recognize the driver's emotional state and provide corresponding feedback. However, existing systems are insufficient in supporting drivers while considering their mental state, making it difficult to provide feedback based on emotional state or analyze driving tendencies. Furthermore, the use of data to optimize insurance premiums based on driving skills and emotional state is limited.

[0780] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0781] In this invention, the server includes means for analyzing video acquired from an imaging device installed in the vehicle in real time and recognizing road signs, means for recognizing emotional states using an AI model generated from the acquired video and audio data, and means for accumulating individual driving information and emotional state data and analyzing driving tendencies based on that data. This enables real-time support and safety improvements in response to the driver's emotional state, and also realizes personalized optimization of insurance premiums based on driving characteristics.

[0782] An "imaging device" is a device installed in a vehicle to capture images of the surroundings.

[0783] "Analysis" is the process of examining acquired data in detail and extracting specific information.

[0784] "Road sign recognition" refers to identifying specific shapes or patterns from acquired video data and determining that they are a type of road sign.

[0785] A "generative AI model" is an artificial intelligence algorithm or system that learns from large amounts of data and is used to perform a specific task.

[0786] "Emotional state" refers to the mental and emotional condition of the driver, as inferred from their facial expressions, tone of voice, and other factors.

[0787] "Feedback" refers to the process of providing information to the driver, either verbally or visually, based on acquired information and analysis results, in order to assist them.

[0788] "Driving information" refers to all data related to the operation of a vehicle, including information such as speed, location, and route.

[0789] An "external system" is an external information processing device or network that exists separately from the in-vehicle system and is connected for the purpose of sharing and analyzing data.

[0790] "Insurance premium optimization" means evaluating the driver's characteristics and risks based on acquired data, and calculating an appropriate insurance premium accordingly.

[0791] A "mobile device" is an electronic device that a user can carry with them, and includes smartphones, tablet devices, and other similar devices.

[0792] This invention is a system that combines an imaging device mounted on a vehicle with a generative AI model for recognizing emotional states to provide safer and more personalized driving assistance.

[0793] The server manages the main data processing, while the terminal performs critical real-time processing. The terminal acquires road video and driver audio using imaging and acoustic input devices installed in the vehicle. The video data is analyzed by image analysis software to recognize road signs. Meanwhile, the audio data is processed through acoustic analysis algorithms to identify emotional states. In this step, a generative AI model used to train a specific emotion model plays a crucial role.

[0794] Subsequently, the device generates and provides voice feedback to the user based on the recognized sign information and emotional state. This feedback is customized to take the user's emotional state into account and provide appropriate driving assistance.

[0795] The server stores driving information and emotional state data transmitted from the terminal and performs analysis based on this data. This analysis reveals driving trends and allows for the provision of more detailed improvement suggestions to the user. As a result, improved driver safety and driving performance are expected.

[0796] For example, if a user shows signs of frustration while driving, the device provides appropriate feedback such as, "Take a deep breath and relax." If the driving then proceeds smoothly, the server evaluates this and provides positive feedback to the user.

[0797] An example of a prompt is, "Design a program that analyzes the driver's emotional state from video and audio data acquired by an in-vehicle camera and generates real-time feedback." This prompt suggests the purpose of the invention and its specific implementation.

[0798] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0799] Step 1:

[0800] The terminal acquires road video and driver audio data from the vehicle's mounted imaging and audio input devices. Inputs include real-time video frames and audio signals. This data is recorded in an appropriate format as it is used directly in the next processing step.

[0801] Step 2:

[0802] The device analyzes acquired video data using image analysis software to recognize road signs. Video data is used as input. The image analysis algorithm identifies signs through feature extraction and pattern matching, and outputs the recognized sign information.

[0803] Step 3:

[0804] The device analyzes the input voice data using a generating AI model to recognize the driver's emotional state. Voice data is input, and processes for extracting acoustic features and estimating the emotional state are performed. The output is a label indicating the emotional state.

[0805] Step 4:

[0806] The device generates voice feedback based on recognized road sign information and emotional state. The sign information and emotional state labels are used as input, and a text generation engine generates an appropriate voice message. The output is voice feedback.

[0807] Step 5:

[0808] The device provides the user with generated voice feedback. The generated voice message is then played through the car's speakers. This allows the user to receive real-time instructions and advice.

[0809] Step 6:

[0810] The terminal transmits driving information and emotional state data, which are the results of all processing steps, to the server. The transmitted data is then used for subsequent analysis and evaluation.

[0811] Step 7:

[0812] The server stores the received data and analyzes driving trends. Past driving information and sentiment data are stored as input, and a report on driving trends based on a statistical model is generated as output.

[0813] Step 8:

[0814] Based on the analysis results, the server generates specific driving improvement suggestions for the user. Using the driving trend report as input, it outputs specific improvement points and advice, which are then provided to the user as feedback for the next driving session.

[0815] (Application Example 2)

[0816] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0817] In situations where a driver's emotional fluctuations during operation can affect driving performance, this project aims to improve safety by understanding the driver's emotional state in real time and providing appropriate feedback. Furthermore, it aims to contribute to reducing the burden on drivers by providing a new method for optimizing insurance coverage using emotional information.

[0818] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0819] In this invention, the server includes means for analyzing video acquired from an imaging device installed in the vehicle in real time and recognizing road signals; means for accumulating driving-related data and emotional information and analyzing driving tendencies based on that data; and means for generating improvement suggestions for the driver based on the analyzed driving tendencies and emotional information, and further presenting them visually and audibly. This enables safe driving assistance that takes into account the driver's emotional state and optimization of insurance premiums.

[0820] An "imaging device" is a device attached to a vehicle to acquire image information of the surroundings.

[0821] "Signals" refers to visual information that affects driving, including traffic signs and signals on the road.

[0822] The term "driver" refers to the person operating a vehicle, and their psychological state and driving skills affect the safety of the ride.

[0823] "Emotional information" refers to data representing the psychological state obtained from the pilot's facial expressions and voice.

[0824] "Real-time analysis" refers to a process where information is processed instantly the moment it is acquired, and results are derived immediately.

[0825] "Feedback" refers to information and instructions provided to the pilot based on analysis results, with the role of improving or supporting their actions.

[0826] "Driving-related data" refers to all information related to the operation of the vehicle, including information on speed, location, and driving style.

[0827] "Driving tendencies" refer to information that indicates the driver's driving style and patterns, extracted from accumulated data.

[0828] "Means of visual and auditory presentation" refers to methods of conveying information through displays and speakers in a way that the operator can immediately understand.

[0829] "Insurance premium optimization" refers to a method of calculating insurance premiums rationally by taking into account the driver's driving history and emotional stability.

[0830] The system implementing this invention is a driving assistance system that utilizes emotion recognition technology. This system consists of various devices installed in the vehicle and an external information processing system, and analyzes the driver's emotional state and driving data in real time to support safe driving.

[0831] First, the server uses an imaging device installed in the vehicle to capture images of the driver's facial expressions and the surrounding road environment. This video data is processed in real time using image recognition libraries such as OpenCV to recognize traffic signals and other visual road information. Simultaneously, emotional information is obtained from the driver's facial expressions. Emotion recognition uses machine learning models built with TensorFlow or PyTorch.

[0832] Next, the device generates feedback based on this information. It generates voice messages using Amazon Polly or the Google Text-to-Speech API, and also provides visual feedback to the driver via a display device. This allows the driver to receive appropriate driving assistance in real time, taking into account their emotional state.

[0833] The data is sent to a cloud-based external information processing system, where it is stored and analyzed by a server. During this process, driving trends are analyzed, and a data model for optimizing insurance premiums is generated using machine learning. In this process, a generative AI model is used to construct prompt messages based on past data. An example of a prompt message is, "Generate driving assistance messages that take into account the user's current emotional state. For example, suggest a message to encourage relaxation if the user is irritated."

[0834] To give a specific example, if excellent driving results in a stable emotional state even after long periods of driving, the server will generate and provide positive feedback as a reward to the driver. This encourages the driver to constantly be aware of their own driving and emotional stability, resulting in a safer and more comfortable driving experience.

[0835] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0836] Step 1:

[0837] The server acquires video data of the driver's facial expressions and the surrounding road environment through an imaging device installed in the vehicle. The input is video data from the imaging device, and image analysis is performed based on this data. The software used here is an image processing library such as OpenCV.

[0838] Step 2:

[0839] The server uses the acquired video data to perform image analysis and recognize traffic signals and other visual road information on the road. This process outputs the recognized visual information. A traffic signal recognition algorithm is used for data calculation.

[0840] Step 3:

[0841] The server inputs the pilot's facial expression data into an emotion recognition program, which then analyzes the emotional information. The input is the pilot's facial expression data, and the output is data indicating the emotional state. The calculations are performed using a machine learning model based on TensorFlow.

[0842] Step 4:

[0843] The device generates feedback based on signal recognition results and sentiment information. This feedback generation program uses Amazon Polly or the Google Text-to-Speech API to create audio data. Visual information is displayed on the screen. The input is signal recognition results and sentiment information, and the output is feedback to the operator.

[0844] Step 5:

[0845] The server uses a generated AI model to send piloting data and emotional information to a cloud system, where it is stored in a database. The inputs here are piloting data and emotional information, and the output is stored data on the cloud. Data storage processing is performed by the database management system.

[0846] Step 6:

[0847] The server analyzes accumulated data to determine driving trends. Based on the analysis results, it attempts to optimize insurance premiums. The input consists of driving history data and emotional stability data, and the output is an optimized insurance premium proposal. The analysis is performed using machine learning algorithms.

[0848] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0849] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0850] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0851] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0852] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. In the upper and lower directions of the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. Also, the upper side of the concentric circles is where "pleasant" emotions are located, and the lower side is where "unpleasant" emotions are located. In this way, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0853] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0854] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0855] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0856] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0857] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0858] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0859] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0860] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0861] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0862] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0863] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0864] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0865] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0866] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0867] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0868] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0869] The following is further disclosed regarding the embodiments described above.

[0870] (Claim 1)

[0871] A means of recognizing road signs by analyzing video footage acquired from cameras installed on vehicles in real time,

[0872] A means for generating and providing audio feedback to the driver based on recognized signs,

[0873] A means of accumulating individual driving data and analyzing driving trends based on that data,

[0874] A means for generating improvement suggestions for drivers based on analyzed driving trends,

[0875] A means of optimizing insurance premiums based on driving skills by linking driving data with an external system,

[0876] A system that includes this.

[0877] (Claim 2)

[0878] The system according to claim 1, further comprising means for acquiring speed information and location information while driving, and for evaluating the driver's driving behavior using this information.

[0879] (Claim 3)

[0880] The system according to claim 1, comprising means for providing improvement suggestions to the driver via a smartphone.

[0881] "Example 1"

[0882] (Claim 1)

[0883] A processing device that processes moving images obtained from a video acquisition device mounted on a vehicle in real time and identifies road markings,

[0884] An output device that generates audio output information based on identified markings and notifies the operator,

[0885] An analysis device that collects individual driving data and analyzes driving trends based on said data,

[0886] A generation device that creates improvement suggestions for the driver based on the analyzed driving trends,

[0887] A device that adjusts insurance premiums according to driving skill by linking driving data with an external mechanism,

[0888] A system that includes this.

[0889] (Claim 2)

[0890] The system according to claim 1, further comprising a device that collects speed information and position information during operation and uses this information to evaluate the operator's driving actions.

[0891] (Claim 3)

[0892] The system according to claim 1, comprising a device that provides improvement suggestions to the operator via a portable information terminal.

[0893] "Application Example 1"

[0894] (Claim 1)

[0895] A means of recognizing road signs by analyzing images acquired in real time from an imaging device mounted on a vehicle,

[0896] A means for generating and providing voice notifications to the operator based on recognized signs,

[0897] A means for accumulating individual flight data and analyzing flight tendencies based on that data,

[0898] A means for generating improvement guidelines for the pilot based on the analyzed piloting tendencies,

[0899] A means of optimizing insurance premiums based on piloting skills by linking piloting data with an external device,

[0900] A means of providing visual notification via a display device installed inside the vehicle,

[0901] A system that includes this.

[0902] (Claim 2)

[0903] The system according to claim 1, further comprising means for acquiring speed information and position information during operation and evaluating the operator's operation using this information, and means for providing feedback using voice and a display device.

[0904] (Claim 3)

[0905] The system according to claim 1, comprising means for providing improvement guidelines to the operator via a portable information terminal.

[0906] "Example 2 of combining an emotion engine"

[0907] (Claim 1)

[0908] A means of recognizing road signs by analyzing video acquired from an imaging device installed in a vehicle in real time,

[0909] A means of recognizing emotional states using an AI model generated from acquired video and audio data,

[0910] A means for generating and providing voice feedback to the user based on recognized signs and emotional states,

[0911] A means for accumulating individual driving information and emotional state data, and for analyzing driving trends based on that data,

[0912] A means for generating improvement suggestions for the user based on analyzed driving tendencies and emotional state,

[0913] A means of optimizing insurance premiums based on driving skills and emotional state by linking driving information with an external system,

[0914] A system that includes this.

[0915] (Claim 2)

[0916] The system according to claim 1, further comprising means for acquiring speed information and location information while driving, and for evaluating the user's driving behavior using this information.

[0917] (Claim 3)

[0918] The system according to claim 1, comprising means for providing improvement suggestions to the user via a mobile device.

[0919] "Application example 2 when combining with an emotional engine"

[0920] (Claim 1)

[0921] A means of recognizing road signals by analyzing video acquired from an imaging device installed in a vehicle in real time,

[0922] A means for generating and providing voice feedback to the operator based on recognized signals,

[0923] A means for accumulating individual piloting-related data and emotional information, and for analyzing piloting tendencies based on that data,

[0924] Based on the analyzed piloting tendencies and emotional information, a means is provided to generate improvement suggestions for the pilot and to present them visually and audibly.

[0925] A means of optimizing insurance premiums based on piloting skills and emotional stability by linking piloting-related data and emotional information with an external information processing system,

[0926] A system that includes this.

[0927] (Claim 2)

[0928] The system according to claim 1, further comprising means for acquiring speed information and position information during operation and using this information to evaluate the operator's operation behavior, and a display device for providing real-time feedback according to the operator's emotional state.

[0929] (Claim 3)

[0930] The system according to claim 1, comprising means for providing improvement suggestions to the operator via a portable information terminal. [Explanation of Symbols]

[0931] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of recognizing road signs by analyzing video acquired from an imaging device mounted on a vehicle in real time, A means for generating and providing voice notifications to the operator based on recognized signs, A means for accumulating individual flight data and analyzing flight tendencies based on that data, A means for generating improvement guidelines for the pilot based on the analyzed piloting tendencies, A means of optimizing insurance premiums based on piloting skills by linking piloting data with an external device, A means of providing visual notification via a display device installed inside the vehicle, A system that includes this.

2. The system according to claim 1, further comprising means for acquiring speed information and position information during operation and evaluating the operator's operation using this information, and means for providing feedback using voice and a display device.

3. The system according to claim 1, comprising means for providing improvement guidelines to the operator via a portable information terminal.

Images