system

Abstract

We provide the system. [Solution] A means of acquiring driver's driving status data, A means of analyzing acquired driving condition data to calculate the accident risk for individual drivers, A means for generating a customized virtual reality accident experience for a specific driver, A means to enable drivers to participate in virtual reality accident experiences, A means of recording data based on a virtual reality accident experience in which the driver participated, A means of visualizing the driver's progress by analyzing recorded data, A system that includes means of presenting challenges for safe driving based on the driver's progress.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Conventional uniform driving safety education methods cannot take into account the characteristics and weaknesses of individual drivers, and as a result, it has been difficult to effectively improve the specific risk factors of drivers. Furthermore, since such education methods cannot be linked with the actual driving state data of drivers, there is a problem that it is difficult to contribute to the maintenance and improvement of long-term safety awareness and driving skills.

Means for Solving the Problems

[0005] This invention provides a system that acquires and analyzes driver driving state data and generates a customized virtual reality accident experience for each individual driver based on the analysis results. This system has the function of allowing the driver to participate in the virtual reality accident experience and record data based on that experience. Furthermore, by presenting challenges for safe driving based on the progress visualized through the analysis of the recorded data, it promotes the continuous improvement of the driver's skills.

[0006] "Driving status data" refers to data that indicates the driver's driving characteristics and behavior, and includes speed, braking, vehicle position, driving time, and tendencies toward distraction.

[0007] "Analysis" is the process of using acquired driving data to evaluate driver behavior patterns and accident risk.

[0008] "Accident risk" is an indicator that shows the likelihood that a driver's driving patterns will lead to various accidents.

[0009] A "customized virtual reality accident experience" refers to an experience in a virtual environment specifically designed based on the driver's characteristics and accident risk, simulating specific accident scenarios.

[0010] "Recording" refers to the act of saving data from when a driver participates in a virtual reality accident simulation. This data includes information about the driver's reactions and actions.

[0011] "Visualization" is a technique that clearly displays analyzed data using diagrams and graphs, making it easier to understand the driver's progress and areas for improvement.

[0012] The "Safe Driving Challenge" is a set of tasks designed to improve drivers' driving skills, and is presented through game-like formats and simulations. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a tagged processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, a tagged RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, a tagged storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0019] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] This invention is implemented as a customized educational tool to improve drivers' safe driving. This system utilizes driver data and, based on that data, generates an AI that provides an optimized virtual reality (VR) accident experience.

[0035] The server first acquires driver data from relevant equipment and then analyzes this data. The purpose of the analysis is to identify the driver's behavior patterns and assess accident risk based on them. This includes factors such as speed, frequency of sudden braking, and driving time.

[0036] Based on the information obtained through analysis, the server uses a generative AI to design a VR accident experience optimized for each driver's characteristics and tendencies. This VR experience can address specific risks that drivers may face, such as a simulation that reproduces a situation where a rear-end collision occurs due to sudden braking.

[0037] Next, the terminal receives customized VR data sent from the server, allowing the driver to participate in these simulations through a VR headset. The driver then experiences the accident scenarios firsthand through the VR experience, and their reactions and actions are recorded.

[0038] After the experience, the device presents the driver with safe driving challenges. These challenges, for example, require the driver to safely complete a specific driving task within a set time limit. In this way, the driver's skills can be improved in a game-like format.

[0039] The server aggregates these records and generates a dashboard that visualizes the progress of each driver. This dashboard is provided to drivers and administrators, offering feedback to help improve drivers' safe driving skills. For example, if the dashboard shows a decrease in the frequency of sudden braking, drivers can see the improvement in that area.

[0040] Thus, the system of the present invention provides individual drivers with a customized risk experience and promotes the establishment of long-term safe driving habits through specific feedback.

[0041] The following describes the processing flow.

[0042] Step 1:

[0043] The server acquires the driver's driving data. This is the process of receiving data transmitted from devices such as in-vehicle systems and dashcams, and storing that information in a database.

[0044] Step 2:

[0045] The server analyzes the acquired driving data using an AI algorithm. The analysis is performed to identify the characteristics of the driver's driving pattern and to evaluate risk factors such as collisions and sudden braking.

[0046] Step 3:

[0047] The server generates accident scenarios optimized for each individual driver based on the analysis results. The generating AI designs accident scenarios tailored to the driver's characteristics and converts them into a virtual reality format.

[0048] Step 4:

[0049] The terminal receives VR data sent from the server and prepares it for the driver to experience using a VR headset. This preparation includes initializing the VR environment and synchronizing with the device.

[0050] Step 5:

[0051] Users participate in a VR accident simulation through their device. They experience driving scenarios in virtual reality and get a feel for the risks involved in actual driving.

[0052] Step 6:

[0053] The device records user interaction data and reactions during the VR experience. This data forms the basis for analyzing user behavior and providing feedback.

[0054] Step 7:

[0055] The server analyzes data from the VR experience and generates a dashboard that visualizes the driver's progress and improvements. This dashboard shows improvements and achievements to both users and administrators.

[0056] Step 8:

[0057] The device presents users with appropriate safe driving challenges and provides tasks to further hone their skills. This allows users to continuously work towards increasing their safety awareness.

[0058] (Example 1)

[0059] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0060] While modern drivers recognize the importance of safe driving, they are often unable to effectively manage accident risks in their daily driving activities. Traditional training methods struggle to provide personalized instruction based on each driver's behavior and tendencies, and therefore fail to adequately improve practical safe driving skills. This invention aims to solve these problems by providing a training system that allows for detailed evaluation of each driver's risk profile and practice of countermeasures in a virtual environment.

[0061] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0062] In this invention, the server includes means for monitoring the driver's driving state and collecting data, means for analyzing the collected driving data and evaluating the accident risk of individual drivers, and means for using a generative AI model to generate a virtual reality environment adapted according to the driver's characteristics. This makes it possible to provide each driver with a risk experience tailored to their individual needs and subsequent specific guidance and feedback within the virtual reality environment.

[0063] "Driving conditions" refers to information about the driver's actions and the vehicle's movements while operating the vehicle, and includes speed, acceleration, braking, steering, etc.

[0064] "Data analysis" refers to the process of processing diverse collected data using statistical methods and machine learning to evaluate driver behavior patterns and risks.

[0065] "Accident risk" refers to the degree of likelihood of a vehicle accident predicted based on specific driving behaviors and environmental conditions.

[0066] A "virtual reality environment" refers to a simulated three-dimensional environment that users can experience with a sense of realism using computer graphics and specialized equipment.

[0067] "Generative AI models" refer to artificial intelligence algorithms used to generate outputs tailored to specific purposes, such as generative adversarial networks (GANs) and transformer models.

[0068] "Recording" refers to the process of saving data such as the driver's actions and reactions within the virtual reality environment in a format that allows for later analysis and evaluation.

[0069] "Progress" refers to information about specific improvements and changes in proficiency that a driver has achieved through past driving experiences and training.

[0070] "Visualization" refers to the process of displaying analyzed data and information in visual formats such as graphs and charts to make them easier to understand.

[0071] A "challenge" refers to a specific driving task or challenge set for a driver in order to improve their safe driving skills.

[0072] The embodiment of the invention aims to provide drivers with an innovative educational system to improve their safe driving skills. Specifically, it is designed so that the server, terminals, and users function according to their respective roles.

[0073] The server collects driver data in real time from sensors and positioning systems installed in the vehicle. The collected data includes speed, acceleration, braking frequency, and driving time. Based on this data, machine learning algorithms are used to analyze the data and evaluate the accident risk for each driver. This analysis uses programming languages ​​such as Python and machine learning frameworks such as Tensorflow® and PyTorch.

[0074] Based on the analysis results, the server uses a generative AI model to construct a VR accident experience tailored to each individual driver. The generative AI model is responsible for generating complex virtual reality scenarios based on prompt statements. For example, a specific prompt statement could be, "Design a VR experience that recreates a rear-end collision based on a scenario in which the driver frequently uses sudden braking."

[0075] Simulation data is sent to a terminal, which then enables the user to experience virtual reality via a VR headset. This process may utilize 3D development platforms such as Unreal Engine or Unity. The terminal also updates a database by recording the user's actions and reactions during the VR experience and sending this data to a server.

[0076] Users experience the VR environment through a VR headset and receive feedback that allows them to reflect on their own driving behavior. After the VR experience, a safe driving challenge is presented from the device, and tasks are set that lead to specific improvements in driving skills.

[0077] This system is designed to make it easier for individual drivers to learn safe driving through explicit feedback and experiential learning.

[0078] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0079] Step 1:

[0080] The server collects driver data from sensors and positioning systems installed in the vehicle. This data includes vehicle speed, acceleration, number of sudden braking incidents, driving time, and route taken. The collected driving data is entered into the server's database and stored in an organized manner.

[0081] Step 2:

[0082] The server analyzes the collected data using machine learning algorithms. The input is driver data, and the output includes behavioral patterns characteristic of each driver and an assessment of accident risk. The analysis uses programming languages ​​such as Python and frameworks like TensorFlow and PyTorch to extract hidden patterns in the data and classify and evaluate driver behavior.

[0083] Step 3:

[0084] The server generates a virtual reality accident experience using a generative AI model based on the analysis results. The input is characteristic data of driving behavior obtained through analysis, and prompts are used to output instructions to the generative AI to construct a specific scenario. An example of a prompt is, "Create a simulation of a rear-end collision based on sudden braking." The AI ​​generates an appropriate VR experience according to these instructions.

[0085] Step 4:

[0086] The terminal receives the generated virtual reality data from the server. On the terminal, the VR data is converted into a usable format, preparing it for the user to experience through a VR headset. Here, the input is the generated VR data, and the output is the VR simulation experienced by the user.

[0087] Step 5:

[0088] The user wears a VR headset via a terminal and experiences a customized virtual reality scenario. The user's reactions and actions are recorded in real time and sent to a server via the terminal. The input here is the user's actions and reactions, and the output is a detailed record of that data.

[0089] Step 6:

[0090] After the experience ends, the device presents the user with a safe driving challenge. The input is behavioral data based on the previous VR experience, and the output is a driving task aimed at improving the driver's skills. For example, instructions such as "Drive the course at a constant speed and avoid sudden braking" may be presented.

[0091] Step 7:

[0092] The server re-analyzes the data obtained from the experience and challenges to generate a dashboard. This dashboard graphically displays the driver's progress and areas for improvement. The input is user behavior data and challenge results, and the output is analyzed visual data. This allows drivers and managers to receive feedback that helps improve their safe driving skills.

[0093] (Application Example 1)

[0094] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0095] To improve the reaction time and accuracy of drivers operating automated vehicles, there is a need to efficiently provide education and training tailored to the individual characteristics of each driver. However, conventional systems have problems in that such individualized support is difficult, and the effectiveness of such training is limited.

[0096] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0097] In this invention, the server includes means for acquiring information on the driver's driving state, means for analyzing the acquired driving state information to calculate the accident risk for each driver, and means for providing a virtual situation in which the driver evaluates reaction time and accuracy in operating an automated vehicle. This enables the provision of education and training optimized for the individual characteristics of each driver, and fosters the ability to respond quickly and accurately in actual driving situations.

[0098] "Driver's driving status information" is a general term for related data such as speed, steering operation, and braking information generated when the driver operates the vehicle.

[0099] "Virtual reality accident experience" is an educational tool in which drivers virtually experience accident situations through virtual reality simulations, allowing them to practically learn how to deal with risk situations.

[0100] An "automated control vehicle" is a general term for a vehicle that can operate itself in accordance with traffic conditions, minimizing the involvement of a human driver.

[0101] "Reaction time and accuracy" refers to how quickly and accurately a driver can respond to unexpected situations, and is an important indicator of safe driving.

[0102] "Driver progress information" is a general term for information that shows, in numerical or graphical form, the extent to which a driver has improved their safe driving skills through education and training.

[0103] "Providing evaluation" means analyzing areas for improvement and strengths regarding specific performance and behaviors demonstrated by the driver during training in a virtual environment or during actual driving, and providing the driver with specific feedback.

[0104] A "display screen" is a display used by drivers and supervisors to visually check the driver's progress and training results, and to provide necessary guidance and evaluation.

[0105] This invention is a system that provides education and training tailored to the individual characteristics of drivers. The server acquires information on the driver's driving state and uses dedicated software (e.g., Python) to analyze it. From the analysis results, a generative AI model (e.g., TensorFlow) is used to design a virtual accident experience optimized for the driver. In this virtual environment, realistic accident scenarios are reproduced using a development platform such as Unity.

[0106] The terminal receives virtual reality data transmitted from the server, allowing the driver to experience it using smart glasses or a head-mounted display. Through this experience, the driver receives practical training to improve their reaction time and accuracy in operating automated vehicles.

[0107] One concrete example is a scenario that simulates changes in road conditions due to sudden weather changes. Drivers can undergo virtual training in a situation simulating a slippery road surface due to sudden rain, learning skills to maintain an appropriate following distance. The server analyzes the driver's reactions and behavioral data acquired during these virtual experiences and generates a display screen that visualizes the driver's progress.

[0108] Another example of a prompt using a generated AI model is: "Due to rain, simulate in VR a situation where the vehicle in front suddenly brakes. This scenario will be used to measure the reaction time of a test driver." Based on this prompt, the AI ​​generates a simulation optimized for driver training.

[0109] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0110] Step 1:

[0111] The server acquires information about the driver's driving state. Specifically, it collects data such as speed, steering input, and braking information from sensors and cameras mounted on the vehicle. The input is this sensor data, and the output is integrated driving state information for analysis.

[0112] Step 2:

[0113] The server analyzes the acquired driving status information using Python. This analysis identifies driving patterns to calculate accident risk. The input is integrated driving status information, and the output is the calculated accident risk profile for each individual driver.

[0114] Step 3:

[0115] The server uses a generative AI model (TensorFlow) to generate a virtual environment accident experience optimized for the driver. The input here is the driver's accident risk profile, and the output is simulation data for a customized virtual environment. The AI ​​generates appropriate simulation content by utilizing prompts.

[0116] Step 4:

[0117] The terminal receives virtual environment simulation data sent from the server and outputs it to smart glasses or a head-mounted display. It converts the data format so that the driver can experience it in a VR environment. The input is virtual environment simulation data, and the output is display data for the user to experience VR.

[0118] Step 5:

[0119] The user wears smart glasses and participates in a driving simulation within a virtual environment. The driver's reaction time and operational accuracy are recorded. The input is the displayed VR experience environment, and the output is the driver's behavior data.

[0120] Step 6:

[0121] The server analyzes driver behavior data and generates progress information. The analysis includes an evaluation of the behavior data, and the progress information is presented through a dashboard. The input is driver behavior data, and the output is a visualized progress dashboard.

[0122] Step 7:

[0123] The terminal displays a driver progress dashboard to the user and provides feedback to improve driving skills. This allows the driver to identify areas for improvement. The input is the progress dashboard, and the output is visual feedback to the user.

[0124] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0125] This invention is a customized educational tool aimed at improving drivers' safe driving skills. It is a system that utilizes the driver's driving data and emotional state to provide virtual reality (VR) experiences and feedback that address specific risks. In particular, by incorporating an emotion engine, this invention realizes an optimal learning environment that takes the driver's emotional state into account.

[0126] The server first acquires data on the driver's driving conditions. This includes indicators such as vehicle speed, accelerator and brake operation, distance traveled, and driving time. The acquired data is then analyzed to calculate accident risks specific to the driver. For example, the driver's driving habits are evaluated based on factors such as the number of sudden braking incidents and the frequency of speeding.

[0127] Furthermore, the server uses an emotion engine to collect emotional data from the driver. This is done through facial recognition and voice analysis during driving, in order to understand the driver's stress level, decreased attention span, and other factors.

[0128] After analysis, the server generates a customized VR accident experience tailored to the driver's characteristics and emotional state. This experience is designed to reflect specific risk situations, such as a scenario that simulates a rear-end collision when the driver is under high stress. The VR data is configured so that the scenario is dynamically adjusted based on the driver's emotional state, which changes in real time.

[0129] The device receives VR data transmitted from the server and delivers that experience to the user via a VR headset. The user can experience a realistic accident scenario in virtual reality and feel their own emotional reactions within that scenario.

[0130] The data obtained through the experience is recorded on the device and sent back to the server. The server then analyzes this data and generates feedback that also takes into account the driver's mental state. This feedback includes specific advice to strengthen the driver's weaknesses and information that helps improve driving skills.

[0131] Ultimately, the server visualizes the driver's progress and provides it to users and administrators through a dashboard. This dashboard shows the driver's emotional state evolving and the degree of improvement in their driving skills, enabling more effective guidance and training.

[0132] Thus, the system of the present invention promotes the formation of long-term and practical safe driving habits through a learning experience that reflects the driver's emotions in real time.

[0133] The following describes the processing flow.

[0134] Step 1:

[0135] The server acquires driver status data. This is a process that collects data in real time from in-vehicle systems and external devices and stores it in a database.

[0136] Step 2:

[0137] The server analyzes the acquired driving data using an AI algorithm and calculates the accident risk for each driver. The analysis includes indicators to evaluate driving characteristics such as speed, frequency of sudden braking, and driving time.

[0138] Step 3:

[0139] The server uses an emotion engine to acquire the driver's emotional state. The emotion engine analyzes stress levels and attention levels from the driver's voice and facial expressions while driving, and evaluates the driver's internal state.

[0140] Step 4:

[0141] The server generates individually customized VR accident experiences based on the driver's driving data and emotional state. This process includes scenario design to recreate specific accident situations that the driver might face.

[0142] Step 5:

[0143] The terminal receives VR data transmitted from the server and provides the driver with the experience using a VR headset. The preparatory stage involves loading the VR environment and synchronizing the device.

[0144] Step 6:

[0145] Users participate in a VR accident experience using their device. They respond to various scenarios that unfold within virtual reality, experiencing and feeling their own emotions and actions.

[0146] Step 7:

[0147] The device records the driver's behavioral data and emotional responses during the VR experience and sends it to a server. This data is used to later evaluate the results of the experience.

[0148] Step 8:

[0149] The server analyzes the collected data and generates feedback based on the driver's emotions and driving skills. This feedback includes suggestions for improvement that take into account the driver's mental state.

[0150] Step 9:

[0151] The server visualizes driver progress data and provides it to drivers and administrators as a dashboard. This allows drivers to track their progress and receive necessary guidance.

[0152] Step 10:

[0153] The device presents drivers with the next challenge for safe driving and provides a structured environment for continuous skill improvement.

[0154] (Example 2)

[0155] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0156] To improve drivers' safe driving skills, an educational system is needed that considers not only driving behavior but also emotional states. However, conventional systems have failed to provide a customized learning experience that adequately reflects individual emotional states. As a result, there has been a lack of feedback and training tailored to the driver's own weaknesses and emotions, making it difficult to effectively improve driving skills.

[0157] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0158] In this invention, the server includes means for acquiring driver driving state data, means for analyzing driver emotional data to understand the driver's emotional state, and means for generating a customized virtual reality accident experience according to the emotional state. This makes it possible to provide more practical and effective feedback and training through individualized accident experiences based not only on the driver's driving characteristics but also on their emotional state.

[0159] "Driving status data" refers to data collected to record the driver's driving behavior, including indicators such as vehicle speed, accelerator and brake operation, distance traveled, and driving time.

[0160] "Accident risk" refers to numerical values ​​or indicators analyzed to assess the likelihood that a driver's driving behavior or habits will cause a specific accident.

[0161] "Emotional data" refers to information obtained from methods such as facial recognition and voice analysis to understand the emotional state of a driver while they are driving.

[0162] "Virtual reality accident experience" is a technology that allows drivers to experience accident scenarios in a virtual space, providing an experience that simulates real-world accident situations.

[0163] "Feedback" refers to providing drivers with specific advice and information based on data obtained through virtual reality accident simulations, which helps them improve their driving skills and habits.

[0164] A "dashboard" is an interface that visually displays the driver's progress and emotional state over time, enabling drivers and managers to monitor progress and provide guidance.

[0165] This invention is a customized educational system for improving drivers' safe driving skills. The system utilizes the driver's driving data and emotional state to provide virtual reality (VR) experiences and feedback that address specific accident risks.

[0166] The server collects driver data using the vehicle's onboard sensor system. This sensor system includes vehicle speed sensors, accelerator and brake position sensors, and an odometer. The data is transmitted to the server and recorded in a database. Algorithms are used to analyze the data and calculate each driver's accident risk. Furthermore, an emotion engine is used to analyze the driver's facial expressions and voice through in-car cameras and microphones to assess their emotional state.

[0167] Based on the analysis results, the server generates a customized VR accident experience tailored to the driver's risk characteristics and emotional state. It utilizes a generation AI model and prompts to generate scenarios. For example, a prompt such as "Generate a scenario that simulates a rear-end collision under high stress levels" might be used. This scenario is encoded in VR playback format and sent to the terminal.

[0168] The device delivers the experience to the user via a VR headset. The VR headset integrates the user's field of view and plays real-time visual and audio content tailored to the scenario. Users can experience realistic accident situations within virtual reality.

[0169] Data collected during the user's experience is sent back to the server. The server analyzes the experience data and generates feedback that takes into account the driver's risk characteristics and emotional state. This feedback includes specific advice needed to improve driving habits. Users and administrators can use the server-provided dashboard to visualize the driver's progress and plan more effective guidance and training.

[0170] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0171] Step 1:

[0172] The server collects driving status data from the vehicle. Inputs include vehicle speed, accelerator and brake operation, and mileage. This data is acquired in real time from sensors mounted on the vehicle. Specifically, a data collection module on the server receives this data and records it in a database. The output is the recorded driving status data.

[0173] Step 2:

[0174] The server analyzes the collected driving data to calculate accident risk. The input is the driving data obtained in step 1. Using a data analysis algorithm, it calculates the number of sudden braking incidents and the frequency of speeding, and evaluates the driver's driving characteristics. The output is accident risk assessment data as a result of the analysis.

[0175] Step 3:

[0176] The server analyzes the driver's emotional data. The input consists of the driver's facial expressions and voice, captured via in-car cameras and microphones. An emotion engine analyzes this data to assess the driver's stress level and decreased attention span. Specifically, it uses image recognition and voice analysis technologies to quantify the emotional state. The output is the driver's emotional state data.

[0177] Step 4:

[0178] The server generates a customized VR accident experience based on the user's emotional state. The inputs are accident risk assessment data from step 2 and emotional state data from step 3. A generation AI model is used to create a VR scenario based on prompts. A specific prompt might be, "Generate a scenario that simulates a rear-end collision while the user is experiencing high stress levels." The output is VR data.

[0179] Step 5:

[0180] The terminal receives VR data sent from the server and provides it to the user through a VR headset. The input is the VR data generated in step 4. Specifically, the terminal's VR application decodes the data and plays the VR experience through the user's sight and hearing. The output is the user's VR accident experience.

[0181] Step 6:

[0182] The server analyzes the user's experience data and generates feedback. The input is the user's reaction data during the experience in step 5. Specifically, the server analyzes the user's heart rate changes and operation history and creates feedback that includes specific advice for improving driving skills. The output is a feedback report.

[0183] Step 7:

[0184] The server visualizes the driver's progress and provides it to users and administrators through a dashboard. Inputs include feedback reports and emotional state trend data. Specifically, it uses visualization tools to display progress graphs and improvement indicators on the dashboard. The output is the visualized progress.

[0185] (Application Example 2)

[0186] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0187] In autonomous driving systems, there is a lack of opportunities to effectively evaluate and improve the driving skills and situational awareness capabilities of both the driver and the system. If this challenge is not addressed, autonomous driving systems may lack the ability to respond safely and appropriately to unexpected situations. Therefore, it is necessary to provide new methods to enhance skill improvement and risk avoidance capabilities by offering real-time, situation-sensitive feedback to both the autonomous driving system and the driver.

[0188] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0189] In this invention, the server includes means for acquiring driver's driving state data and emotional state; means for analyzing the acquired driving state data and emotional state data to calculate the traffic risk for each driver; and means for generating a virtual reality experience that takes into account the driver's emotional state and the vehicle's condition. This allows the driver and the autonomous driving system to evaluate their situational awareness and reaction capabilities in real time within the virtual environment and optimize them as needed.

[0190] "Driving status data" refers to information recorded by the driver or the autonomous driving system during driving, such as vehicle speed, accelerator and brake operation history, mileage, and driving time.

[0191] "Emotional state" refers to data indicating the psychological and physiological responses a driver exhibits while driving, such as stress levels and decreased attention span.

[0192] "Traffic risk" is an indicator that shows the degree to which a driver or an autonomous driving system is likely to cause an accident or problem under specific driving conditions.

[0193] A "virtual reality experience" is a simulated environment created to recreate a specific driving scenario, in which a driver or autonomous driving system participates.

[0194] "Situational awareness" refers to the ability of an autonomous driving system or driver to accurately perceive the surrounding environment and potential changes while driving, and to make appropriate judgments.

[0195] "Responsiveness" refers to the ability of an autonomous driving system or driver to respond quickly to changes in circumstances.

[0196] To implement this invention, a system is needed that collects driving information from the driver or the autonomous driving system, analyzes it, and provides feedback. The server first acquires the driver's driving state data and emotional state through sensors and cameras mounted on the vehicle. The applicable hardware includes cameras for image analysis and various sensors necessary for real-time data collection. OpenCV or other image processing tools are used as emotion recognition software.

[0197] The server combines and analyzes acquired driving data and sentiment data to generate indicators for evaluating the traffic risk of drivers or autonomous driving systems. This analysis is performed using advanced AI algorithms. Programming languages ​​such as Python and C++ are used to implement these algorithms.

[0198] Next, the server generates a virtual reality experience that reflects the driver's emotional state and the vehicle's condition. VR development platforms such as Unity and Unreal Engine are used for this generation. The driver or autonomous driving system improves its situational awareness and reaction capabilities through scenarios within these virtual environments.

[0199] A concrete example would be testing the response of an autonomous driving system in a scenario where an obstacle suddenly appears during rainy weather. Feedback based on such scenarios is visually represented using data and analysis results, as it contributes to improving the capabilities of both the driver and the system.

[0200] An example of a prompt using a generative AI model is: "Create a scenario where a parked vehicle appears ahead in poor visibility conditions due to rain. Evaluate the reaction speed and adaptability of the autonomous driving AI." This allows for learning and improvement in a situation that closely resembles real-world conditions.

[0201] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0202] Step 1:

[0203] The server collects data on the driver's driving state and emotional state through sensors and cameras. Inputs include data such as speed detected by vehicle sensors, accelerator and brake operation history, and driving time. In addition, emotional information based on facial expressions and voice is obtained from image data obtained using cameras. The server filters this data and removes unnecessary noise to generate a dataset suitable for analysis.

[0204] Step 2:

[0205] The server analyzes the collected driving state data and emotional state data to assess the driver's traffic risk. The input data used is the driving state data and emotional state data obtained in Step 1. The analysis utilizes an AI algorithm to calculate a risk score based on the number of sudden braking incidents and the frequency of speeding. The resulting risk assessment information is used in the subsequent generation of virtual reality experiences.

[0206] Step 3:

[0207] The server generates a virtual reality experience based on the driver's emotional state and risk assessment. The inputs here are the risk assessment information and emotional state data generated in step 2. Specific driving scenarios are dynamically created using VR development platforms such as Unity or Unreal Engine. The output is the VR scenario presented to the driver.

[0208] Step 4:

[0209] The user (driver or autonomous driving system) participates in a generated virtual reality experience and evaluates their situational awareness and reaction capabilities within it. Using a VR headset, the scenario is experienced visually and experientially. The output consists of the user's experience and behavioral data within the virtual environment.

[0210] Step 5:

[0211] The server records user behavior data within the VR scenario and generates feedback that contributes to improving the driver's skills. The input is the behavior data collected in step 4. The server analyzes this data and generates feedback that includes skills the driver needs to improve and areas for system optimization. The output is visualized feedback information for the driver and manager.

[0212] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0213] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0214] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0215] [Second Embodiment]

[0216] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0217] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0218] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0219] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0220] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0221] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0222] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0223] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0224] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0225] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0226] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0227] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0228] This invention is implemented as a customized educational tool to improve drivers' safe driving. This system utilizes driver data and, based on that data, generates an AI that provides an optimized virtual reality (VR) accident experience.

[0229] The server first acquires driver data from relevant equipment and then analyzes this data. The purpose of the analysis is to identify the driver's behavior patterns and assess accident risk based on them. This includes factors such as speed, frequency of sudden braking, and driving time.

[0230] Based on the information obtained through analysis, the server uses a generative AI to design a VR accident experience optimized for each driver's characteristics and tendencies. This VR experience can address specific risks that drivers may face, such as a simulation that reproduces a situation where a rear-end collision occurs due to sudden braking.

[0231] Next, the terminal receives customized VR data sent from the server, allowing the driver to participate in these simulations through a VR headset. The driver then experiences the accident scenarios firsthand through the VR experience, and their reactions and actions are recorded.

[0232] After the experience, the device presents the driver with safe driving challenges. These challenges, for example, require the driver to safely complete a specific driving task within a set time limit. In this way, the driver's skills can be improved in a game-like format.

[0233] The server aggregates these records and generates a dashboard that visualizes the progress of each driver. This dashboard is provided to drivers and administrators, offering feedback to help improve drivers' safe driving skills. For example, if the dashboard shows a decrease in the frequency of sudden braking, drivers can see the improvement in that area.

[0234] Thus, the system of the present invention provides individual drivers with a customized risk experience and promotes the establishment of long-term safe driving habits through specific feedback.

[0235] The following describes the processing flow.

[0236] Step 1:

[0237] The server acquires the driver's driving data. This is the process of receiving data transmitted from devices such as in-vehicle systems and dashcams, and storing that information in a database.

[0238] Step 2:

[0239] The server analyzes the acquired driving data using an AI algorithm. The analysis is performed to identify the characteristics of the driver's driving pattern and to evaluate risk factors such as collisions and sudden braking.

[0240] Step 3:

[0241] The server generates accident scenarios optimized for each individual driver based on the analysis results. The generating AI designs accident scenarios tailored to the driver's characteristics and converts them into a virtual reality format.

[0242] Step 4:

[0243] The terminal receives VR data sent from the server and prepares it for the driver to experience using a VR headset. This preparation includes initializing the VR environment and synchronizing with the device.

[0244] Step 5:

[0245] Users participate in a VR accident simulation through their device. They experience driving scenarios in virtual reality and get a feel for the risks involved in actual driving.

[0246] Step 6:

[0247] The device records user interaction data and reactions during the VR experience. This data forms the basis for analyzing user behavior and providing feedback.

[0248] Step 7:

[0249] The server analyzes data from the VR experience and generates a dashboard that visualizes the driver's progress and improvements. This dashboard shows improvements and achievements to both users and administrators.

[0250] Step 8:

[0251] The device presents users with appropriate safe driving challenges and provides tasks to further hone their skills. This allows users to continuously work towards increasing their safety awareness.

[0252] (Example 1)

[0253] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0254] While modern drivers recognize the importance of safe driving, they are often unable to effectively manage accident risks in their daily driving activities. Traditional training methods struggle to provide personalized instruction based on each driver's behavior and tendencies, and therefore fail to adequately improve practical safe driving skills. This invention aims to solve these problems by providing a training system that allows for detailed evaluation of each driver's risk profile and practice of countermeasures in a virtual environment.

[0255] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0256] In this invention, the server includes means for monitoring the driver's driving state and collecting data, means for analyzing the collected driving data and evaluating the accident risk of individual drivers, and means for using a generative AI model to generate a virtual reality environment adapted according to the driver's characteristics. This makes it possible to provide each driver with a risk experience tailored to their individual needs and subsequent specific guidance and feedback within the virtual reality environment.

[0257] "Driving conditions" refers to information about the driver's actions and the vehicle's movements while operating the vehicle, and includes speed, acceleration, braking, steering, etc.

[0258] "Data analysis" refers to the process of processing diverse collected data using statistical methods and machine learning to evaluate driver behavior patterns and risks.

[0259] "Accident risk" refers to the degree of likelihood of a vehicle accident predicted based on specific driving behaviors and environmental conditions.

[0260] A "virtual reality environment" refers to a simulated three-dimensional environment that users can experience with a sense of realism using computer graphics and specialized equipment.

[0261] "Generative AI models" refer to artificial intelligence algorithms used to generate outputs tailored to specific purposes, such as generative adversarial networks (GANs) and transformer models.

[0262] "Recording" refers to the process of saving data such as the driver's actions and reactions within the virtual reality environment in a format that allows for later analysis and evaluation.

[0263] "Progress" refers to information about specific improvements and changes in proficiency that a driver has achieved through past driving experiences and training.

[0264] "Visualization" refers to the process of displaying analyzed data and information in visual formats such as graphs and charts to make them easier to understand.

[0265] A "challenge" refers to a specific driving task or challenge set for a driver in order to improve their safe driving skills.

[0266] The embodiment of the invention aims to provide drivers with an innovative educational system to improve their safe driving skills. Specifically, it is designed so that the server, terminals, and users function according to their respective roles.

[0267] The server collects driver data in real time from sensors and positioning systems installed in the vehicle. The collected data includes speed, acceleration, braking frequency, and driving time. Based on this data, machine learning algorithms are used to analyze the data and evaluate the accident risk for each driver. This analysis uses programming languages ​​such as Python and machine learning frameworks such as TensorFlow and PyTorch.

[0268] Based on the analysis results, the server uses a generative AI model to construct a VR accident experience tailored to each individual driver. The generative AI model is responsible for generating complex virtual reality scenarios based on prompt statements. For example, a specific prompt statement could be, "Design a VR experience that recreates a rear-end collision based on a scenario in which the driver frequently uses sudden braking."

[0269] Simulation data is sent to a terminal, which then enables the user to experience virtual reality via a VR headset. This process may utilize 3D development platforms such as Unreal Engine or Unity. The terminal also updates a database by recording the user's actions and reactions during the VR experience and sending this data to a server.

[0270] Users experience the VR environment through a VR headset and receive feedback that allows them to reflect on their own driving behavior. After the VR experience, a safe driving challenge is presented from the device, and tasks are set that lead to specific improvements in driving skills.

[0271] This system is designed to make it easier for individual drivers to learn safe driving through explicit feedback and experiential learning.

[0272] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0273] Step 1:

[0274] The server collects driver data from sensors and positioning systems installed in the vehicle. This data includes vehicle speed, acceleration, number of sudden braking incidents, driving time, and route taken. The collected driving data is entered into the server's database and stored in an organized manner.

[0275] Step 2:

[0276] The server analyzes the collected data using machine learning algorithms. The input is driver data, and the output includes behavioral patterns characteristic of each driver and an assessment of accident risk. The analysis uses programming languages ​​such as Python and frameworks like TensorFlow and PyTorch to extract hidden patterns in the data and classify and evaluate driver behavior.

[0277] Step 3:

[0278] The server generates a virtual reality accident experience using a generative AI model based on the analysis results. The input is characteristic data of driving behavior obtained through analysis, and prompts are used to output instructions to the generative AI to construct a specific scenario. An example of a prompt is, "Create a simulation of a rear-end collision based on sudden braking." The AI ​​generates an appropriate VR experience according to these instructions.

[0279] Step 4:

[0280] The terminal receives the generated virtual reality data from the server. On the terminal, the VR data is converted into a usable format, preparing it for the user to experience through a VR headset. Here, the input is the generated VR data, and the output is the VR simulation experienced by the user.

[0281] Step 5:

[0282] The user wears a VR headset through the terminal and experiences a customized virtual reality scenario. The user's reactions and actions are recorded in real time and transmitted to the server via the terminal. The input here is the user's actions and reactions, and the output is a detailed record of that data.

[0283] Step 6:

[0284] After the experience, the terminal presents a safe driving challenge to the user. The input is the action data based on the previous VR experience, and the output is a driving task aimed at improving the driver's skills. For example, instructions such as "Drive on the course at a constant speed and avoid sudden braking" are presented.

[0285] Step 7:

[0286] The server re-analyzes the data obtained from the experience and the challenge and generates a dashboard. This dashboard graphically displays the driver's progress and areas for improvement. The input is the user's action data and the challenge results, and the output is the analyzed visual data. Thereby, the driver and the administrator can receive feedback useful for improving safe driving skills.

[0287] (Application Example 1)

[0288] [[ID=​​​​​​​​​​

[0291] In this invention, the server includes means for acquiring information on the driver's driving state, means for analyzing the acquired driving state information to calculate the accident risk for each driver, and means for providing a virtual situation in which the driver evaluates reaction time and accuracy in operating an automated vehicle. This enables the provision of education and training optimized for the individual characteristics of each driver, and fosters the ability to respond quickly and accurately in actual driving situations.

[0292] "Driver's driving status information" is a general term for related data such as speed, steering operation, and braking information generated when the driver operates the vehicle.

[0293] "Virtual reality accident experience" is an educational tool in which drivers virtually experience accident situations through virtual reality simulations, allowing them to practically learn how to deal with risk situations.

[0294] An "automated control vehicle" is a general term for a vehicle that can operate itself in accordance with traffic conditions, minimizing the involvement of a human driver.

[0295] "Reaction time and accuracy" refers to how quickly and accurately a driver can respond to unexpected situations, and is an important indicator of safe driving.

[0296] "Driver progress information" is a general term for information that shows, in numerical or graphical form, the extent to which a driver has improved their safe driving skills through education and training.

[0297] "Providing evaluation" means analyzing areas for improvement and strengths regarding specific performance and behaviors demonstrated by the driver during training in a virtual environment or during actual driving, and providing the driver with specific feedback.

[0298] A "display screen" is a display used by drivers and supervisors to visually check the driver's progress and training results, and to provide necessary guidance and evaluation.

[0299] This invention is a system that provides education and training tailored to the individual characteristics of drivers. The server acquires information on the driver's driving state and uses dedicated software (e.g., Python) to analyze it. From the analysis results, a generative AI model (e.g., TensorFlow) is used to design a virtual accident experience optimized for the driver. In this virtual environment, realistic accident scenarios are reproduced using a development platform such as Unity.

[0300] The terminal receives virtual reality data transmitted from the server, allowing the driver to experience it using smart glasses or a head-mounted display. Through this experience, the driver receives practical training to improve their reaction time and accuracy in operating automated vehicles.

[0301] One concrete example is a scenario that simulates changes in road conditions due to sudden weather changes. Drivers can undergo virtual training in a situation simulating a slippery road surface due to sudden rain, learning skills to maintain an appropriate following distance. The server analyzes the driver's reactions and behavioral data acquired during these virtual experiences and generates a display screen that visualizes the driver's progress.

[0302] Another example of a prompt using a generated AI model is: "Due to rain, simulate in VR a situation where the vehicle in front suddenly brakes. This scenario will be used to measure the reaction time of a test driver." Based on this prompt, the AI ​​generates a simulation optimized for driver training.

[0303] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0304] Step 1:

[0305] The server acquires the driving state information of the driver. Specifically, it collects data such as speed, steering wheel operation, and braking information from sensors and cameras installed in the vehicle. The input is these sensor data, and the output is the integrated driving state information for analysis.

[0306] Step 2:

[0307] The server analyzes the acquired driving state information using Python. In this analysis, the driving pattern is identified to calculate the accident risk. The input is the integrated driving state information, and the output is the calculated accident risk profile for each individual driver.

[0308] Step 3:

[0309] The server utilizes a generative AI model (TensorFlow) to generate an optimal virtual environment accident experience for the driver. Here, the input is the driver's accident risk profile, and the output is the simulation data of the customized virtual environment. The prompt text is utilized, and the AI generates appropriate simulation content.

[0310] Step 4:

[0311] The terminal receives the virtual environment simulation data transmitted from the server and outputs it to smart glasses or a head-mounted display. The data format is converted so that the driver can experience it in a VR environment. The input is the virtual environment simulation data, and the output is the display data for the user to have a VR experience.

[0312] Step 5:

[0313] ​​​​​​​

[0315] The server analyzes driver behavior data and generates progress information. The analysis includes an evaluation of the behavior data, and the progress information is presented through a dashboard. The input is driver behavior data, and the output is a visualized progress dashboard.

[0316] Step 7:

[0317] The terminal displays a driver progress dashboard to the user and provides feedback to improve driving skills. This allows the driver to identify areas for improvement. The input is the progress dashboard, and the output is visual feedback to the user.

[0318] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0319] This invention is a customized educational tool aimed at improving drivers' safe driving skills. It is a system that utilizes the driver's driving data and emotional state to provide virtual reality (VR) experiences and feedback that address specific risks. In particular, by incorporating an emotion engine, this invention realizes an optimal learning environment that takes the driver's emotional state into account.

[0320] The server first acquires data on the driver's driving conditions. This includes indicators such as vehicle speed, accelerator and brake operation, distance traveled, and driving time. The acquired data is then analyzed to calculate accident risks specific to the driver. For example, the driver's driving habits are evaluated based on factors such as the number of sudden braking incidents and the frequency of speeding.

[0321] Furthermore, the server uses an emotion engine to collect emotional data from the driver. This is done through facial recognition and voice analysis during driving, in order to understand the driver's stress level, decreased attention span, and other factors.

[0322] After analysis, the server generates a customized VR accident experience tailored to the driver's characteristics and emotional state. This experience is designed to reflect specific risk situations, such as a scenario that simulates a rear-end collision when the driver is under high stress. The VR data is configured so that the scenario is dynamically adjusted based on the driver's emotional state, which changes in real time.

[0323] The device receives VR data transmitted from the server and delivers that experience to the user via a VR headset. The user can experience a realistic accident scenario in virtual reality and feel their own emotional reactions within that scenario.

[0324] The data obtained through the experience is recorded on the device and sent back to the server. The server then analyzes this data and generates feedback that also takes into account the driver's mental state. This feedback includes specific advice to strengthen the driver's weaknesses and information that helps improve driving skills.

[0325] Ultimately, the server visualizes the driver's progress and provides it to users and administrators through a dashboard. This dashboard shows the driver's emotional state evolving and the degree of improvement in their driving skills, enabling more effective guidance and training.

[0326] Thus, the system of the present invention promotes the formation of long-term and practical safe driving habits through a learning experience that reflects the driver's emotions in real time.

[0327] The following describes the processing flow.

[0328] Step 1:

[0329] The server acquires driver status data. This is a process that collects data in real time from in-vehicle systems and external devices and stores it in a database.

[0330] Step 2:

[0331] The server analyzes the acquired driving data using an AI algorithm and calculates the accident risk for each driver. The analysis includes indicators to evaluate driving characteristics such as speed, frequency of sudden braking, and driving time.

[0332] Step 3:

[0333] The server uses an emotion engine to acquire the driver's emotional state. The emotion engine analyzes stress levels and attention levels from the driver's voice and facial expressions while driving, and evaluates the driver's internal state.

[0334] Step 4:

[0335] The server generates individually customized VR accident experiences based on the driver's driving data and emotional state. This process includes scenario design to recreate specific accident situations that the driver might face.

[0336] Step 5:

[0337] The terminal receives VR data transmitted from the server and provides the driver with the experience using a VR headset. The preparatory stage involves loading the VR environment and synchronizing the device.

[0338] Step 6:

[0339] Users participate in a VR accident experience using their device. They respond to various scenarios that unfold within virtual reality, experiencing and feeling their own emotions and actions.

[0340] Step 7:

[0341] The device records the driver's behavioral data and emotional responses during the VR experience and sends it to a server. This data is used to later evaluate the results of the experience.

[0342] Step 8:

[0343] The server analyzes the collected data and generates feedback based on the driver's emotions and driving skills. This feedback includes suggestions for improvement that take into account the driver's mental state.

[0344] Step 9:

[0345] The server visualizes driver progress data and provides it to drivers and administrators as a dashboard. This allows drivers to track their progress and receive necessary guidance.

[0346] Step 10:

[0347] The device presents drivers with the next challenge for safe driving and provides a structured environment for continuous skill improvement.

[0348] (Example 2)

[0349] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0350] To improve drivers' safe driving skills, an educational system is needed that considers not only driving behavior but also emotional states. However, conventional systems have failed to provide a customized learning experience that adequately reflects individual emotional states. As a result, there has been a lack of feedback and training tailored to the driver's own weaknesses and emotions, making it difficult to effectively improve driving skills.

[0351] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0352] In this invention, the server includes means for acquiring driver driving state data, means for analyzing driver emotional data to understand the driver's emotional state, and means for generating a customized virtual reality accident experience according to the emotional state. This makes it possible to provide more practical and effective feedback and training through individualized accident experiences based not only on the driver's driving characteristics but also on their emotional state.

[0353] "Driving status data" refers to data collected to record the driver's driving behavior, including indicators such as vehicle speed, accelerator and brake operation, distance traveled, and driving time.

[0354] "Accident risk" refers to numerical values ​​or indicators analyzed to assess the likelihood that a driver's driving behavior or habits will cause a specific accident.

[0355] "Emotional data" refers to information obtained from methods such as facial recognition and voice analysis to understand the emotional state of a driver while they are driving.

[0356] "Virtual reality accident experience" is a technology that allows drivers to experience accident scenarios in a virtual space, providing an experience that simulates real-world accident situations.

[0357] "Feedback" refers to providing drivers with specific advice and information based on data obtained through virtual reality accident simulations, which helps them improve their driving skills and habits.

[0358] A "dashboard" is an interface that visually displays the driver's progress and emotional state over time, enabling drivers and managers to monitor progress and provide guidance.

[0359] This invention is a customized educational system for improving drivers' safe driving skills. The system utilizes the driver's driving data and emotional state to provide virtual reality (VR) experiences and feedback that address specific accident risks.

[0360] The server collects driver data using the vehicle's onboard sensor system. This sensor system includes vehicle speed sensors, accelerator and brake position sensors, and an odometer. The data is transmitted to the server and recorded in a database. Algorithms are used to analyze the data and calculate each driver's accident risk. Furthermore, an emotion engine is used to analyze the driver's facial expressions and voice through in-car cameras and microphones to assess their emotional state.

[0361] Based on the analysis results, the server generates a customized VR accident experience tailored to the driver's risk characteristics and emotional state. It utilizes a generation AI model and prompts to generate scenarios. For example, a prompt such as "Generate a scenario that simulates a rear-end collision under high stress levels" might be used. This scenario is encoded in VR playback format and sent to the terminal.

[0362] The device delivers the experience to the user via a VR headset. The VR headset integrates the user's field of view and plays real-time visual and audio content tailored to the scenario. Users can experience realistic accident situations within virtual reality.

[0363] Data collected during the user's experience is sent back to the server. The server analyzes the experience data and generates feedback that takes into account the driver's risk characteristics and emotional state. This feedback includes specific advice needed to improve driving habits. Users and administrators can use the server-provided dashboard to visualize the driver's progress and plan more effective guidance and training.

[0364] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0365] Step 1:

[0366] The server collects driving status data from the vehicle. Inputs include vehicle speed, accelerator and brake operation, and mileage. This data is acquired in real time from sensors mounted on the vehicle. Specifically, a data collection module on the server receives this data and records it in a database. The output is the recorded driving status data.

[0367] Step 2:

[0368] The server analyzes the collected driving data to calculate accident risk. The input is the driving data obtained in step 1. Using a data analysis algorithm, it calculates the number of sudden braking incidents and the frequency of speeding, and evaluates the driver's driving characteristics. The output is accident risk assessment data as a result of the analysis.

[0369] Step 3:

[0370] The server analyzes the driver's emotional data. The input consists of the driver's facial expressions and voice, captured via in-car cameras and microphones. An emotion engine analyzes this data to assess the driver's stress level and decreased attention span. Specifically, it uses image recognition and voice analysis technologies to quantify the emotional state. The output is the driver's emotional state data.

[0371] Step 4:

[0372] The server generates a customized VR accident experience based on the user's emotional state. The inputs are accident risk assessment data from step 2 and emotional state data from step 3. A generation AI model is used to create a VR scenario based on prompts. A specific prompt might be, "Generate a scenario that simulates a rear-end collision while the user is experiencing high stress levels." The output is VR data.

[0373] Step 5:

[0374] The terminal receives VR data sent from the server and provides it to the user through a VR headset. The input is the VR data generated in step 4. Specifically, the terminal's VR application decodes the data and plays the VR experience through the user's sight and hearing. The output is the user's VR accident experience.

[0375] Step 6:

[0376] The server analyzes the user's experience data and generates feedback. The input is the user's reaction data during the experience in step 5. Specifically, the server analyzes the user's heart rate changes and operation history and creates feedback that includes specific advice for improving driving skills. The output is a feedback report.

[0377] Step 7:

[0378] The server visualizes the driver's progress and provides it to users and administrators through a dashboard. Inputs include feedback reports and emotional state trend data. Specifically, it uses visualization tools to display progress graphs and improvement indicators on the dashboard. The output is the visualized progress.

[0379] (Application Example 2)

[0380] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0381] In autonomous driving systems, there is a lack of opportunities to effectively evaluate and improve the driving skills and situational awareness capabilities of both the driver and the system. If this challenge is not addressed, autonomous driving systems may lack the ability to respond safely and appropriately to unexpected situations. Therefore, it is necessary to provide new methods to enhance skill improvement and risk avoidance capabilities by offering real-time, situation-sensitive feedback to both the autonomous driving system and the driver.

[0382] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0383] In this invention, the server includes means for acquiring driver's driving state data and emotional state; means for analyzing the acquired driving state data and emotional state data to calculate the traffic risk for each driver; and means for generating a virtual reality experience that takes into account the driver's emotional state and the vehicle's condition. This allows the driver and the autonomous driving system to evaluate their situational awareness and reaction capabilities in real time within the virtual environment and optimize them as needed.

[0384] "Driving status data" refers to information recorded by the driver or the autonomous driving system during driving, such as vehicle speed, accelerator and brake operation history, mileage, and driving time.

[0385] "Emotional state" refers to data indicating the psychological and physiological responses a driver exhibits while driving, such as stress levels and decreased attention span.

[0386] "Traffic risk" is an indicator that shows the degree to which a driver or an autonomous driving system is likely to cause an accident or problem under specific driving conditions.

[0387] A "virtual reality experience" is a simulated environment created to recreate a specific driving scenario, in which a driver or autonomous driving system participates.

[0388] "Situational awareness" refers to the ability of an autonomous driving system or driver to accurately perceive the surrounding environment and potential changes while driving, and to make appropriate judgments.

[0389] "Responsiveness" refers to the ability of an autonomous driving system or driver to respond quickly to changes in circumstances.

[0390] To implement this invention, a system is needed that collects driving information from the driver or the autonomous driving system, analyzes it, and provides feedback. The server first acquires the driver's driving state data and emotional state through sensors and cameras mounted on the vehicle. The applicable hardware includes cameras for image analysis and various sensors necessary for real-time data collection. OpenCV or other image processing tools are used as emotion recognition software.

[0391] The server combines and analyzes acquired driving data and sentiment data to generate indicators for evaluating the traffic risk of drivers or autonomous driving systems. This analysis is performed using advanced AI algorithms. Programming languages ​​such as Python and C++ are used to implement these algorithms.

[0392] Next, the server generates a virtual reality experience that reflects the driver's emotional state and the vehicle's condition. VR development platforms such as Unity and Unreal Engine are used for this generation. The driver or autonomous driving system improves its situational awareness and reaction capabilities through scenarios within these virtual environments.

[0393] A concrete example would be testing the response of an autonomous driving system in a scenario where an obstacle suddenly appears during rainy weather. Feedback based on such scenarios is visually represented using data and analysis results, as it contributes to improving the capabilities of both the driver and the system.

[0394] An example of a prompt using a generative AI model is: "Create a scenario where a parked vehicle appears ahead in poor visibility conditions due to rain. Evaluate the reaction speed and adaptability of the autonomous driving AI." This allows for learning and improvement in a situation that closely resembles real-world conditions.

[0395] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0396] Step 1:

[0397] The server collects data on the driver's driving state and emotional state through sensors and cameras. Inputs include data such as speed detected by vehicle sensors, accelerator and brake operation history, and driving time. In addition, emotional information based on facial expressions and voice is obtained from image data obtained using cameras. The server filters this data and removes unnecessary noise to generate a dataset suitable for analysis.

[0398] Step 2:

[0399] The server analyzes the collected driving state data and emotional state data to assess the driver's traffic risk. The input data used is the driving state data and emotional state data obtained in Step 1. The analysis utilizes an AI algorithm to calculate a risk score based on the number of sudden braking incidents and the frequency of speeding. The resulting risk assessment information is used in the subsequent generation of virtual reality experiences.

[0400] Step 3:

[0401] The server generates a virtual reality experience based on the driver's emotional state and risk assessment. The inputs here are the risk assessment information and emotional state data generated in step 2. Specific driving scenarios are dynamically created using VR development platforms such as Unity or Unreal Engine. The output is the VR scenario presented to the driver.

[0402] Step 4:

[0403] The user (driver or autonomous driving system) participates in a generated virtual reality experience and evaluates their situational awareness and reaction capabilities within it. Using a VR headset, the scenario is experienced visually and experientially. The output consists of the user's experience and behavioral data within the virtual environment.

[0404] Step 5:

[0405] The server records user behavior data within the VR scenario and generates feedback that contributes to improving the driver's skills. The input is the behavior data collected in step 4. The server analyzes this data and generates feedback that includes skills the driver needs to improve and areas for system optimization. The output is visualized feedback information for the driver and manager.

[0406] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0407] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0408] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0409] [Third Embodiment]

[0410] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0411] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0412] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0413] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0414] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0415] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0416] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0417] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0418] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0419] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0420] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0421] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0422] This invention is implemented as a customized educational tool to improve drivers' safe driving. This system utilizes driver data and, based on that data, generates an AI that provides an optimized virtual reality (VR) accident experience.

[0423] The server first acquires driver data from relevant equipment and then analyzes this data. The purpose of the analysis is to identify the driver's behavior patterns and assess accident risk based on them. This includes factors such as speed, frequency of sudden braking, and driving time.

[0424] Based on the information obtained through analysis, the server uses a generative AI to design a VR accident experience optimized for each driver's characteristics and tendencies. This VR experience can address specific risks that drivers may face, such as a simulation that reproduces a situation where a rear-end collision occurs due to sudden braking.

[0425] Next, the terminal receives customized VR data sent from the server, allowing the driver to participate in these simulations through a VR headset. The driver then experiences the accident scenarios firsthand through the VR experience, and their reactions and actions are recorded.

[0426] After the experience, the device presents the driver with safe driving challenges. These challenges, for example, require the driver to safely complete a specific driving task within a set time limit. In this way, the driver's skills can be improved in a game-like format.

[0427] The server aggregates these records and generates a dashboard that visualizes the progress of each driver. This dashboard is provided to drivers and administrators, offering feedback to help improve drivers' safe driving skills. For example, if the dashboard shows a decrease in the frequency of sudden braking, drivers can see the improvement in that area.

[0428] Thus, the system of the present invention provides individual drivers with a customized risk experience and promotes the establishment of long-term safe driving habits through specific feedback.

[0429] The following describes the processing flow.

[0430] Step 1:

[0431] The server acquires the driver's driving data. This is the process of receiving data transmitted from devices such as in-vehicle systems and dashcams, and storing that information in a database.

[0432] Step 2:

[0433] The server analyzes the acquired driving data using an AI algorithm. The analysis is performed to identify the characteristics of the driver's driving pattern and to evaluate risk factors such as collisions and sudden braking.

[0434] Step 3:

[0435] The server generates accident scenarios optimized for each individual driver based on the analysis results. The generating AI designs accident scenarios tailored to the driver's characteristics and converts them into a virtual reality format.

[0436] Step 4:

[0437] The terminal receives VR data sent from the server and prepares it for the driver to experience using a VR headset. This preparation includes initializing the VR environment and synchronizing with the device.

[0438] Step 5:

[0439] Users participate in a VR accident simulation through their device. They experience driving scenarios in virtual reality and get a feel for the risks involved in actual driving.

[0440] Step 6:

[0441] The device records user interaction data and reactions during the VR experience. This data forms the basis for analyzing user behavior and providing feedback.

[0442] Step 7:

[0443] The server analyzes data from the VR experience and generates a dashboard that visualizes the driver's progress and improvements. This dashboard shows improvements and achievements to both users and administrators.

[0444] Step 8:

[0445] The device presents users with appropriate safe driving challenges and provides tasks to further hone their skills. This allows users to continuously work towards increasing their safety awareness.

[0446] (Example 1)

[0447] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0448] While modern drivers recognize the importance of safe driving, they are often unable to effectively manage accident risks in their daily driving activities. Traditional training methods struggle to provide personalized instruction based on each driver's behavior and tendencies, and therefore fail to adequately improve practical safe driving skills. This invention aims to solve these problems by providing a training system that allows for detailed evaluation of each driver's risk profile and practice of countermeasures in a virtual environment.

[0449] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0450] In this invention, the server includes means for monitoring the driver's driving state and collecting data, means for analyzing the collected driving data and evaluating the accident risk of individual drivers, and means for using a generative AI model to generate a virtual reality environment adapted according to the driver's characteristics. This makes it possible to provide each driver with a risk experience tailored to their individual needs and subsequent specific guidance and feedback within the virtual reality environment.

[0451] "Driving conditions" refers to information about the driver's actions and the vehicle's movements while operating the vehicle, and includes speed, acceleration, braking, steering, etc.

[0452] "Data analysis" refers to the process of processing diverse collected data using statistical methods and machine learning to evaluate driver behavior patterns and risks.

[0453] "Accident risk" refers to the degree of likelihood of a vehicle accident predicted based on specific driving behaviors and environmental conditions.

[0454] A "virtual reality environment" refers to a simulated three-dimensional environment that users can experience with a sense of realism using computer graphics and specialized equipment.

[0455] "Generative AI models" refer to artificial intelligence algorithms used to generate outputs tailored to specific purposes, such as generative adversarial networks (GANs) and transformer models.

[0456] "Recording" refers to the process of saving data such as the driver's actions and reactions within the virtual reality environment in a format that allows for later analysis and evaluation.

[0457] "Progress" refers to information about specific improvements and changes in proficiency that a driver has achieved through past driving experiences and training.

[0458] "Visualization" refers to the process of displaying analyzed data and information in visual formats such as graphs and charts to make them easier to understand.

[0459] A "challenge" refers to a specific driving task or challenge set for a driver in order to improve their safe driving skills.

[0460] The embodiment of the invention aims to provide drivers with an innovative educational system to improve their safe driving skills. Specifically, it is designed so that the server, terminals, and users function according to their respective roles.

[0461] The server collects driver data in real time from sensors and positioning systems installed in the vehicle. The collected data includes speed, acceleration, braking frequency, and driving time. Based on this data, machine learning algorithms are used to analyze the data and evaluate the accident risk for each driver. This analysis uses programming languages ​​such as Python and machine learning frameworks such as TensorFlow and PyTorch.

[0462] Based on the analysis results, the server uses a generative AI model to construct a VR accident experience tailored to each individual driver. The generative AI model is responsible for generating complex virtual reality scenarios based on prompt statements. For example, a specific prompt statement could be, "Design a VR experience that recreates a rear-end collision based on a scenario in which the driver frequently uses sudden braking."

[0463] Simulation data is sent to a terminal, which then enables the user to experience virtual reality via a VR headset. This process may utilize 3D development platforms such as Unreal Engine or Unity. The terminal also updates a database by recording the user's actions and reactions during the VR experience and sending this data to a server.

[0464] Users experience the VR environment through a VR headset and receive feedback that allows them to reflect on their own driving behavior. After the VR experience, a safe driving challenge is presented from the device, and tasks are set that lead to specific improvements in driving skills.

[0465] This system is designed to make it easier for individual drivers to learn safe driving through explicit feedback and experiential learning.

[0466] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0467] Step 1:

[0468] The server collects driver data from sensors and positioning systems installed in the vehicle. This data includes vehicle speed, acceleration, number of sudden braking incidents, driving time, and route taken. The collected driving data is entered into the server's database and stored in an organized manner.

[0469] Step 2:

[0470] The server analyzes the collected data using machine learning algorithms. The input is driver data, and the output includes behavioral patterns characteristic of each driver and an assessment of accident risk. The analysis uses programming languages ​​such as Python and frameworks like TensorFlow and PyTorch to extract hidden patterns in the data and classify and evaluate driver behavior.

[0471] Step 3:

[0472] The server generates a virtual reality accident experience using a generative AI model based on the analysis results. The input is characteristic data of driving behavior obtained through analysis, and prompts are used to output instructions to the generative AI to construct a specific scenario. An example of a prompt is, "Create a simulation of a rear-end collision based on sudden braking." The AI ​​generates an appropriate VR experience according to these instructions.

[0473] Step 4:

[0474] The terminal receives the generated virtual reality data from the server. On the terminal, the VR data is converted into a usable format, preparing it for the user to experience through a VR headset. Here, the input is the generated VR data, and the output is the VR simulation experienced by the user.

[0475] Step 5:

[0476] The user wears a VR headset via a terminal and experiences a customized virtual reality scenario. The user's reactions and actions are recorded in real time and sent to a server via the terminal. The input here is the user's actions and reactions, and the output is a detailed record of that data.

[0477] Step 6:

[0478] After the experience ends, the device presents the user with a safe driving challenge. The input is behavioral data based on the previous VR experience, and the output is a driving task aimed at improving the driver's skills. For example, instructions such as "Drive the course at a constant speed and avoid sudden braking" may be presented.

[0479] Step 7:

[0480] The server re-analyzes the data obtained from the experience and challenges to generate a dashboard. This dashboard graphically displays the driver's progress and areas for improvement. The input is user behavior data and challenge results, and the output is analyzed visual data. This allows drivers and managers to receive feedback that helps improve their safe driving skills.

[0481] (Application Example 1)

[0482] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0483] To improve the reaction time and accuracy of drivers operating automated vehicles, there is a need to efficiently provide education and training tailored to the individual characteristics of each driver. However, conventional systems have problems in that such individualized support is difficult, and the effectiveness of such training is limited.

[0484] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0485] In this invention, the server includes means for acquiring information on the driver's driving state, means for analyzing the acquired driving state information to calculate the accident risk for each driver, and means for providing a virtual situation in which the driver evaluates reaction time and accuracy in operating an automated vehicle. This enables the provision of education and training optimized for the individual characteristics of each driver, and fosters the ability to respond quickly and accurately in actual driving situations.

[0486] "Driver's driving status information" is a general term for related data such as speed, steering operation, and braking information generated when the driver operates the vehicle.

[0487] "Virtual reality accident experience" is an educational tool in which drivers virtually experience accident situations through virtual reality simulations, allowing them to practically learn how to deal with risk situations.

[0488] An "automated control vehicle" is a general term for a vehicle that can operate itself in accordance with traffic conditions, minimizing the involvement of a human driver.

[0489] "Reaction time and accuracy" refers to how quickly and accurately a driver can respond to unexpected situations, and is an important indicator of safe driving.

[0490] "Driver progress information" is a general term for information that shows, in numerical or graphical form, the extent to which a driver has improved their safe driving skills through education and training.

[0491] "Providing evaluation" means analyzing areas for improvement and strengths regarding specific performance and behaviors demonstrated by the driver during training in a virtual environment or during actual driving, and providing the driver with specific feedback.

[0492] A "display screen" is a display used by drivers and supervisors to visually check the driver's progress and training results, and to provide necessary guidance and evaluation.

[0493] This invention is a system that provides education and training tailored to the individual characteristics of drivers. The server acquires information on the driver's driving state and uses dedicated software (e.g., Python) to analyze it. From the analysis results, a generative AI model (e.g., TensorFlow) is used to design a virtual accident experience optimized for the driver. In this virtual environment, realistic accident scenarios are reproduced using a development platform such as Unity.

[0494] The terminal receives virtual reality data transmitted from the server, allowing the driver to experience it using smart glasses or a head-mounted display. Through this experience, the driver receives practical training to improve their reaction time and accuracy in operating automated vehicles.

[0495] One concrete example is a scenario that simulates changes in road conditions due to sudden weather changes. Drivers can undergo virtual training in a situation simulating a slippery road surface due to sudden rain, learning skills to maintain an appropriate following distance. The server analyzes the driver's reactions and behavioral data acquired during these virtual experiences and generates a display screen that visualizes the driver's progress.

[0496] Another example of a prompt using a generated AI model is: "Due to rain, simulate in VR a situation where the vehicle in front suddenly brakes. This scenario will be used to measure the reaction time of a test driver." Based on this prompt, the AI ​​generates a simulation optimized for driver training.

[0497] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0498] Step 1:

[0499] The server acquires information about the driver's driving state. Specifically, it collects data such as speed, steering input, and braking information from sensors and cameras mounted on the vehicle. The input is this sensor data, and the output is integrated driving state information for analysis.

[0500] Step 2:

[0501] The server analyzes the acquired driving status information using Python. This analysis identifies driving patterns to calculate accident risk. The input is integrated driving status information, and the output is the calculated accident risk profile for each individual driver.

[0502] Step 3:

[0503] The server uses a generative AI model (TensorFlow) to generate a virtual environment accident experience optimized for the driver. The input here is the driver's accident risk profile, and the output is simulation data for a customized virtual environment. The AI ​​generates appropriate simulation content by utilizing prompts.

[0504] Step 4:

[0505] The terminal receives virtual environment simulation data sent from the server and outputs it to smart glasses or a head-mounted display. It converts the data format so that the driver can experience it in a VR environment. The input is virtual environment simulation data, and the output is display data for the user to experience VR.

[0506] Step 5:

[0507] The user wears smart glasses and participates in a driving simulation within a virtual environment. The driver's reaction time and operational accuracy are recorded. The input is the displayed VR experience environment, and the output is the driver's behavior data.

[0508] Step 6:

[0509] The server analyzes driver behavior data and generates progress information. The analysis includes an evaluation of the behavior data, and the progress information is presented through a dashboard. The input is driver behavior data, and the output is a visualized progress dashboard.

[0510] Step 7:

[0511] The terminal displays a driver progress dashboard to the user and provides feedback to improve driving skills. This allows the driver to identify areas for improvement. The input is the progress dashboard, and the output is visual feedback to the user.

[0512] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0513] This invention is a customized educational tool aimed at improving drivers' safe driving skills. It is a system that utilizes the driver's driving data and emotional state to provide virtual reality (VR) experiences and feedback that address specific risks. In particular, by incorporating an emotion engine, this invention realizes an optimal learning environment that takes the driver's emotional state into account.

[0514] The server first acquires data on the driver's driving conditions. This includes indicators such as vehicle speed, accelerator and brake operation, distance traveled, and driving time. The acquired data is then analyzed to calculate accident risks specific to the driver. For example, the driver's driving habits are evaluated based on factors such as the number of sudden braking incidents and the frequency of speeding.

[0515] Furthermore, the server uses an emotion engine to collect emotional data from the driver. This is done through facial recognition and voice analysis during driving, in order to understand the driver's stress level, decreased attention span, and other factors.

[0516] After analysis, the server generates a customized VR accident experience tailored to the driver's characteristics and emotional state. This experience is designed to reflect specific risk situations, such as a scenario that simulates a rear-end collision when the driver is under high stress. The VR data is configured so that the scenario is dynamically adjusted based on the driver's emotional state, which changes in real time.

[0517] The device receives VR data transmitted from the server and delivers that experience to the user via a VR headset. The user can experience a realistic accident scenario in virtual reality and feel their own emotional reactions within that scenario.

[0518] The data obtained through the experience is recorded on the device and sent back to the server. The server then analyzes this data and generates feedback that also takes into account the driver's mental state. This feedback includes specific advice to strengthen the driver's weaknesses and information that helps improve driving skills.

[0519] Ultimately, the server visualizes the driver's progress and provides it to users and administrators through a dashboard. This dashboard shows the driver's emotional state evolving and the degree of improvement in their driving skills, enabling more effective guidance and training.

[0520] Thus, the system of the present invention promotes the formation of long-term and practical safe driving habits through a learning experience that reflects the driver's emotions in real time.

[0521] The following describes the processing flow.

[0522] Step 1:

[0523] The server acquires driver status data. This is a process that collects data in real time from in-vehicle systems and external devices and stores it in a database.

[0524] Step 2:

[0525] The server analyzes the acquired driving data using an AI algorithm and calculates the accident risk for each driver. The analysis includes indicators to evaluate driving characteristics such as speed, frequency of sudden braking, and driving time.

[0526] Step 3:

[0527] The server uses an emotion engine to acquire the driver's emotional state. The emotion engine analyzes stress levels and attention levels from the driver's voice and facial expressions while driving, and evaluates the driver's internal state.

[0528] Step 4:

[0529] The server generates individually customized VR accident experiences based on the driver's driving data and emotional state. This process includes scenario design to recreate specific accident situations that the driver might face.

[0530] Step 5:

[0531] The terminal receives VR data transmitted from the server and provides the driver with the experience using a VR headset. The preparatory stage involves loading the VR environment and synchronizing the device.

[0532] Step 6:

[0533] Users participate in a VR accident experience using their device. They respond to various scenarios that unfold within virtual reality, experiencing and feeling their own emotions and actions.

[0534] Step 7:

[0535] The device records the driver's behavioral data and emotional responses during the VR experience and sends it to a server. This data is used to later evaluate the results of the experience.

[0536] Step 8:

[0537] The server analyzes the collected data and generates feedback based on the driver's emotions and driving skills. This feedback includes suggestions for improvement that take into account the driver's mental state.

[0538] Step 9:

[0539] The server visualizes driver progress data and provides it to drivers and administrators as a dashboard. This allows drivers to track their progress and receive necessary guidance.

[0540] Step 10:

[0541] The device presents drivers with the next challenge for safe driving and provides a structured environment for continuous skill improvement.

[0542] (Example 2)

[0543] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0544] To improve drivers' safe driving skills, an educational system is needed that considers not only driving behavior but also emotional states. However, conventional systems have failed to provide a customized learning experience that adequately reflects individual emotional states. As a result, there has been a lack of feedback and training tailored to the driver's own weaknesses and emotions, making it difficult to effectively improve driving skills.

[0545] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0546] In this invention, the server includes means for acquiring driver driving state data, means for analyzing driver emotional data to understand the driver's emotional state, and means for generating a customized virtual reality accident experience according to the emotional state. This makes it possible to provide more practical and effective feedback and training through individualized accident experiences based not only on the driver's driving characteristics but also on their emotional state.

[0547] "Driving status data" refers to data collected to record the driver's driving behavior, including indicators such as vehicle speed, accelerator and brake operation, distance traveled, and driving time.

[0548] "Accident risk" refers to numerical values ​​or indicators analyzed to assess the likelihood that a driver's driving behavior or habits will cause a specific accident.

[0549] "Emotional data" refers to information obtained from methods such as facial recognition and voice analysis to understand the emotional state of a driver while they are driving.

[0550] "Virtual reality accident experience" is a technology that allows drivers to experience accident scenarios in a virtual space, providing an experience that simulates real-world accident situations.

[0551] "Feedback" refers to providing drivers with specific advice and information based on data obtained through virtual reality accident simulations, which helps them improve their driving skills and habits.

[0552] A "dashboard" is an interface that visually displays the driver's progress and emotional state over time, enabling drivers and managers to monitor progress and provide guidance.

[0553] This invention is a customized educational system for improving drivers' safe driving skills. The system utilizes the driver's driving data and emotional state to provide virtual reality (VR) experiences and feedback that address specific accident risks.

[0554] The server collects driver data using the vehicle's onboard sensor system. This sensor system includes vehicle speed sensors, accelerator and brake position sensors, and an odometer. The data is transmitted to the server and recorded in a database. Algorithms are used to analyze the data and calculate each driver's accident risk. Furthermore, an emotion engine is used to analyze the driver's facial expressions and voice through in-car cameras and microphones to assess their emotional state.

[0555] Based on the analysis results, the server generates a customized VR accident experience tailored to the driver's risk characteristics and emotional state. It utilizes a generation AI model and prompts to generate scenarios. For example, a prompt such as "Generate a scenario that simulates a rear-end collision under high stress levels" might be used. This scenario is encoded in VR playback format and sent to the terminal.

[0556] The device delivers the experience to the user via a VR headset. The VR headset integrates the user's field of view and plays real-time visual and audio content tailored to the scenario. Users can experience realistic accident situations within virtual reality.

[0557] Data collected during the user's experience is sent back to the server. The server analyzes the experience data and generates feedback that takes into account the driver's risk characteristics and emotional state. This feedback includes specific advice needed to improve driving habits. Users and administrators can use the server-provided dashboard to visualize the driver's progress and plan more effective guidance and training.

[0558] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0559] Step 1:

[0560] The server collects driving status data from the vehicle. Inputs include vehicle speed, accelerator and brake operation, and mileage. This data is acquired in real time from sensors mounted on the vehicle. Specifically, a data collection module on the server receives this data and records it in a database. The output is the recorded driving status data.

[0561] Step 2:

[0562] The server analyzes the collected driving data to calculate accident risk. The input is the driving data obtained in step 1. Using a data analysis algorithm, it calculates the number of sudden braking incidents and the frequency of speeding, and evaluates the driver's driving characteristics. The output is accident risk assessment data as a result of the analysis.

[0563] Step 3:

[0564] The server analyzes the driver's emotional data. The input consists of the driver's facial expressions and voice, captured via in-car cameras and microphones. An emotion engine analyzes this data to assess the driver's stress level and decreased attention span. Specifically, it uses image recognition and voice analysis technologies to quantify the emotional state. The output is the driver's emotional state data.

[0565] Step 4:

[0566] The server generates a customized VR accident experience based on the user's emotional state. The inputs are accident risk assessment data from step 2 and emotional state data from step 3. A generation AI model is used to create a VR scenario based on prompts. A specific prompt might be, "Generate a scenario that simulates a rear-end collision while the user is experiencing high stress levels." The output is VR data.

[0567] Step 5:

[0568] The terminal receives VR data sent from the server and provides it to the user through a VR headset. The input is the VR data generated in step 4. Specifically, the terminal's VR application decodes the data and plays the VR experience through the user's sight and hearing. The output is the user's VR accident experience.

[0569] Step 6:

[0570] The server analyzes the user's experience data and generates feedback. The input is the user's reaction data during the experience in step 5. Specifically, the server analyzes the user's heart rate changes and operation history and creates feedback that includes specific advice for improving driving skills. The output is a feedback report.

[0571] Step 7:

[0572] The server visualizes the driver's progress and provides it to users and administrators through a dashboard. Inputs include feedback reports and emotional state trend data. Specifically, it uses visualization tools to display progress graphs and improvement indicators on the dashboard. The output is the visualized progress.

[0573] (Application Example 2)

[0574] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0575] In autonomous driving systems, there is a lack of opportunities to effectively evaluate and improve the driving skills and situational awareness capabilities of both the driver and the system. If this challenge is not addressed, autonomous driving systems may lack the ability to respond safely and appropriately to unexpected situations. Therefore, it is necessary to provide new methods to enhance skill improvement and risk avoidance capabilities by offering real-time, situation-sensitive feedback to both the autonomous driving system and the driver.

[0576] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0577] In this invention, the server includes means for acquiring driver's driving state data and emotional state; means for analyzing the acquired driving state data and emotional state data to calculate the traffic risk for each driver; and means for generating a virtual reality experience that takes into account the driver's emotional state and the vehicle's condition. This allows the driver and the autonomous driving system to evaluate their situational awareness and reaction capabilities in real time within the virtual environment and optimize them as needed.

[0578] "Driving status data" refers to information recorded by the driver or the autonomous driving system during driving, such as vehicle speed, accelerator and brake operation history, mileage, and driving time.

[0579] "Emotional state" refers to data indicating the psychological and physiological responses a driver exhibits while driving, such as stress levels and decreased attention span.

[0580] "Traffic risk" is an indicator that shows the degree to which a driver or an autonomous driving system is likely to cause an accident or problem under specific driving conditions.

[0581] A "virtual reality experience" is a simulated environment created to recreate a specific driving scenario, in which a driver or autonomous driving system participates.

[0582] "Situational awareness" refers to the ability of an autonomous driving system or driver to accurately perceive the surrounding environment and potential changes while driving, and to make appropriate judgments.

[0583] "Responsiveness" refers to the ability of an autonomous driving system or driver to respond quickly to changes in circumstances.

[0584] To implement this invention, a system is needed that collects driving information from the driver or the autonomous driving system, analyzes it, and provides feedback. The server first acquires the driver's driving state data and emotional state through sensors and cameras mounted on the vehicle. The applicable hardware includes cameras for image analysis and various sensors necessary for real-time data collection. OpenCV or other image processing tools are used as emotion recognition software.

[0585] The server combines and analyzes acquired driving data and sentiment data to generate indicators for evaluating the traffic risk of drivers or autonomous driving systems. This analysis is performed using advanced AI algorithms. Programming languages ​​such as Python and C++ are used to implement these algorithms.

[0586] Next, the server generates a virtual reality experience that reflects the driver's emotional state and the vehicle's condition. VR development platforms such as Unity and Unreal Engine are used for this generation. The driver or autonomous driving system improves its situational awareness and reaction capabilities through scenarios within these virtual environments.

[0587] A concrete example would be testing the response of an autonomous driving system in a scenario where an obstacle suddenly appears during rainy weather. Feedback based on such scenarios is visually represented using data and analysis results, as it contributes to improving the capabilities of both the driver and the system.

[0588] An example of a prompt using a generative AI model is: "Create a scenario where a parked vehicle appears ahead in poor visibility conditions due to rain. Evaluate the reaction speed and adaptability of the autonomous driving AI." This allows for learning and improvement in a situation that closely resembles real-world conditions.

[0589] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0590] Step 1:

[0591] The server collects data on the driver's driving state and emotional state through sensors and cameras. Inputs include data such as speed detected by vehicle sensors, accelerator and brake operation history, and driving time. In addition, emotional information based on facial expressions and voice is obtained from image data obtained using cameras. The server filters this data and removes unnecessary noise to generate a dataset suitable for analysis.

[0592] Step 2:

[0593] The server analyzes the collected driving state data and emotional state data to assess the driver's traffic risk. The input data used is the driving state data and emotional state data obtained in Step 1. The analysis utilizes an AI algorithm to calculate a risk score based on the number of sudden braking incidents and the frequency of speeding. The resulting risk assessment information is used in the subsequent generation of virtual reality experiences.

[0594] Step 3:

[0595] The server generates a virtual reality experience based on the driver's emotional state and risk assessment. The inputs here are the risk assessment information and emotional state data generated in step 2. Specific driving scenarios are dynamically created using VR development platforms such as Unity or Unreal Engine. The output is the VR scenario presented to the driver.

[0596] Step 4:

[0597] The user (driver or autonomous driving system) participates in a generated virtual reality experience and evaluates their situational awareness and reaction capabilities within it. Using a VR headset, the scenario is experienced visually and experientially. The output consists of the user's experience and behavioral data within the virtual environment.

[0598] Step 5:

[0599] The server records user behavior data within the VR scenario and generates feedback that contributes to improving the driver's skills. The input is the behavior data collected in step 4. The server analyzes this data and generates feedback that includes skills the driver needs to improve and areas for system optimization. The output is visualized feedback information for the driver and manager.

[0600] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0601] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0602] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0603] [Fourth Embodiment]

[0604] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0605] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0606] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0607] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0608] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0609] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0610] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0611] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0612] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0613] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0614] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0615] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0616] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0617] This invention is implemented as a customized educational tool to improve drivers' safe driving. This system utilizes driver data and, based on that data, generates an AI that provides an optimized virtual reality (VR) accident experience.

[0618] The server first acquires driver data from relevant equipment and then analyzes this data. The purpose of the analysis is to identify the driver's behavior patterns and assess accident risk based on them. This includes factors such as speed, frequency of sudden braking, and driving time.

[0619] Based on the information obtained through analysis, the server uses a generative AI to design a VR accident experience optimized for each driver's characteristics and tendencies. This VR experience can address specific risks that drivers may face, such as a simulation that reproduces a situation where a rear-end collision occurs due to sudden braking.

[0620] Next, the terminal receives customized VR data sent from the server, allowing the driver to participate in these simulations through a VR headset. The driver then experiences the accident scenarios firsthand through the VR experience, and their reactions and actions are recorded.

[0621] After the experience, the device presents the driver with safe driving challenges. These challenges, for example, require the driver to safely complete a specific driving task within a set time limit. In this way, the driver's skills can be improved in a game-like format.

[0622] The server aggregates these records and generates a dashboard that visualizes the progress of each driver. This dashboard is provided to drivers and administrators, offering feedback to help improve drivers' safe driving skills. For example, if the dashboard shows a decrease in the frequency of sudden braking, drivers can see the improvement in that area.

[0623] Thus, the system of the present invention provides individual drivers with a customized risk experience and promotes the establishment of long-term safe driving habits through specific feedback.

[0624] The following describes the processing flow.

[0625] Step 1:

[0626] The server acquires the driver's driving data. This is the process of receiving data transmitted from devices such as in-vehicle systems and dashcams, and storing that information in a database.

[0627] Step 2:

[0628] The server analyzes the acquired driving data using an AI algorithm. The analysis is performed to identify the characteristics of the driver's driving pattern and to evaluate risk factors such as collisions and sudden braking.

[0629] Step 3:

[0630] The server generates accident scenarios optimized for each individual driver based on the analysis results. The generating AI designs accident scenarios tailored to the driver's characteristics and converts them into a virtual reality format.

[0631] Step 4:

[0632] The terminal receives VR data sent from the server and prepares it for the driver to experience using a VR headset. This preparation includes initializing the VR environment and synchronizing with the device.

[0633] Step 5:

[0634] Users participate in a VR accident simulation through their device. They experience driving scenarios in virtual reality and get a feel for the risks involved in actual driving.

[0635] Step 6:

[0636] The device records user interaction data and reactions during the VR experience. This data forms the basis for analyzing user behavior and providing feedback.

[0637] Step 7:

[0638] The server analyzes data from the VR experience and generates a dashboard that visualizes the driver's progress and improvements. This dashboard shows improvements and achievements to both users and administrators.

[0639] Step 8:

[0640] The device presents users with appropriate safe driving challenges and provides tasks to further hone their skills. This allows users to continuously work towards increasing their safety awareness.

[0641] (Example 1)

[0642] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0643] While modern drivers recognize the importance of safe driving, they are often unable to effectively manage accident risks in their daily driving activities. Traditional training methods struggle to provide personalized instruction based on each driver's behavior and tendencies, and therefore fail to adequately improve practical safe driving skills. This invention aims to solve these problems by providing a training system that allows for detailed evaluation of each driver's risk profile and practice of countermeasures in a virtual environment.

[0644] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0645] In this invention, the server includes means for monitoring the driver's driving state and collecting data, means for analyzing the collected driving data and evaluating the accident risk of individual drivers, and means for using a generative AI model to generate a virtual reality environment adapted according to the driver's characteristics. This makes it possible to provide each driver with a risk experience tailored to their individual needs and subsequent specific guidance and feedback within the virtual reality environment.

[0646] "Driving conditions" refers to information about the driver's actions and the vehicle's movements while operating the vehicle, and includes speed, acceleration, braking, steering, etc.

[0647] "Data analysis" refers to the process of processing diverse collected data using statistical methods and machine learning to evaluate driver behavior patterns and risks.

[0648] "Accident risk" refers to the degree of likelihood of a vehicle accident predicted based on specific driving behaviors and environmental conditions.

[0649] A "virtual reality environment" refers to a simulated three-dimensional environment that users can experience with a sense of realism using computer graphics and specialized equipment.

[0650] "Generative AI models" refer to artificial intelligence algorithms used to generate outputs tailored to specific purposes, such as generative adversarial networks (GANs) and transformer models.

[0651] "Recording" refers to the process of saving data such as the driver's actions and reactions within the virtual reality environment in a format that allows for later analysis and evaluation.

[0652] "Progress" refers to information about specific improvements and changes in proficiency that a driver has achieved through past driving experiences and training.

[0653] "Visualization" refers to the process of displaying analyzed data and information in visual formats such as graphs and charts to make them easier to understand.

[0654] A "challenge" refers to a specific driving task or challenge set for a driver in order to improve their safe driving skills.

[0655] The embodiment of the invention aims to provide drivers with an innovative educational system to improve their safe driving skills. Specifically, it is designed so that the server, terminals, and users function according to their respective roles.

[0656] The server collects driver data in real time from sensors and positioning systems installed in the vehicle. The collected data includes speed, acceleration, braking frequency, and driving time. Based on this data, machine learning algorithms are used to analyze the data and evaluate the accident risk for each driver. This analysis uses programming languages ​​such as Python and machine learning frameworks such as TensorFlow and PyTorch.

[0657] Based on the analysis results, the server uses a generative AI model to construct a VR accident experience tailored to each individual driver. The generative AI model is responsible for generating complex virtual reality scenarios based on prompt statements. For example, a specific prompt statement could be, "Design a VR experience that recreates a rear-end collision based on a scenario in which the driver frequently uses sudden braking."

[0658] Simulation data is sent to a terminal, which then enables the user to experience virtual reality via a VR headset. This process may utilize 3D development platforms such as Unreal Engine or Unity. The terminal also updates a database by recording the user's actions and reactions during the VR experience and sending this data to a server.

[0659] Users experience the VR environment through a VR headset and receive feedback that allows them to reflect on their own driving behavior. After the VR experience, a safe driving challenge is presented from the device, and tasks are set that lead to specific improvements in driving skills.

[0660] This system is designed to make it easier for individual drivers to learn safe driving through explicit feedback and experiential learning.

[0661] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0662] Step 1:

[0663] The server collects driver data from sensors and positioning systems installed in the vehicle. This data includes vehicle speed, acceleration, number of sudden braking incidents, driving time, and route taken. The collected driving data is entered into the server's database and stored in an organized manner.

[0664] Step 2:

[0665] The server analyzes the collected data using machine learning algorithms. The input is driver data, and the output includes behavioral patterns characteristic of each driver and an assessment of accident risk. The analysis uses programming languages ​​such as Python and frameworks like TensorFlow and PyTorch to extract hidden patterns in the data and classify and evaluate driver behavior.

[0666] Step 3:

[0667] The server generates a virtual reality accident experience using a generative AI model based on the analysis results. The input is characteristic data of driving behavior obtained through analysis, and prompts are used to output instructions to the generative AI to construct a specific scenario. An example of a prompt is, "Create a simulation of a rear-end collision based on sudden braking." The AI ​​generates an appropriate VR experience according to these instructions.

[0668] Step 4:

[0669] The terminal receives the generated virtual reality data from the server. On the terminal, the VR data is converted into a usable format, preparing it for the user to experience through a VR headset. Here, the input is the generated VR data, and the output is the VR simulation experienced by the user.

[0670] Step 5:

[0671] The user wears a VR headset via a terminal and experiences a customized virtual reality scenario. The user's reactions and actions are recorded in real time and sent to a server via the terminal. The input here is the user's actions and reactions, and the output is a detailed record of that data.

[0672] Step 6:

[0673] After the experience ends, the device presents the user with a safe driving challenge. The input is behavioral data based on the previous VR experience, and the output is a driving task aimed at improving the driver's skills. For example, instructions such as "Drive the course at a constant speed and avoid sudden braking" may be presented.

[0674] Step 7:

[0675] The server re-analyzes the data obtained from the experience and challenges to generate a dashboard. This dashboard graphically displays the driver's progress and areas for improvement. The input is user behavior data and challenge results, and the output is analyzed visual data. This allows drivers and managers to receive feedback that helps improve their safe driving skills.

[0676] (Application Example 1)

[0677] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0678] To improve the reaction time and accuracy of drivers operating automated vehicles, there is a need to efficiently provide education and training tailored to the individual characteristics of each driver. However, conventional systems have problems in that such individualized support is difficult, and the effectiveness of such training is limited.

[0679] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0680] In this invention, the server includes means for acquiring information on the driver's driving state, means for analyzing the acquired driving state information to calculate the accident risk for each driver, and means for providing a virtual situation in which the driver evaluates reaction time and accuracy in operating an automated vehicle. This enables the provision of education and training optimized for the individual characteristics of each driver, and fosters the ability to respond quickly and accurately in actual driving situations.

[0681] "Driver's driving status information" is a general term for related data such as speed, steering operation, and braking information generated when the driver operates the vehicle.

[0682] "Virtual reality accident experience" is an educational tool in which drivers virtually experience accident situations through virtual reality simulations, allowing them to practically learn how to deal with risk situations.

[0683] An "automated control vehicle" is a general term for a vehicle that can operate itself in accordance with traffic conditions, minimizing the involvement of a human driver.

[0684] "Reaction time and accuracy" refers to how quickly and accurately a driver can respond to unexpected situations, and is an important indicator of safe driving.

[0685] "Driver progress information" is a general term for information that shows, in numerical or graphical form, the extent to which a driver has improved their safe driving skills through education and training.

[0686] "Providing evaluation" means analyzing areas for improvement and strengths regarding specific performance and behaviors demonstrated by the driver during training in a virtual environment or during actual driving, and providing the driver with specific feedback.

[0687] A "display screen" is a display used by drivers and supervisors to visually check the driver's progress and training results, and to provide necessary guidance and evaluation.

[0688] This invention is a system that provides education and training tailored to the individual characteristics of drivers. The server acquires information on the driver's driving state and uses dedicated software (e.g., Python) to analyze it. From the analysis results, a generative AI model (e.g., TensorFlow) is used to design a virtual accident experience optimized for the driver. In this virtual environment, realistic accident scenarios are reproduced using a development platform such as Unity.

[0689] The terminal receives virtual reality data transmitted from the server, allowing the driver to experience it using smart glasses or a head-mounted display. Through this experience, the driver receives practical training to improve their reaction time and accuracy in operating automated vehicles.

[0690] One concrete example is a scenario that simulates changes in road conditions due to sudden weather changes. Drivers can undergo virtual training in a situation simulating a slippery road surface due to sudden rain, learning skills to maintain an appropriate following distance. The server analyzes the driver's reactions and behavioral data acquired during these virtual experiences and generates a display screen that visualizes the driver's progress.

[0691] Another example of a prompt using a generated AI model is: "Due to rain, simulate in VR a situation where the vehicle in front suddenly brakes. This scenario will be used to measure the reaction time of a test driver." Based on this prompt, the AI ​​generates a simulation optimized for driver training.

[0692] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0693] Step 1:

[0694] The server acquires information about the driver's driving state. Specifically, it collects data such as speed, steering input, and braking information from sensors and cameras mounted on the vehicle. The input is this sensor data, and the output is integrated driving state information for analysis.

[0695] Step 2:

[0696] The server analyzes the acquired driving status information using Python. This analysis identifies driving patterns to calculate accident risk. The input is integrated driving status information, and the output is the calculated accident risk profile for each individual driver.

[0697] Step 3:

[0698] The server uses a generative AI model (TensorFlow) to generate a virtual environment accident experience optimized for the driver. The input here is the driver's accident risk profile, and the output is simulation data for a customized virtual environment. The AI ​​generates appropriate simulation content by utilizing prompts.

[0699] Step 4:

[0700] The terminal receives virtual environment simulation data sent from the server and outputs it to smart glasses or a head-mounted display. It converts the data format so that the driver can experience it in a VR environment. The input is virtual environment simulation data, and the output is display data for the user to experience VR.

[0701] Step 5:

[0702] The user wears smart glasses and participates in a driving simulation within a virtual environment. The driver's reaction time and operational accuracy are recorded. The input is the displayed VR experience environment, and the output is the driver's behavior data.

[0703] Step 6:

[0704] The server analyzes driver behavior data and generates progress information. The analysis includes an evaluation of the behavior data, and the progress information is presented through a dashboard. The input is driver behavior data, and the output is a visualized progress dashboard.

[0705] Step 7:

[0706] The terminal displays a driver progress dashboard to the user and provides feedback to improve driving skills. This allows the driver to identify areas for improvement. The input is the progress dashboard, and the output is visual feedback to the user.

[0707] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0708] This invention is a customized educational tool aimed at improving drivers' safe driving skills. It is a system that utilizes the driver's driving data and emotional state to provide virtual reality (VR) experiences and feedback that address specific risks. In particular, by incorporating an emotion engine, this invention realizes an optimal learning environment that takes the driver's emotional state into account.

[0709] The server first acquires data on the driver's driving conditions. This includes indicators such as vehicle speed, accelerator and brake operation, distance traveled, and driving time. The acquired data is then analyzed to calculate accident risks specific to the driver. For example, the driver's driving habits are evaluated based on factors such as the number of sudden braking incidents and the frequency of speeding.

[0710] Furthermore, the server uses an emotion engine to collect emotional data from the driver. This is done through facial recognition and voice analysis during driving, in order to understand the driver's stress level, decreased attention span, and other factors.

[0711] After analysis, the server generates a customized VR accident experience tailored to the driver's characteristics and emotional state. This experience is designed to reflect specific risk situations, such as a scenario that simulates a rear-end collision when the driver is under high stress. The VR data is configured so that the scenario is dynamically adjusted based on the driver's emotional state, which changes in real time.

[0712] The device receives VR data transmitted from the server and delivers that experience to the user via a VR headset. The user can experience a realistic accident scenario in virtual reality and feel their own emotional reactions within that scenario.

[0713] The data obtained through the experience is recorded on the device and sent back to the server. The server then analyzes this data and generates feedback that also takes into account the driver's mental state. This feedback includes specific advice to strengthen the driver's weaknesses and information that helps improve driving skills.

[0714] Ultimately, the server visualizes the driver's progress and provides it to users and administrators through a dashboard. This dashboard shows the driver's emotional state evolving and the degree of improvement in their driving skills, enabling more effective guidance and training.

[0715] Thus, the system of the present invention promotes the formation of long-term and practical safe driving habits through a learning experience that reflects the driver's emotions in real time.

[0716] The following describes the processing flow.

[0717] Step 1:

[0718] The server acquires driver status data. This is a process that collects data in real time from in-vehicle systems and external devices and stores it in a database.

[0719] Step 2:

[0720] The server analyzes the acquired driving data using an AI algorithm and calculates the accident risk for each driver. The analysis includes indicators to evaluate driving characteristics such as speed, frequency of sudden braking, and driving time.

[0721] Step 3:

[0722] The server uses an emotion engine to acquire the driver's emotional state. The emotion engine analyzes stress levels and attention levels from the driver's voice and facial expressions while driving, and evaluates the driver's internal state.

[0723] Step 4:

[0724] The server generates individually customized VR accident experiences based on the driver's driving data and emotional state. This process includes scenario design to recreate specific accident situations that the driver might face.

[0725] Step 5:

[0726] The terminal receives VR data transmitted from the server and provides the driver with the experience using a VR headset. The preparatory stage involves loading the VR environment and synchronizing the device.

[0727] Step 6:

[0728] Users participate in a VR accident experience using their device. They respond to various scenarios that unfold within virtual reality, experiencing and feeling their own emotions and actions.

[0729] Step 7:

[0730] The device records the driver's behavioral data and emotional responses during the VR experience and sends it to a server. This data is used to later evaluate the results of the experience.

[0731] Step 8:

[0732] The server analyzes the collected data and generates feedback based on the driver's emotions and driving skills. This feedback includes suggestions for improvement that take into account the driver's mental state.

[0733] Step 9:

[0734] The server visualizes driver progress data and provides it to drivers and administrators as a dashboard. This allows drivers to track their progress and receive necessary guidance.

[0735] Step 10:

[0736] The device presents drivers with the next challenge for safe driving and provides a structured environment for continuous skill improvement.

[0737] (Example 2)

[0738] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0739] To improve drivers' safe driving skills, an educational system is needed that considers not only driving behavior but also emotional states. However, conventional systems have failed to provide a customized learning experience that adequately reflects individual emotional states. As a result, there has been a lack of feedback and training tailored to the driver's own weaknesses and emotions, making it difficult to effectively improve driving skills.

[0740] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0741] In this invention, the server includes means for acquiring driver driving state data, means for analyzing driver emotional data to understand the driver's emotional state, and means for generating a customized virtual reality accident experience according to the emotional state. This makes it possible to provide more practical and effective feedback and training through individualized accident experiences based not only on the driver's driving characteristics but also on their emotional state.

[0742] "Driving status data" refers to data collected to record the driver's driving behavior, including indicators such as vehicle speed, accelerator and brake operation, distance traveled, and driving time.

[0743] "Accident risk" refers to numerical values ​​or indicators analyzed to assess the likelihood that a driver's driving behavior or habits will cause a specific accident.

[0744] "Emotional data" refers to information obtained from methods such as facial recognition and voice analysis to understand the emotional state of a driver while they are driving.

[0745] "Virtual reality accident experience" is a technology that allows drivers to experience accident scenarios in a virtual space, providing an experience that simulates real-world accident situations.

[0746] "Feedback" refers to providing drivers with specific advice and information based on data obtained through virtual reality accident simulations, which helps them improve their driving skills and habits.

[0747] A "dashboard" is an interface that visually displays the driver's progress and emotional state over time, enabling drivers and managers to monitor progress and provide guidance.

[0748] This invention is a customized educational system for improving drivers' safe driving skills. The system utilizes the driver's driving data and emotional state to provide virtual reality (VR) experiences and feedback that address specific accident risks.

[0749] The server collects driver data using the vehicle's onboard sensor system. This sensor system includes vehicle speed sensors, accelerator and brake position sensors, and an odometer. The data is transmitted to the server and recorded in a database. Algorithms are used to analyze the data and calculate each driver's accident risk. Furthermore, an emotion engine is used to analyze the driver's facial expressions and voice through in-car cameras and microphones to assess their emotional state.

[0750] Based on the analysis results, the server generates a customized VR accident experience tailored to the driver's risk characteristics and emotional state. It utilizes a generation AI model and prompts to generate scenarios. For example, a prompt such as "Generate a scenario that simulates a rear-end collision under high stress levels" might be used. This scenario is encoded in VR playback format and sent to the terminal.

[0751] The device delivers the experience to the user via a VR headset. The VR headset integrates the user's field of view and plays real-time visual and audio content tailored to the scenario. Users can experience realistic accident situations within virtual reality.

[0752] Data collected during the user's experience is sent back to the server. The server analyzes the experience data and generates feedback that takes into account the driver's risk characteristics and emotional state. This feedback includes specific advice needed to improve driving habits. Users and administrators can use the server-provided dashboard to visualize the driver's progress and plan more effective guidance and training.

[0753] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0754] Step 1:

[0755] The server collects driving status data from the vehicle. Inputs include vehicle speed, accelerator and brake operation, and mileage. This data is acquired in real time from sensors mounted on the vehicle. Specifically, a data collection module on the server receives this data and records it in a database. The output is the recorded driving status data.

[0756] Step 2:

[0757] The server analyzes the collected driving data to calculate accident risk. The input is the driving data obtained in step 1. Using a data analysis algorithm, it calculates the number of sudden braking incidents and the frequency of speeding, and evaluates the driver's driving characteristics. The output is accident risk assessment data as a result of the analysis.

[0758] Step 3:

[0759] The server analyzes the driver's emotional data. The input consists of the driver's facial expressions and voice, captured via in-car cameras and microphones. An emotion engine analyzes this data to assess the driver's stress level and decreased attention span. Specifically, it uses image recognition and voice analysis technologies to quantify the emotional state. The output is the driver's emotional state data.

[0760] Step 4:

[0761] The server generates a customized VR accident experience based on the user's emotional state. The inputs are accident risk assessment data from step 2 and emotional state data from step 3. A generation AI model is used to create a VR scenario based on prompts. A specific prompt might be, "Generate a scenario that simulates a rear-end collision while the user is experiencing high stress levels." The output is VR data.

[0762] Step 5:

[0763] The terminal receives VR data sent from the server and provides it to the user through a VR headset. The input is the VR data generated in step 4. Specifically, the terminal's VR application decodes the data and plays the VR experience through the user's sight and hearing. The output is the user's VR accident experience.

[0764] Step 6:

[0765] The server analyzes the user's experience data and generates feedback. The input is the user's reaction data during the experience in step 5. Specifically, the server analyzes the user's heart rate changes and operation history and creates feedback that includes specific advice for improving driving skills. The output is a feedback report.

[0766] Step 7:

[0767] The server visualizes the driver's progress and provides it to users and administrators through a dashboard. Inputs include feedback reports and emotional state trend data. Specifically, it uses visualization tools to display progress graphs and improvement indicators on the dashboard. The output is the visualized progress.

[0768] (Application Example 2)

[0769] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0770] In autonomous driving systems, there is a lack of opportunities to effectively evaluate and improve the driving skills and situational awareness capabilities of both the driver and the system. If this challenge is not addressed, autonomous driving systems may lack the ability to respond safely and appropriately to unexpected situations. Therefore, it is necessary to provide new methods to enhance skill improvement and risk avoidance capabilities by offering real-time, situation-sensitive feedback to both the autonomous driving system and the driver.

[0771] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0772] In this invention, the server includes means for acquiring driver's driving state data and emotional state; means for analyzing the acquired driving state data and emotional state data to calculate the traffic risk for each driver; and means for generating a virtual reality experience that takes into account the driver's emotional state and the vehicle's condition. This allows the driver and the autonomous driving system to evaluate their situational awareness and reaction capabilities in real time within the virtual environment and optimize them as needed.

[0773] "Driving status data" refers to information recorded by the driver or the autonomous driving system during driving, such as vehicle speed, accelerator and brake operation history, mileage, and driving time.

[0774] "Emotional state" refers to data indicating the psychological and physiological responses a driver exhibits while driving, such as stress levels and decreased attention span.

[0775] "Traffic risk" is an indicator that shows the degree to which a driver or an autonomous driving system is likely to cause an accident or problem under specific driving conditions.

[0776] A "virtual reality experience" is a simulated environment created to recreate a specific driving scenario, in which a driver or autonomous driving system participates.

[0777] "Situational awareness" refers to the ability of an autonomous driving system or driver to accurately perceive the surrounding environment and potential changes while driving, and to make appropriate judgments.

[0778] "Responsiveness" refers to the ability of an autonomous driving system or driver to respond quickly to changes in circumstances.

[0779] To implement this invention, a system is needed that collects driving information from the driver or the autonomous driving system, analyzes it, and provides feedback. The server first acquires the driver's driving state data and emotional state through sensors and cameras mounted on the vehicle. The applicable hardware includes cameras for image analysis and various sensors necessary for real-time data collection. OpenCV or other image processing tools are used as emotion recognition software.

[0780] The server combines and analyzes acquired driving data and sentiment data to generate indicators for evaluating the traffic risk of drivers or autonomous driving systems. This analysis is performed using advanced AI algorithms. Programming languages ​​such as Python and C++ are used to implement these algorithms.

[0781] Next, the server generates a virtual reality experience that reflects the driver's emotional state and the vehicle's condition. VR development platforms such as Unity and Unreal Engine are used for this generation. The driver or autonomous driving system improves its situational awareness and reaction capabilities through scenarios within these virtual environments.

[0782] A concrete example would be testing the response of an autonomous driving system in a scenario where an obstacle suddenly appears during rainy weather. Feedback based on such scenarios is visually represented using data and analysis results, as it contributes to improving the capabilities of both the driver and the system.

[0783] An example of a prompt using a generative AI model is: "Create a scenario where a parked vehicle appears ahead in poor visibility conditions due to rain. Evaluate the reaction speed and adaptability of the autonomous driving AI." This allows for learning and improvement in a situation that closely resembles real-world conditions.

[0784] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0785] Step 1:

[0786] The server collects data on the driver's driving state and emotional state through sensors and cameras. Inputs include data such as speed detected by vehicle sensors, accelerator and brake operation history, and driving time. In addition, emotional information based on facial expressions and voice is obtained from image data obtained using cameras. The server filters this data and removes unnecessary noise to generate a dataset suitable for analysis.

[0787] Step 2:

[0788] The server analyzes the collected driving state data and emotional state data to assess the driver's traffic risk. The input data used is the driving state data and emotional state data obtained in Step 1. The analysis utilizes an AI algorithm to calculate a risk score based on the number of sudden braking incidents and the frequency of speeding. The resulting risk assessment information is used in the subsequent generation of virtual reality experiences.

[0789] Step 3:

[0790] The server generates a virtual reality experience based on the driver's emotional state and risk assessment. The inputs here are the risk assessment information and emotional state data generated in step 2. Specific driving scenarios are dynamically created using VR development platforms such as Unity or Unreal Engine. The output is the VR scenario presented to the driver.

[0791] Step 4:

[0792] The user (driver or autonomous driving system) participates in a generated virtual reality experience and evaluates their situational awareness and reaction capabilities within it. Using a VR headset, the scenario is experienced visually and experientially. The output consists of the user's experience and behavioral data within the virtual environment.

[0793] Step 5:

[0794] The server records user behavior data within the VR scenario and generates feedback that contributes to improving the driver's skills. The input is the behavior data collected in step 4. The server analyzes this data and generates feedback that includes skills the driver needs to improve and areas for system optimization. The output is visualized feedback information for the driver and manager.

[0795] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0796] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0797] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0798] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0799] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. In the upper and lower directions of the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. Also, the upper side of the concentric circles is where "pleasant" emotions are located, and the lower side is where "unpleasant" emotions are located. In this way, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0800] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0801] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0802] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0803] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0804] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0805] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0806] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0807] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0808] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0809] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0810] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0811] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0812] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0813] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0814] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0815] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0816] The following is further disclosed regarding the embodiments described above.

[0817] (Claim 1)

[0818] [Means for acquiring driver's driving status data,

[0819] [A means of analyzing acquired driving condition data to calculate the accident risk for individual drivers,

[0820] [Means for generating a customized virtual reality accident experience for a specific driver,

[0821] [Means that enable drivers to participate in virtual reality accident experiences,

[0822] [Means for recording data based on a virtual reality accident experience in which the driver participated,

[0823] [Means for analyzing recorded data to visualize the driver's progress,

[0824] A system that includes means of presenting challenges for safe driving based on the driver's progress.

[0825] (Claim 2)

[0826] [The system according to claim 1, which provides specific feedback to reinforce the driver's weaknesses based on a generated virtual reality accident experience.

[0827] (Claim 3)

[0828] [The system according to claim 1, which provides a dashboard that visualizes driver progress data, enabling drivers and managers to check progress and provide guidance through the dashboard.

[0829] "Example 1"

[0830] (Claim 1)

[0831] [Means for monitoring the driver's driving condition and collecting data,

[0832] [Means for analyzing collected driving data and evaluating the accident risk of individual drivers,

[0833] [Methods for generating a virtual reality environment adapted according to the driver's characteristics, using a generative AI model,

[0834] [Means to enable drivers to experience generated virtual reality accident scenarios,

[0835] [Means for recording the driver's reactions and actions during a virtual reality experience,

[0836] [Means for analyzing recorded behavioral data and visually displaying the driver's progress,

[0837] A system that includes means for presenting tasks to improve safe driving skills based on the driver's progress and driving behavior.

[0838] (Claim 2)

[0839] [The system according to claim 1, which identifies areas for improvement in a driver's driving skills through generated virtual reality accident scenarios and provides specific instructional feedback based on those areas.]

[0840] (Claim 3)

[0841] The system according to claim 1, which provides a dashboard that displays the results of driver behavior analysis in a graphical format, enabling drivers and supervisors to understand progress and provide appropriate guidance using the dashboard.

[0842] "Application Example 1"

[0843] (Claim 1)

[0844] [Means for obtaining information on the driver's driving status,

[0845] [Methods for analyzing acquired driving condition information to calculate the accident risk of individual drivers,

[0846] [Means for generating a customized virtual environment accident experience for a specific driver,

[0847] [Means that enable drivers to participate in a virtual accident experience,

[0848] [Means for recording information based on a virtual accident experience in which the driver participated,

[0849] [Means for analyzing recorded information to visualize the driver's progress,

[0850] [Means providing a virtual situation in which the driver can evaluate reaction time and accuracy in operating an automated vehicle,

[0851] A system that includes means for presenting tasks for safe driving based on the driver's progress.

[0852] (Claim 2)

[0853] [The system according to claim 1, which provides specific evaluations to strengthen the driver's weaknesses based on the generated virtual environment accident experience.

[0854] (Claim 3)

[0855] [The system according to claim 1, which provides a display screen that visualizes the driver's progress information, enabling the driver and supervisor to check the progress and provide guidance through the display screen.

[0856] "Example 2 of combining an emotion engine"

[0857] (Claim 1)

[0858] [Means for acquiring driver's driving status data,

[0859] [A means of analyzing acquired driving condition data to calculate the accident risk for individual drivers,

[0860] [Methods for understanding the driver's emotional state by analyzing driver emotional data,

[0861] [Means for generating a customized virtual reality accident experience for a specific driver according to their emotional state,

[0862] [Means that enable drivers to participate in virtual reality accident experiences,

[0863] [Means for recording data based on a virtual reality accident experience in which the driver participated,

[0864] [Means for analyzing recorded data to visualize the driver's progress,

[0865] A system that includes means of presenting challenges for safe driving based on the driver's progress.

[0866] (Claim 2)

[0867] [The system according to claim 1, which provides specific feedback that takes into account the driver's weaknesses and emotional state based on a generated virtual reality accident experience.

[0868] (Claim 3)

[0869] [The system according to claim 1, which provides a dashboard that visualizes the driver's progress data and emotional state changes, enabling the driver and manager to check progress and provide guidance through the dashboard.

[0870] "Application example 2 when combining with an emotional engine"

[0871] (Claim 1)

[0872] [Means for acquiring driver's driving state data and emotional state,

[0873] [A means of analyzing acquired driving state data and emotional data to calculate the traffic risk of individual drivers,

[0874] [Means for generating a virtual reality experience that takes into account the driver's emotional state and the vehicle's condition,

[0875] [Means that enable drivers and automated systems to participate in the experience in a virtual environment,

[0876] [Methods for recording data after participation and visualizing emotional states and reactions,

[0877] A system that includes means to analyze recorded data and present challenges aimed at improving driving skills.

[0878] (Claim 2)

[0879] The system according to claim 1, which provides guidelines for evaluating and optimizing the reaction speed and situational awareness of an automated system based on a generated virtual reality experience.

[0880] (Claim 3)

[0881] [The system according to claim 1, which provides a display device for visualizing driver or automated system progress data, enabling users and administrators to check progress and provide guidance through the display device. [Explanation of Symbols]

[0882] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of acquiring driver's driving status data, A means of analyzing acquired driving condition data to calculate the accident risk for individual drivers, A means for generating a customized virtual reality accident experience for a specific driver, A means to enable drivers to participate in virtual reality accident experiences, A means of recording data based on a virtual reality accident experience in which the driver participated, A means of visualizing the driver's progress by analyzing recorded data, A system that includes means of presenting challenges for safe driving based on the driver's progress.

2. The system according to claim 1, which provides specific feedback to strengthen the driver's weaknesses based on a generated virtual reality accident experience.

3. The system according to claim 1, which provides a dashboard that visualizes driver progress data, enabling drivers and managers to check progress and provide guidance through the dashboard.

Images