system

Abstract

We provide the system. [Solution] A means of collecting and analyzing the latest real-world data, A means for generating automotive simulation data based on collected data, A means of conducting benchmark tests using the generated simulation data, A means of evaluating test results and providing those evaluation results to the user, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] As a problem in the development of autonomous driving technology, the development cost has increased significantly because each automobile manufacturer conducts data generation and testing individually. In addition, since each company's system is tested according to its own standards, there is a lack of standard quality standards across the industry. Therefore, there is a need to establish a highly safe autonomous driving system aiming for zero accidents while improving development efficiency.

Means for Solving the Problems

[0005] This invention provides a system that collects and analyzes the latest real-world data and generates diverse automotive simulation data based on this data. The generated simulation data is executed in a unified benchmark test environment, allowing the performance of each autonomous driving system to be evaluated based on standard criteria. Furthermore, the test results are provided to users and used as feedback to improve each company's systems. This system makes it possible to simultaneously achieve a reduction in development costs and an improvement in safety across the industry.

[0006] "Real-world data" refers to observational data collected from actual situations related to vehicle operation and traffic conditions.

[0007] "Simulation data" refers to data generated on a computer to virtually reproduce driving scenarios for automobiles.

[0008] A "benchmark test" is a testing process to measure and evaluate the performance of an autonomous driving system based on established standard scenarios and criteria.

[0009] "Evaluation results" refer to the analysis and conclusions regarding the performance of the autonomous driving system, based on data obtained from benchmark tests.

[0010] "Feedback" refers to information and advice provided to users based on evaluation results, with the aim of improving the performance of the autonomous driving system. [Brief explanation of the drawing]

[0011] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4]This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0012] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0013] First, let's explain the terminology used in the following explanation.

[0014] In the following embodiments, the labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0015] In the following embodiments, the labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0016] In the following embodiments, the labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0017] In the following embodiments, the labeled communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), and the like.

[0018] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0019] [First Embodiment]

[0020] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0021] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0022] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0023] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0024] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0025] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0026] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0027] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0028] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0029] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0030] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0031] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0032] This invention provides a comprehensive system for the efficient and safe development of autonomous driving technology. This system integrates data collection, simulation data generation, benchmark testing, evaluation, and feedback.

[0033] The server collects real-world data, including traffic conditions and accident data, from various sensors and devices, and stores the relevant information in a database. The collected data is not used directly; the server analyzes it and converts it into a common data format. This analysis process optimizes the data for simulation.

[0034] Furthermore, the server generates detailed simulation data to realize a virtual driving simulation based on the accumulated analysis data. This generation process can include a wide variety of driving scenarios and can be customized to take into account vehicle types and specific driving conditions.

[0035] The generated simulation data is applied to a benchmark test environment prepared on the server. Users can upload their self-developed autonomous driving software to this environment from their terminal and receive performance evaluations based on standardized scenarios. The tests are automatically conducted by the server, and the results are collected.

[0036] The collected test results are analyzed by the server, and each performance metric is identified. These results are provided to the user via the terminal, allowing developers to improve and modify the software based on the evaluation. The server also generates specific guidance for performance improvement as feedback, providing advice to the user.

[0037] As a concrete example, users can use simulation data simulating urban traffic congestion to evaluate how autonomous driving software detects and avoids pedestrians and obstacles. Based on these results, users can tune the software's algorithms to build a safer and more efficient autonomous driving system.

[0038] Thus, by using this system, the development of autonomous driving technology can be standardized, safety can be improved, and development costs and time can be expected to be reduced.

[0039] The following describes the processing flow.

[0040] Step 1:

[0041] The server collects up-to-date real-world data on traffic conditions and accidents from various sensors and external databases. This includes road traffic flow, weather conditions, and pedestrian movement.

[0042] Step 2:

[0043] The server analyzes the collected data, performing noise reduction and imputing missing values. This formats the data for simulation. This data preparation is a crucial step in ensuring the accuracy of the simulation.

[0044] Step 3:

[0045] Based on the analyzed data, the server generates simulation data to realize a detailed car driving simulation. Here, various driving scenarios are considered, and detailed models are built for each region and vehicle type.

[0046] Step 4:

[0047] The user uploads the autonomous driving software to the server via their terminal. This allows the software to access the benchmark test environment prepared on the system, and the system is ready to begin testing.

[0048] Step 5:

[0049] The server uses the generated simulation data to evaluate the performance of the uploaded autonomous driving software based on benchmark test scenarios. The tests proceed automatically, and various performance metrics are measured.

[0050] Step 6:

[0051] The server collects and analyzes the test results. From the data obtained from the tests, the software's performance and areas for improvement are identified.

[0052] Step 7:

[0053] The terminal provides the user with the analysis results from the server. Based on these results, the user has the opportunity to improve the algorithms of the autonomous driving software and create new versions.

[0054] Step 8:

[0055] The server provides users with specific suggestions and guidelines for performance improvement as feedback. Users can use this as a reference to rethink their design in order to improve development efficiency and accuracy.

[0056] (Example 1)

[0057] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0058] In the development of autonomous driving technology, systems that are highly safe and efficient are required. However, integrating the collection of real-world data, the generation of simulation data, the conduct of benchmark tests, and the evaluation and provision of feedback on test results is technically complex and leads to increased development costs and time. The present invention aims to provide a system that streamlines and integrates these development processes, enabling developers to improve autonomous driving technology more efficiently and effectively.

[0059] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0060] In this invention, the server includes means for collecting real-world environmental information from various detection devices and equipment and storing it in an information resource; means for analyzing the collected data, converting it into a common format, and optimizing it for use in a virtual environment; and means for generating diverse driving scenarios aimed at realizing a virtual environment based on the optimized data. This enables integrated management of the development process of autonomous driving technology and efficient and safe improvement of system performance.

[0061] "Various detection devices and equipment" refers to hardware such as cameras, sensors, and GPS devices used to collect traffic environment and accident data in real time.

[0062] "Real-world environmental information" refers to data about real-world conditions such as traffic conditions, location information, the movement and location of obstacles, and weather.

[0063] "Information resources" refer to databases and storage mechanisms that store collected data and manage it for later analysis and use.

[0064] A "common format" refers to a unified data format that allows for consistent management of diverse data collected from different devices, facilitating subsequent analysis and use.

[0065] A "virtual environment" refers to a simulated driving space built on a computer to reproduce real-world traffic conditions and test driving software.

[0066] A "driving scenario" refers to a specific set of driving conditions used in the evaluation and development of autonomous driving systems, taking into account different road conditions, weather conditions, vehicle behavior, and other factors.

[0067] This invention provides a comprehensive system for efficiently developing autonomous driving technology. Specific embodiments of this system are described below.

[0068] First, the server receives real-world environmental information, such as traffic conditions and accident data, from various detection devices installed in the autonomous vehicle, including cameras, LiDAR sensors, and GPS devices. This information is stored in a database resource, preparing it for the next analysis step.

[0069] Next, the server analyzes the data stored in the information resources and converts it into a common format. This conversion process ensures that data in different formats can be managed consistently, making it easier to use in subsequent virtual environments. Data cleaning algorithms remove noise from the data and extract meaningful information at this stage.

[0070] Based on the analyzed data, the server generates driving scenarios. By utilizing the generation AI model, virtual driving scenarios based on various driving conditions can be created. For example, scenarios that take into account specific weather conditions and road conditions can be created. As a concrete example, it is possible to generate simulation data for urban traffic congestion.

[0071] Users upload autonomous driving software to a virtual environment on a server via their terminal and perform benchmark tests. These tests evaluate the software's performance based on standardized driving scenarios.

[0072] Test results are analyzed and collected by the server, and performance metrics are clearly displayed. Finally, feedback and specific advice generated based on the evaluation results are provided to the user via the terminal, and the user uses this to improve the software.

[0073] As an example of a prompt, inputting the command "Generate scenario data to evaluate a pedestrian detection algorithm in urban traffic congestion" into the generating AI model will generate the relevant simulation data.

[0074] Thus, by implementing the present invention, it is possible to streamline the development process of autonomous driving technology, thereby improving safety and reducing development costs.

[0075] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0076] Step 1:

[0077] The server receives traffic environment and accident data as input from various detection devices and stores it in its information resources. This process collects data in real time from cameras, LiDAR sensors, and GPS devices. The storage of this data in the information resources forms the foundation for subsequent analysis steps.

[0078] Step 2:

[0079] The server analyzes the accumulated data as input and converts it into a common format. In the specific data cleaning process, noise is removed, relevant information is integrated, and the data is processed into a unified format. This conversion makes data from different devices available in a consistent format. Optimized data is then prepared as output.

[0080] Step 3:

[0081] The server takes optimized data as input and generates driving scenarios using a generated AI model. In this process, inherent parameters are considered to simulate diverse driving conditions and situations. The output is provided as driving scenarios that can be used for testing in a virtual environment.

[0082] Step 4:

[0083] The user inputs the autonomous driving software into a virtual environment on the server via a terminal. The server then performs benchmark tests based on pre-generated driving scenarios. Specifically, it evaluates the performance of how the autonomous driving software functions within the virtual environment. The output obtained from this test consists of various performance indicators.

[0084] Step 5:

[0085] The server analyzes the test results as input and extracts each performance metric. This analysis generates specific improvement suggestions to provide to the user. The output consists of evaluation results and feedback based on them, which are provided to the user as advice via the terminal. This process allows the user to utilize the results for software tuning and improvement suggestions.

[0086] (Application Example 1)

[0087] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0088] Developing autonomous driving technology requires simulations and tests using vast amounts of real-world data, but this presents significant challenges in terms of cost and time. Furthermore, there is a lack of means for developers to receive real-time feedback and efficiently improve autonomous driving software.

[0089] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0090] In this invention, the server includes means for collecting and analyzing the latest environmental information, means for generating vehicle simulation information based on the collected information, and means for conducting standardized tests using the generated simulation information. This enables developers to efficiently and safely advance the development of autonomous driving technology.

[0091] "Environmental information" refers to real-world data such as traffic conditions, road conditions, and weather that are necessary for autonomous vehicles to operate.

[0092] "Analysis" refers to statistical or computational processing to break down collected environmental information data and find meaning in it.

[0093] "Simulation information" refers to data representing a hypothetical driving scenario generated based on analyzed environmental information.

[0094] A "standardized test" is a test designed to consistently evaluate the performance of autonomous driving software under similar conditions.

[0095] "Users" refers to developers and researchers who use this system to develop or improve autonomous driving software.

[0096] A "generative AI model" refers to a model formed by algorithms that use machine learning techniques to extract insights from data.

[0097] "Real-time evaluation" refers to a process that allows for immediate review of test results and feedback, enabling immediate action to be taken in the development process.

[0098] The system implementing this invention is configured in which a server and terminals cooperate using network communication. The server analyzes environmental information collected from various sensors and devices and generates a virtual driving scenario based on this information. The generated simulation information is used for standardized tests to evaluate the vehicle's autonomous driving software.

[0099] The terminal's role is to allow users (developers and researchers) to interactively review evaluation results and acquire data to improve autonomous driving software. The terminal receives real-time feedback from the server, providing users with visual and analytical information. The terminal can also utilize generative AI models to gain further insights from environmental information and test results.

[0100] The main technical elements of this system are as follows:

[0101] 1. Data Collection and Analysis: The server collects traffic conditions and surrounding environmental data via various sensors (cameras, LiDAR, etc.). This enables reliable simulations that mimic actual driving conditions.

[0102] 2. Simulation Generation: Based on the analyzed information, driving scenarios are created using simulation software such as Python or SimPy. This allows for pre-testing of various driving conditions and emergency responses.

[0103] 3. Test Implementation and Feedback Generation: Autonomous driving software is evaluated using a standardized test framework, and its performance is stored in a database. Test results are analyzed using NumPy and Pandas, allowing users to receive feedback for improvement.

[0104] As a concrete example, this system can be used to generate simulation data that mimics urban traffic congestion, and the pedestrian and obstacle detection performance of autonomous driving software can be evaluated. Based on the results of this test, the software algorithm can be improved to achieve safer and more efficient autonomous driving. An example of a prompt used in such a system evaluation process would be, "Test a driving algorithm to safely control the vehicle while maintaining a constant distance from pedestrians in an urban traffic congestion scenario, and provide suggestions for improvement."

[0105] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0106] Step 1:

[0107] The server collects environmental information from various sensors and devices. This information includes traffic conditions, road conditions, and weather information. This data is stored in a database for later analysis. The input is raw data from sensors, and the output is integrated environmental information data.

[0108] Step 2:

[0109] The server analyzes the collected environmental information. Using Python, it analyzes traffic conditions and the positional relationships of objects, and processes the data by converting it into a common format. The input is raw environmental information data, and the output is formatted, analyzed data.

[0110] Step 3:

[0111] The server generates simulation information using the analyzed data. It uses simulation software (e.g., SimPy) to create a virtual operating scenario that considers multiple operating conditions. The input is the analyzed data, and the output is the simulation information.

[0112] Step 4:

[0113] The terminal receives the generated simulation information and performs standardized tests. The user specifies the test conditions and uses computing resources on the server to evaluate the autonomous driving software. The input is simulation information, and the output is test result data.

[0114] Step 5:

[0115] The server analyzes test results and generates feedback. It utilizes machine learning models (generative AI models) to extract insights from test results and provide concrete suggestions for performance improvement. The input is test result data, and the output is analysis results and feedback information.

[0116] Step 6:

[0117] The terminal presents the user with feedback from the server. The user then uses this feedback to consider ways to improve the autonomous driving software. This process involves using prompts to further analyze the insights gained from the generated AI model. The input is feedback information, and the output is a set of improvement suggestions formatted in a way that is easy for the user to understand.

[0118] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0119] This invention is a comprehensive system for improving efficiency and safety in the development of autonomous driving technology, and further aims to make the developer experience more personalized and optimized by incorporating an emotion engine that recognizes user emotions.

[0120] The server first collects real-world data and analyzes data such as traffic conditions, accident history, and environmental conditions. This data is processed to enable car driving simulations and converted into simulation datasets. The server then uses a scenario generation engine to create standard and customized driving scenarios.

[0121] Next, the user uploads the autonomous driving software via their terminal and connects to a benchmark test environment on the server. In this environment, the software's performance is tested based on generated scenarios. This provides detailed test results according to standardized evaluation criteria.

[0122] A key feature of this invention is the inclusion of an emotion engine, which allows the terminal to recognize emotions from the user's facial expressions, tone of voice, and operation patterns. This emotion information is fed back to the server, and the method of presenting test results is adjusted according to the user's emotions. For example, if the emotion engine determines that the user is stressed, the server provides clearly organized information and modifies the display procedure to facilitate user comprehension.

[0123] Furthermore, the results of the emotion engine are reflected in the feedback, supplementing it with specific advice and guidance that users need. This makes it easier for users to improve the software and enhances the efficiency and effectiveness of the entire development process.

[0124] For example, when a user tests a scenario in a specific situation (e.g., nighttime driving in an urban area), if the emotion engine recognizes the user's anxiety or tension, the server will simplify the presentation of information and clarify the points to emphasize, thereby reducing the user's workload. In this way, flexible support tailored to the user's emotional state can be provided, further refining the development of autonomous driving technology.

[0125] The following describes the processing flow.

[0126] Step 1:

[0127] The server collects real-world data such as traffic conditions and environmental conditions from various sensors and databases. This includes visibility data during nighttime driving and traffic density information in urban areas.

[0128] Step 2:

[0129] The server analyzes the collected data, removes noise, and converts it into a format optimized for simulation. During this process, missing data is filled in, and standardization is performed to ensure consistency.

[0130] Step 3:

[0131] The server uses the analyzed data to generate specific simulation data for car driving. This simulation data includes multiple driving scenarios, creating situations based on specific conditions and locations.

[0132] Step 4:

[0133] The user uploads the autonomous driving software to the server using their device. This operation allows the software to be evaluated in a benchmark test environment.

[0134] Step 5:

[0135] The server activates the emotion engine and collects facial expressions and voice data from the user's device via the camera and microphone. This prepares the system to recognize the user's emotional state in real time.

[0136] Step 6:

[0137] The server performs benchmark tests using simulation data and meticulously records the results. This includes data on whether the software handled the conditions well.

[0138] Step 7:

[0139] The server analyzes the test results and evaluates each performance metric. In this process, the emotion engine adjusts how the test results are presented based on the user's emotions as perceived.

[0140] Step 8:

[0141] The device displays adjusted test results to the user. The display includes highlights and diagrams to make it easy for the user to understand.

[0142] Step 9:

[0143] The emotion engine integrates the analysis results into feedback and generates specific improvement suggestions and advice on the server.

[0144] Step 10:

[0145] Users receive feedback on their devices, which is then used to improve the autonomous driving software. This ensures an efficient and appropriate development process.

[0146] (Example 2)

[0147] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0148] In developing autonomous driving technology, it is essential to utilize real-world data and conduct efficient and safe evaluations. Furthermore, providing appropriate feedback and information tailored to the user's emotional state is also crucial. Current technology struggles to fully meet these requirements, particularly in terms of system design that considers user emotions.

[0149] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0150] In this invention, the server includes means for collecting and analyzing the latest real-world information, means for generating simulated data for transportation equipment based on the collected information, and means for conducting standard tests using the generated simulated data. This makes it possible to recognize the user's emotions and provide information based on those emotions.

[0151] "Real-world information" refers to data about events and conditions that occur in the real world, including traffic conditions, accident history, and environmental conditions.

[0152] "Simulated data for transportation equipment" refers to simulation data used to reproduce the operation of transportation equipment based on collected real-world information.

[0153] A "reference test" is a standardized test method conducted to evaluate the performance of developed software or systems using simulated data for transportation equipment.

[0154] "Means of recognizing emotions" refers to technologies that analyze data such as the user's facial expressions, tone of voice, and operation patterns to identify their emotional state.

[0155] "Means of adjusting the method of information delivery" refers to technologies that optimize the way information is presented according to the user's emotions, aiming to provide information in an easy-to-understand and effective manner.

[0156] This invention is a system for supporting the efficient development of autonomous driving technology, and is configured as follows.

[0157] The server first collects the latest real-world information. This information includes traffic conditions, accident history, and environmental conditions. This collection is carried out using sensing technology, and the data is aggregated on the server. Next, the server generates simulated data for transportation equipment based on this real-world information. This simulated data is generated using a simulation execution engine and is used as the basis data for driving simulations.

[0158] The user uploads autonomous driving software to the server via a terminal. The server runs the software sent from the terminal in a standard test environment. This test is conducted based on generated simulated data and adheres to performance evaluation criteria. The terminal also has an emotion recognition function that analyzes the user's emotions in real time. This emotion data is sent to the server, which adjusts the way the test results are presented based on the user's emotions. This allows the user to receive evaluation results in an easy-to-understand format.

[0159] For example, if a user conducts a test simulating nighttime driving in an urban area, and the server detects tension from the user's tone of voice and facial expression, it will organize the information clearly and highlight key points. In this way, information is provided according to the user's emotional state, which can improve the quality of the development process.

[0160] An example of a prompt used in a generative AI model is the input, "Please explain in detail how to use emotion recognition technology to present test results of autonomous driving software in a way that matches the user's emotions."

[0161] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0162] Step 1:

[0163] The server automatically collects real-world information such as traffic conditions, accident history, and environmental conditions from external sensor systems. This data is stored in storage. As input, it receives data from multiple data sources (e.g., cameras, radar, traffic APIs), performs data cleaning, and converts it into well-formed data. As output, it generates an analyzable dataset suitable for driving simulations.

[0164] Step 2:

[0165] The server uses this dataset to generate simulated data for transportation equipment. This process utilizes a simulation execution engine to set simulation parameters and build scenarios. It receives the dataset generated in step 1 as input and uses a scenario generation algorithm to construct standard and customized simulated data. The output is a detailed operating scenario, resulting in a complete dataset for simulation.

[0166] Step 3:

[0167] The user uploads autonomous driving software from their device to the server. This is done through a user-friendly UI. The server runs the uploaded software within a standard test framework. The input is a software package sent by the user; the server decompresses this software and runs it in the test environment. The output generates software operation logs and performance metrics, which are used for evaluation in the next step.

[0168] Step 4:

[0169] The device recognizes emotions through the user's facial expressions, tone of voice, and operation patterns. This uses camera and microphone input, and an emotion recognition algorithm is executed in real time. As input, the user's real-time data is input to the device, and as output, an index indicating the user's emotional state is calculated and sent to the server.

[0170] Step 5:

[0171] The server adjusts how test results are presented based on the user's emotional state. It analyzes emotional data and summarizes and emphasizes the information presented. The server requires the emotional data obtained in step 4 and the test results from step 3 as input, and applies an optimization algorithm for presentation. As output, feedback and advice that takes the user's emotions into account are generated and presented in a way that is easy for the user to understand.

[0172] (Application Example 2)

[0173] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0174] In the development of autonomous driving technology, there is a need to improve development efficiency and quality by effectively utilizing real-world data while providing feedback that takes into account the emotional state of the developers. Conventional systems have not made adjustments that reflect such individualized emotions, which has sometimes increased the workload of developers.

[0175] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0176] In this invention, the server includes means for collecting and analyzing the latest real-world information, means for generating vehicle simulation information based on the collected information, means for conducting standard tests using the generated simulation information, means for evaluating the test results and providing the evaluation results to the user, and means for generating emotional information using an emotional recognition engine that recognizes the user's emotions and adjusting the information presentation means based on that emotional information. This makes it possible to adjust information according to the user's emotional state, providing personalized support and improving the efficiency of the entire development process.

[0177] "Latest real-world information" refers to various dynamic data that occurs in the real world, including traffic conditions, accident history, and environmental conditions.

[0178] "Vehicle simulation information" refers to a set of data generated based on real-world information to simulate the operation of autonomous vehicles in a virtual environment.

[0179] A "standard test" is a standardized test conducted using vehicle simulation data to evaluate the performance of autonomous driving technology.

[0180] An "emotion recognition engine" is a computer system that analyzes emotions from a user's facial expressions, voice, and operation patterns.

[0181] "Emotional information" refers to data that indicates the user's current emotional state, based on the results of analysis by an emotion recognition engine.

[0182] In the system that realizes this invention, a server first collects the latest real-world information such as traffic conditions and environmental conditions, and generates vehicle simulation information by analyzing it. Based on the generated simulation information, the server conducts standard tests and evaluates the results. The evaluated test results are provided to the user and displayed on the user's terminal.

[0183] The emotion recognition engine analyzes the user's facial expressions, voice, and operation patterns to generate emotional information. Based on this emotional information, the server adjusts how information is provided to the user. For example, if the user is stressed, the information presented is made more concise and key points are highlighted. Conversely, if the user is relaxed, more detailed information is provided.

[0184] The server and terminal systems utilize emotion recognition APIs such as "Microsoft® Azure® Emotion API" and "Google® Cloud Vision AI" to analyze user emotions in real time. Additionally, "Google Maps API" and other services are used to provide appropriate assistance and guidance for navigation.

[0185] For example, if a user is traveling long distances in an autonomous vehicle and fatigue is detected from their facial expression, the emotion recognition engine will suggest a break and display the route to the nearest rest stop on the device. Furthermore, for users relaxing in their car on their way home from work, the system will support safer travel by displaying more detailed traffic conditions than usual.

[0186] Examples of prompts for a generative AI model are as follows:

[0187] "Estimate the user's emotions from their facial expressions and voice, and decide what information to provide based on their driving situation. If the user is stressed, provide concise information; if they are tired, encourage them to take a break; and if they are relaxed, provide detailed route guidance."

[0188] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0189] Step 1:

[0190] The server collects the latest real-world information from various sensors and databases. Input data includes traffic conditions, accident history, and environmental conditions. Based on this input, data analysis algorithms are used to process the information and convert it into vehicle simulation information. The output is this vehicle simulation information.

[0191] Step 2:

[0192] The server performs baseline tests using the generated vehicle simulation information. The tests simulate autonomous driving behavior in a virtual environment and evaluate its performance. The input is vehicle simulation information, and the test outputs evaluation data as a result.

[0193] Step 3:

[0194] The user's device activates an emotion recognition engine and captures the user's facial expressions and voice using the camera and microphone. This captured information is sent as input to an emotion recognition API for data analysis. The output is emotion information indicating the user's emotional state.

[0195] Step 4:

[0196] The server receives emotional information and adjusts how test results and driving assistance information are presented based on it. For example, if it determines that the user is stressed, it processes the information to be concise and emphasize key points. The input is emotional information and test results, and the output is information tailored to the user.

[0197] Step 5:

[0198] The user receives pre-configured information on their device and uses it to monitor the autonomous vehicle's operation or select rest stops. The input is pre-configured information from the server, and the output is the user's next selected action.

[0199] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0200] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0201] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0202] [Second Embodiment]

[0203] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0204] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0205] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0206] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0207] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0208] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0209] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0210] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0211] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0212] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0213] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0214] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0215] This invention provides a comprehensive system for the efficient and safe development of autonomous driving technology. This system integrates data collection, simulation data generation, benchmark testing, evaluation, and feedback.

[0216] The server collects real-world data, including traffic conditions and accident data, from various sensors and devices, and stores the relevant information in a database. The collected data is not used directly; the server analyzes it and converts it into a common data format. This analysis process optimizes the data for simulation.

[0217] Furthermore, the server generates detailed simulation data to realize a virtual driving simulation based on the accumulated analysis data. This generation process can include a wide variety of driving scenarios and can be customized to take into account vehicle types and specific driving conditions.

[0218] The generated simulation data is applied to a benchmark test environment prepared on the server. Users can upload their self-developed autonomous driving software to this environment from their terminal and receive performance evaluations based on standardized scenarios. The tests are automatically conducted by the server, and the results are collected.

[0219] The collected test results are analyzed by the server, and each performance metric is identified. These results are provided to the user via the terminal, allowing developers to improve and modify the software based on the evaluation. The server also generates specific guidance for performance improvement as feedback, providing advice to the user.

[0220] As a concrete example, users can use simulation data simulating urban traffic congestion to evaluate how autonomous driving software detects and avoids pedestrians and obstacles. Based on these results, users can tune the software's algorithms to build a safer and more efficient autonomous driving system.

[0221] Thus, by using this system, the development of autonomous driving technology can be standardized, safety can be improved, and development costs and time can be expected to be reduced.

[0222] The following describes the processing flow.

[0223] Step 1:

[0224] The server collects up-to-date real-world data on traffic conditions and accidents from various sensors and external databases. This includes road traffic flow, weather conditions, and pedestrian movement.

[0225] Step 2:

[0226] The server analyzes the collected data, performing noise reduction and imputing missing values. This formats the data for simulation. This data preparation is a crucial step in ensuring the accuracy of the simulation.

[0227] Step 3:

[0228] Based on the analyzed data, the server generates simulation data to realize a detailed car driving simulation. Here, various driving scenarios are considered, and detailed models are built for each region and vehicle type.

[0229] Step 4:

[0230] The user uploads the autonomous driving software to the server via their terminal. This allows the software to access the benchmark test environment prepared on the system, and the system is ready to begin testing.

[0231] Step 5:

[0232] The server uses the generated simulation data to evaluate the performance of the uploaded autonomous driving software based on benchmark test scenarios. The tests proceed automatically, and various performance metrics are measured.

[0233] Step 6:

[0234] The server collects and analyzes the test results. From the data obtained from the tests, the software's performance and areas for improvement are identified.

[0235] Step 7:

[0236] The terminal provides the user with the analysis results from the server. Based on these results, the user has the opportunity to improve the algorithms of the autonomous driving software and create new versions.

[0237] Step 8:

[0238] The server provides users with specific suggestions and guidelines for performance improvement as feedback. Users can use this as a reference to rethink their design in order to improve development efficiency and accuracy.

[0239] (Example 1)

[0240] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0241] In the development of autonomous driving technology, systems that are highly safe and efficient are required. However, integrating the collection of real-world data, the generation of simulation data, the conduct of benchmark tests, and the evaluation and provision of feedback on test results is technically complex and leads to increased development costs and time. The present invention aims to provide a system that streamlines and integrates these development processes, enabling developers to improve autonomous driving technology more efficiently and effectively.

[0242] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0243] In this invention, the server includes means for collecting real-world environmental information from various detection devices and equipment and storing it in an information resource; means for analyzing the collected data, converting it into a common format, and optimizing it for use in a virtual environment; and means for generating diverse driving scenarios aimed at realizing a virtual environment based on the optimized data. This enables integrated management of the development process of autonomous driving technology and efficient and safe improvement of system performance.

[0244] "Various detection devices and equipment" refers to hardware such as cameras, sensors, and GPS devices used to collect traffic environment and accident data in real time.

[0245] "Real-world environmental information" refers to data about real-world conditions such as traffic conditions, location information, the movement and location of obstacles, and weather.

[0246] "Information resources" refer to databases and storage mechanisms that store collected data and manage it for later analysis and use.

[0247] A "common format" refers to a unified data format that allows for consistent management of diverse data collected from different devices, facilitating subsequent analysis and use.

[0248] A "virtual environment" refers to a simulated driving space built on a computer to reproduce real-world traffic conditions and test driving software.

[0249] A "driving scenario" refers to a specific set of driving conditions used in the evaluation and development of autonomous driving systems, taking into account different road conditions, weather conditions, vehicle behavior, and other factors.

[0250] This invention provides a comprehensive system for efficiently developing autonomous driving technology. Specific embodiments of this system are described below.

[0251] First, the server receives real-world environmental information, such as traffic conditions and accident data, from various detection devices installed in the autonomous vehicle, including cameras, LiDAR sensors, and GPS devices. This information is stored in a database resource, preparing it for the next analysis step.

[0252] Next, the server analyzes the data stored in the information resources and converts it into a common format. This conversion process ensures that data in different formats can be managed consistently, making it easier to use in subsequent virtual environments. Data cleaning algorithms remove noise from the data and extract meaningful information at this stage.

[0253] Based on the analyzed data, the server generates driving scenarios. By utilizing the generation AI model, virtual driving scenarios based on various driving conditions can be created. For example, scenarios that take into account specific weather conditions and road conditions can be created. As a concrete example, it is possible to generate simulation data for urban traffic congestion.

[0254] Users upload autonomous driving software to a virtual environment on a server via their terminal and perform benchmark tests. These tests evaluate the software's performance based on standardized driving scenarios.

[0255] Test results are analyzed and collected by the server, and performance metrics are clearly displayed. Finally, feedback and specific advice generated based on the evaluation results are provided to the user via the terminal, and the user uses this to improve the software.

[0256] As an example of a prompt, inputting the command "Generate scenario data to evaluate a pedestrian detection algorithm in urban traffic congestion" into the generating AI model will generate the relevant simulation data.

[0257] Thus, by implementing the present invention, it is possible to streamline the development process of autonomous driving technology, thereby improving safety and reducing development costs.

[0258] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0259] Step 1:

[0260] The server receives traffic environment and accident data as input from various detection devices and stores it in its information resources. This process collects data in real time from cameras, LiDAR sensors, and GPS devices. The storage of this data in the information resources forms the foundation for subsequent analysis steps.

[0261] Step 2:

[0262] The server analyzes the accumulated data as input and converts it into a common format. In the specific data cleaning process, noise is removed, relevant information is integrated, and the data is processed into a unified format. This conversion makes data from different devices available in a consistent format. Optimized data is then prepared as output.

[0263] Step 3:

[0264] The server takes optimized data as input and generates driving scenarios using a generated AI model. In this process, inherent parameters are considered to simulate diverse driving conditions and situations. The output is provided as driving scenarios that can be used for testing in a virtual environment.

[0265] Step 4:

[0266] The user inputs the autonomous driving software into a virtual environment on the server via a terminal. The server then performs benchmark tests based on pre-generated driving scenarios. Specifically, it evaluates the performance of how the autonomous driving software functions within the virtual environment. The output obtained from this test consists of various performance indicators.

[0267] Step 5:

[0268] The server analyzes the test results as input and extracts each performance metric. This analysis generates specific improvement suggestions to provide to the user. The output consists of evaluation results and feedback based on them, which are provided to the user as advice via the terminal. This process allows the user to utilize the results for software tuning and improvement suggestions.

[0269] (Application Example 1)

[0270] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0271] Developing autonomous driving technology requires simulations and tests using vast amounts of real-world data, but this presents significant challenges in terms of cost and time. Furthermore, there is a lack of means for developers to receive real-time feedback and efficiently improve autonomous driving software.

[0272] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0273] In this invention, the server includes means for collecting and analyzing the latest environmental information, means for generating vehicle simulation information based on the collected information, and means for conducting standardized tests using the generated simulation information. This enables developers to efficiently and safely advance the development of autonomous driving technology.

[0274] "Environmental information" refers to real-world data such as traffic conditions, road conditions, and weather that are necessary for autonomous vehicles to operate.

[0275] "Analysis" refers to statistical or computational processing to break down collected environmental information data and find meaning in it.

[0276] "Simulation information" refers to data representing a hypothetical driving scenario generated based on analyzed environmental information.

[0277] A "standardized test" is a test designed to consistently evaluate the performance of autonomous driving software under similar conditions.

[0278] "Users" refers to developers and researchers who use this system to develop or improve autonomous driving software.

[0279] A "generative AI model" refers to a model formed by algorithms that use machine learning techniques to extract insights from data.

[0280] "Real-time evaluation" refers to a process that allows for immediate review of test results and feedback, enabling immediate action to be taken in the development process.

[0281] The system implementing this invention is configured in which a server and terminals cooperate using network communication. The server analyzes environmental information collected from various sensors and devices and generates a virtual driving scenario based on this information. The generated simulation information is used for standardized tests to evaluate the vehicle's autonomous driving software.

[0282] The role of the terminal is for the user (developer or researcher) to interactively check the evaluation results and obtain data for improving the autonomous driving software. The terminal receives feedback from the server in real time and provides visual and analytical information to the user. The terminal can also use the generative AI model to gain further insights from environmental information and test results.

[0283] The main technical elements of this system are as follows:

[0284] 1. Data collection and analysis: The server collects traffic conditions and surrounding environmental data via various sensors (such as cameras, LiDAR, etc.). This enables a highly reliable simulation that mimics actual driving conditions.

[0285] 2. Simulation generation: Based on the analyzed information, driving scenarios are created using simulation software such as Python and SimPy. This allows various driving conditions and emergency responses to be tested in advance.

[0286] 3. Test execution and feedback generation: The autonomous driving software is evaluated using a standardized test framework, and the performance of the software is accumulated in a database. The test results are analyzed using NumPy and Pandas, and the user can obtain feedback for improvement.

[0287] As a specific example, simulation data simulating urban traffic congestion can be generated using this system, and the pedestrian and obstacle detection performance of the autonomous driving software can be evaluated. Based on the results of this test, the software algorithm can be improved to achieve safer and more efficient autonomous driving. An example of a prompt sentence used in the evaluation process of such a system is: "Please test the driving algorithm for safe control while maintaining a constant distance from pedestrians in an urban traffic congestion scenario and present the improvement points."

[0288] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0289] Step 1:

[0290] The server collects environmental information from various sensors and devices. This information includes traffic conditions, road conditions, and weather information. This data is stored in a database for later analysis. The input is raw data from sensors, and the output is integrated environmental information data.

[0291] Step 2:

[0292] The server analyzes the collected environmental information. Using Python, it analyzes traffic conditions and the positional relationships of objects, and processes the data by converting it into a common format. The input is raw environmental information data, and the output is formatted, analyzed data.

[0293] Step 3:

[0294] The server generates simulation information using the analyzed data. It uses simulation software (e.g., SimPy) to create a virtual operating scenario that considers multiple operating conditions. The input is the analyzed data, and the output is the simulation information.

[0295] Step 4:

[0296] The terminal receives the generated simulation information and performs standardized tests. The user specifies the test conditions and uses computing resources on the server to evaluate the autonomous driving software. The input is simulation information, and the output is test result data.

[0297] Step 5:

[0298] The server analyzes test results and generates feedback. It utilizes machine learning models (generative AI models) to extract insights from test results and provide concrete suggestions for performance improvement. The input is test result data, and the output is analysis results and feedback information.

[0299] Step 6:

[0300] The terminal presents the user with feedback from the server. The user then uses this feedback to consider ways to improve the autonomous driving software. This process involves using prompts to further analyze the insights gained from the generated AI model. The input is feedback information, and the output is a set of improvement suggestions formatted in a way that is easy for the user to understand.

[0301] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0302] This invention is a comprehensive system for improving efficiency and safety in the development of autonomous driving technology, and further aims to make the developer experience more personalized and optimized by incorporating an emotion engine that recognizes user emotions.

[0303] The server first collects real-world data and analyzes data such as traffic conditions, accident history, and environmental conditions. This data is processed to enable car driving simulations and converted into simulation datasets. The server then uses a scenario generation engine to create standard and customized driving scenarios.

[0304] Next, the user uploads the autonomous driving software via their terminal and connects to a benchmark test environment on the server. In this environment, the software's performance is tested based on generated scenarios. This provides detailed test results according to standardized evaluation criteria.

[0305] As a feature of the present invention, an emotion engine is installed, and the terminal recognizes emotions from the user's expression, voice tone, operation pattern, etc. This emotion information is fed back to the server, and the presentation method of the test results is adjusted according to the user's emotion. For example, when the emotion engine determines that the user is in a stressed state, the server provides information that is neatly organized and changes the display procedure to facilitate the user's understanding.

[0306] Also, the result of the emotion engine is reflected in the feedback content, and specific advice and guidelines required by the user are supplemented. This makes it easier for the user to promote software improvement and enables improvement of the efficiency and effectiveness in the entire development process.

[0307] As a specific example, when the user tests a scenario in a specific situation (for example, night driving in the city) and the emotion engine recognizes the user's uneasiness or tension, the server simplifies the presentation of information and clarifies the points to be emphasized, thereby devising to reduce the user's workload. In this way, flexible support according to the user's emotional state can be provided, and the development of autonomous driving technology can be further refined.

[0308] The following describes the processing flow.

[0309] Step 1:

[0310] The server collects real-world data such as traffic conditions and environmental conditions from various sensors and databases. This includes visibility data during night driving and traffic density information in urban areas.

[0311] Step 2:

[0312] The server analyzes the collected data, removes noise, and converts it into an optimal format for simulation. In this process, standardization is performed to complement data deficiencies and make it consistent.

[0313] Step 3:

[0314] The server uses the analyzed data to generate specific simulation data for car driving. This simulation data includes multiple driving scenarios, creating situations based on specific conditions and locations.

[0315] Step 4:

[0316] The user uploads the autonomous driving software to the server using their device. This operation allows the software to be evaluated in a benchmark test environment.

[0317] Step 5:

[0318] The server activates the emotion engine and collects facial expressions and voice data from the user's device via the camera and microphone. This prepares the system to recognize the user's emotional state in real time.

[0319] Step 6:

[0320] The server performs benchmark tests using simulation data and meticulously records the results. This includes data on whether the software handled the conditions well.

[0321] Step 7:

[0322] The server analyzes the test results and evaluates each performance metric. In this process, the emotion engine adjusts how the test results are presented based on the user's emotions as perceived.

[0323] Step 8:

[0324] The device displays adjusted test results to the user. The display includes highlights and diagrams to make it easy for the user to understand.

[0325] Step 9:

[0326] The emotion engine integrates the analysis results into feedback and generates specific improvement suggestions and advice on the server.

[0327] Step 10:

[0328] Users receive feedback on their devices, which is then used to improve the autonomous driving software. This ensures an efficient and appropriate development process.

[0329] (Example 2)

[0330] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0331] In developing autonomous driving technology, it is essential to utilize real-world data and conduct efficient and safe evaluations. Furthermore, providing appropriate feedback and information tailored to the user's emotional state is also crucial. Current technology struggles to fully meet these requirements, particularly in terms of system design that considers user emotions.

[0332] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0333] In this invention, the server includes means for collecting and analyzing the latest real-world information, means for generating simulated data for transportation equipment based on the collected information, and means for conducting standard tests using the generated simulated data. This makes it possible to recognize the user's emotions and provide information based on those emotions.

[0334] "Real-world information" refers to data about events and conditions that occur in the real world, including traffic conditions, accident history, and environmental conditions.

[0335] "Simulated data for transportation equipment" refers to simulation data used to reproduce the operation of transportation equipment based on collected real-world information.

[0336] A "reference test" is a standardized test method conducted to evaluate the performance of developed software or systems using simulated data for transportation equipment.

[0337] "Means of recognizing emotions" refers to technologies that analyze data such as the user's facial expressions, tone of voice, and operation patterns to identify their emotional state.

[0338] "Means of adjusting the method of information delivery" refers to technologies that optimize the way information is presented according to the user's emotions, aiming to provide information in an easy-to-understand and effective manner.

[0339] This invention is a system for supporting the efficient development of autonomous driving technology, and is configured as follows.

[0340] The server first collects the latest real-world information. This information includes traffic conditions, accident history, and environmental conditions. This collection is carried out using sensing technology, and the data is aggregated on the server. Next, the server generates simulated data for transportation equipment based on this real-world information. This simulated data is generated using a simulation execution engine and is used as the basis data for driving simulations.

[0341] The user uploads autonomous driving software to the server via a terminal. The server runs the software sent from the terminal in a standard test environment. This test is conducted based on generated simulated data and adheres to performance evaluation criteria. The terminal also has an emotion recognition function that analyzes the user's emotions in real time. This emotion data is sent to the server, which adjusts the way the test results are presented based on the user's emotions. This allows the user to receive evaluation results in an easy-to-understand format.

[0342] For example, if a user conducts a test simulating nighttime driving in an urban area, and the server detects tension from the user's tone of voice and facial expression, it will organize the information clearly and highlight key points. In this way, information is provided according to the user's emotional state, which can improve the quality of the development process.

[0343] An example of a prompt used in a generative AI model is the input, "Please explain in detail how to use emotion recognition technology to present test results of autonomous driving software in a way that matches the user's emotions."

[0344] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0345] Step 1:

[0346] The server automatically collects real-world information such as traffic conditions, accident history, and environmental conditions from external sensor systems. This data is stored in storage. As input, it receives data from multiple data sources (e.g., cameras, radar, traffic APIs), performs data cleaning, and converts it into well-formed data. As output, it generates an analyzable dataset suitable for driving simulations.

[0347] Step 2:

[0348] The server uses this dataset to generate simulated data for transportation equipment. This process utilizes a simulation execution engine to set simulation parameters and build scenarios. It receives the dataset generated in step 1 as input and uses a scenario generation algorithm to construct standard and customized simulated data. The output is a detailed operating scenario, resulting in a complete dataset for simulation.

[0349] Step 3:

[0350] The user uploads autonomous driving software from their device to the server. This is done through a user-friendly UI. The server runs the uploaded software within a standard test framework. The input is a software package sent by the user; the server decompresses this software and runs it in the test environment. The output generates software operation logs and performance metrics, which are used for evaluation in the next step.

[0351] Step 4:

[0352] The device recognizes emotions through the user's facial expressions, tone of voice, and operation patterns. This uses camera and microphone input, and an emotion recognition algorithm is executed in real time. As input, the user's real-time data is input to the device, and as output, an index indicating the user's emotional state is calculated and sent to the server.

[0353] Step 5:

[0354] The server adjusts how test results are presented based on the user's emotional state. It analyzes emotional data and summarizes and emphasizes the information presented. The server requires the emotional data obtained in step 4 and the test results from step 3 as input, and applies an optimization algorithm for presentation. As output, feedback and advice that takes the user's emotions into account are generated and presented in a way that is easy for the user to understand.

[0355] (Application Example 2)

[0356] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0357] In the development of autonomous driving technology, there is a need to improve development efficiency and quality by effectively utilizing real-world data while providing feedback that takes into account the emotional state of the developers. Conventional systems have not made adjustments that reflect such individualized emotions, which has sometimes increased the workload of developers.

[0358] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0359] In this invention, the server includes means for collecting and analyzing the latest real-world information, means for generating vehicle simulation information based on the collected information, means for conducting standard tests using the generated simulation information, means for evaluating the test results and providing the evaluation results to the user, and means for generating emotional information using an emotional recognition engine that recognizes the user's emotions and adjusting the information presentation means based on that emotional information. This makes it possible to adjust information according to the user's emotional state, providing personalized support and improving the efficiency of the entire development process.

[0360] "Latest real-world information" refers to various dynamic data that occurs in the real world, including traffic conditions, accident history, and environmental conditions.

[0361] "Vehicle simulation information" refers to a set of data generated based on real-world information to simulate the operation of autonomous vehicles in a virtual environment.

[0362] A "standard test" is a standardized test conducted using vehicle simulation data to evaluate the performance of autonomous driving technology.

[0363] An "emotion recognition engine" is a computer system that analyzes emotions from a user's facial expressions, voice, and operation patterns.

[0364] "Emotional information" refers to data that indicates the user's current emotional state, based on the results of analysis by an emotion recognition engine.

[0365] In the system that realizes this invention, a server first collects the latest real-world information such as traffic conditions and environmental conditions, and generates vehicle simulation information by analyzing it. Based on the generated simulation information, the server conducts standard tests and evaluates the results. The evaluated test results are provided to the user and displayed on the user's terminal.

[0366] The emotion recognition engine analyzes the user's facial expressions, voice, and operation patterns to generate emotional information. Based on this emotional information, the server adjusts how information is provided to the user. For example, if the user is stressed, the information presented is made more concise and key points are highlighted. Conversely, if the user is relaxed, more detailed information is provided.

[0367] The server and terminal systems utilize emotion recognition APIs such as "Microsoft Azure's Emotion API" and "Google Cloud's Vision AI" to analyze user emotions in real time. Additionally, "Google Maps API" and other services are used to provide appropriate assistance and guidance for navigation.

[0368] For example, if a user is traveling long distances in an autonomous vehicle and fatigue is detected from their facial expression, the emotion recognition engine will suggest a break and display the route to the nearest rest stop on the device. Furthermore, for users relaxing in their car on their way home from work, the system will support safer travel by displaying more detailed traffic conditions than usual.

[0369] Examples of prompts for a generative AI model are as follows:

[0370] "Estimate the user's emotions from their facial expressions and voice, and decide what information to provide based on their driving situation. If the user is stressed, provide concise information; if they are tired, encourage them to take a break; and if they are relaxed, provide detailed route guidance."

[0371] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0372] Step 1:

[0373] The server collects the latest real-world information from various sensors and databases. Input data includes traffic conditions, accident history, and environmental conditions. Based on this input, data analysis algorithms are used to process the information and convert it into vehicle simulation information. The output is this vehicle simulation information.

[0374] Step 2:

[0375] The server performs baseline tests using the generated vehicle simulation information. The tests simulate autonomous driving behavior in a virtual environment and evaluate its performance. The input is vehicle simulation information, and the test outputs evaluation data as a result.

[0376] Step 3:

[0377] The user's device activates an emotion recognition engine and captures the user's facial expressions and voice using the camera and microphone. This captured information is sent as input to an emotion recognition API for data analysis. The output is emotion information indicating the user's emotional state.

[0378] Step 4:

[0379] The server receives emotional information and adjusts how test results and driving assistance information are presented based on it. For example, if it determines that the user is stressed, it processes the information to be concise and emphasize key points. The input is emotional information and test results, and the output is information tailored to the user.

[0380] Step 5:

[0381] The user receives pre-configured information on their device and uses it to monitor the autonomous vehicle's operation or select rest stops. The input is pre-configured information from the server, and the output is the user's next selected action.

[0382] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0383] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0384] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0385] [Third Embodiment]

[0386] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0387] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0388] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0389] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0390] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0391] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0392] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0393] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0394] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0395] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0396] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0397] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0398] This invention provides a comprehensive system for the efficient and safe development of autonomous driving technology. This system integrates data collection, simulation data generation, benchmark testing, evaluation, and feedback.

[0399] The server collects real-world data, including traffic conditions and accident data, from various sensors and devices, and stores the relevant information in a database. The collected data is not used directly; the server analyzes it and converts it into a common data format. This analysis process optimizes the data for simulation.

[0400] Furthermore, the server generates detailed simulation data to realize a virtual driving simulation based on the accumulated analysis data. This generation process can include a wide variety of driving scenarios and can be customized to take into account vehicle types and specific driving conditions.

[0401] The generated simulation data is applied to a benchmark test environment prepared on the server. Users can upload their self-developed autonomous driving software to this environment from their terminal and receive performance evaluations based on standardized scenarios. The tests are automatically conducted by the server, and the results are collected.

[0402] The collected test results are analyzed by the server, and each performance metric is identified. These results are provided to the user via the terminal, allowing developers to improve and modify the software based on the evaluation. The server also generates specific guidance for performance improvement as feedback, providing advice to the user.

[0403] As a concrete example, users can use simulation data simulating urban traffic congestion to evaluate how autonomous driving software detects and avoids pedestrians and obstacles. Based on these results, users can tune the software's algorithms to build a safer and more efficient autonomous driving system.

[0404] Thus, by using this system, the development of autonomous driving technology can be standardized, safety can be improved, and development costs and time can be expected to be reduced.

[0405] The following describes the processing flow.

[0406] Step 1:

[0407] The server collects up-to-date real-world data on traffic conditions and accidents from various sensors and external databases. This includes road traffic flow, weather conditions, and pedestrian movement.

[0408] Step 2:

[0409] The server analyzes the collected data, performing noise reduction and imputing missing values. This formats the data for simulation. This data preparation is a crucial step in ensuring the accuracy of the simulation.

[0410] Step 3:

[0411] Based on the analyzed data, the server generates simulation data to realize a detailed car driving simulation. Here, various driving scenarios are considered, and detailed models are built for each region and vehicle type.

[0412] Step 4:

[0413] The user uploads the autonomous driving software to the server via their terminal. This allows the software to access the benchmark test environment prepared on the system, and the system is ready to begin testing.

[0414] Step 5:

[0415] The server uses the generated simulation data to evaluate the performance of the uploaded autonomous driving software based on benchmark test scenarios. The tests proceed automatically, and various performance metrics are measured.

[0416] Step 6:

[0417] The server collects and analyzes the test results. From the data obtained from the tests, the software's performance and areas for improvement are identified.

[0418] Step 7:

[0419] The terminal provides the user with the analysis results from the server. Based on these results, the user has the opportunity to improve the algorithms of the autonomous driving software and create new versions.

[0420] Step 8:

[0421] The server provides users with specific suggestions and guidelines for performance improvement as feedback. Users can use this as a reference to rethink their design in order to improve development efficiency and accuracy.

[0422] (Example 1)

[0423] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0424] In the development of autonomous driving technology, systems that are highly safe and efficient are required. However, integrating the collection of real-world data, the generation of simulation data, the conduct of benchmark tests, and the evaluation and provision of feedback on test results is technically complex and leads to increased development costs and time. The present invention aims to provide a system that streamlines and integrates these development processes, enabling developers to improve autonomous driving technology more efficiently and effectively.

[0425] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0426] In this invention, the server includes means for collecting real-world environmental information from various detection devices and equipment and storing it in an information resource; means for analyzing the collected data, converting it into a common format, and optimizing it for use in a virtual environment; and means for generating diverse driving scenarios aimed at realizing a virtual environment based on the optimized data. This enables integrated management of the development process of autonomous driving technology and efficient and safe improvement of system performance.

[0427] "Various detection devices and equipment" refers to hardware such as cameras, sensors, and GPS devices used to collect traffic environment and accident data in real time.

[0428] "Real-world environmental information" refers to data about real-world conditions such as traffic conditions, location information, the movement and location of obstacles, and weather.

[0429] "Information resources" refer to databases and storage mechanisms that store collected data and manage it for later analysis and use.

[0430] A "common format" refers to a unified data format that allows for consistent management of diverse data collected from different devices, facilitating subsequent analysis and use.

[0431] A "virtual environment" refers to a simulated driving space built on a computer to reproduce real-world traffic conditions and test driving software.

[0432] A "driving scenario" refers to a specific set of driving conditions used in the evaluation and development of autonomous driving systems, taking into account different road conditions, weather conditions, vehicle behavior, and other factors.

[0433] This invention provides a comprehensive system for efficiently developing autonomous driving technology. Specific embodiments of this system are described below.

[0434] First, the server receives real-world environmental information, such as traffic conditions and accident data, from various detection devices installed in the autonomous vehicle, including cameras, LiDAR sensors, and GPS devices. This information is stored in a database resource, preparing it for the next analysis step.

[0435] Next, the server analyzes the data stored in the information resources and converts it into a common format. This conversion process ensures that data in different formats can be managed consistently, making it easier to use in subsequent virtual environments. Data cleaning algorithms remove noise from the data and extract meaningful information at this stage.

[0436] Based on the analyzed data, the server generates driving scenarios. By utilizing the generation AI model, virtual driving scenarios based on various driving conditions can be created. For example, scenarios that take into account specific weather conditions and road conditions can be created. As a concrete example, it is possible to generate simulation data for urban traffic congestion.

[0437] Users upload autonomous driving software to a virtual environment on a server via their terminal and perform benchmark tests. These tests evaluate the software's performance based on standardized driving scenarios.

[0438] Test results are analyzed and collected by the server, and performance metrics are clearly displayed. Finally, feedback and specific advice generated based on the evaluation results are provided to the user via the terminal, and the user uses this to improve the software.

[0439] As an example of a prompt, inputting the command "Generate scenario data to evaluate a pedestrian detection algorithm in urban traffic congestion" into the generating AI model will generate the relevant simulation data.

[0440] Thus, by implementing the present invention, it is possible to streamline the development process of autonomous driving technology, thereby improving safety and reducing development costs.

[0441] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0442] Step 1:

[0443] The server receives traffic environment and accident data as input from various detection devices and stores it in its information resources. This process collects data in real time from cameras, LiDAR sensors, and GPS devices. The storage of this data in the information resources forms the foundation for subsequent analysis steps.

[0444] Step 2:

[0445] The server analyzes the accumulated data as input and converts it into a common format. In the specific data cleaning process, noise is removed, relevant information is integrated, and the data is processed into a unified format. This conversion makes data from different devices available in a consistent format. Optimized data is then prepared as output.

[0446] Step 3:

[0447] The server takes optimized data as input and generates driving scenarios using a generated AI model. In this process, inherent parameters are considered to simulate diverse driving conditions and situations. The output is provided as driving scenarios that can be used for testing in a virtual environment.

[0448] Step 4:

[0449] The user inputs the autonomous driving software into a virtual environment on the server via a terminal. The server then performs benchmark tests based on pre-generated driving scenarios. Specifically, it evaluates the performance of how the autonomous driving software functions within the virtual environment. The output obtained from this test consists of various performance indicators.

[0450] Step 5:

[0451] The server analyzes the test results as input and extracts each performance metric. This analysis generates specific improvement suggestions to provide to the user. The output consists of evaluation results and feedback based on them, which are provided to the user as advice via the terminal. This process allows the user to utilize the results for software tuning and improvement suggestions.

[0452] (Application Example 1)

[0453] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0454] Developing autonomous driving technology requires simulations and tests using vast amounts of real-world data, but this presents significant challenges in terms of cost and time. Furthermore, there is a lack of means for developers to receive real-time feedback and efficiently improve autonomous driving software.

[0455] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0456] In this invention, the server includes means for collecting and analyzing the latest environmental information, means for generating vehicle simulation information based on the collected information, and means for conducting standardized tests using the generated simulation information. This enables developers to efficiently and safely advance the development of autonomous driving technology.

[0457] "Environmental information" refers to real-world data such as traffic conditions, road conditions, and weather that are necessary for autonomous vehicles to operate.

[0458] "Analysis" refers to statistical or computational processing to break down collected environmental information data and find meaning in it.

[0459] "Simulation information" refers to data representing a hypothetical driving scenario generated based on analyzed environmental information.

[0460] A "standardized test" is a test designed to consistently evaluate the performance of autonomous driving software under similar conditions.

[0461] "Users" refers to developers and researchers who use this system to develop or improve autonomous driving software.

[0462] A "generative AI model" refers to a model formed by algorithms that use machine learning techniques to extract insights from data.

[0463] "Real-time evaluation" refers to a process that allows for immediate review of test results and feedback, enabling immediate action to be taken in the development process.

[0464] The system implementing this invention is configured in which a server and terminals cooperate using network communication. The server analyzes environmental information collected from various sensors and devices and generates a virtual driving scenario based on this information. The generated simulation information is used for standardized tests to evaluate the vehicle's autonomous driving software.

[0465] The terminal's role is to allow users (developers and researchers) to interactively review evaluation results and acquire data to improve autonomous driving software. The terminal receives real-time feedback from the server, providing users with visual and analytical information. The terminal can also utilize generative AI models to gain further insights from environmental information and test results.

[0466] The main technical elements of this system are as follows:

[0467] 1. Data Collection and Analysis: The server collects traffic conditions and surrounding environmental data via various sensors (cameras, LiDAR, etc.). This enables reliable simulations that mimic actual driving conditions.

[0468] 2. Simulation Generation: Based on the analyzed information, driving scenarios are created using simulation software such as Python or SimPy. This allows for pre-testing of various driving conditions and emergency responses.

[0469] 3. Test Implementation and Feedback Generation: Autonomous driving software is evaluated using a standardized test framework, and its performance is stored in a database. Test results are analyzed using NumPy and Pandas, allowing users to receive feedback for improvement.

[0470] As a concrete example, this system can be used to generate simulation data that mimics urban traffic congestion, and the pedestrian and obstacle detection performance of autonomous driving software can be evaluated. Based on the results of this test, the software algorithm can be improved to achieve safer and more efficient autonomous driving. An example of a prompt used in such a system evaluation process would be, "Test a driving algorithm to safely control the vehicle while maintaining a constant distance from pedestrians in an urban traffic congestion scenario, and provide suggestions for improvement."

[0471] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0472] Step 1:

[0473] The server collects environmental information from various sensors and devices. This information includes traffic conditions, road conditions, and weather information. This data is stored in a database for later analysis. The input is raw data from sensors, and the output is integrated environmental information data.

[0474] Step 2:

[0475] The server analyzes the collected environmental information. Using Python, it analyzes traffic conditions and the positional relationships of objects, and processes the data by converting it into a common format. The input is raw environmental information data, and the output is formatted, analyzed data.

[0476] Step 3:

[0477] The server generates simulation information using the analyzed data. It uses simulation software (e.g., SimPy) to create a virtual operating scenario that considers multiple operating conditions. The input is the analyzed data, and the output is the simulation information.

[0478] Step 4:

[0479] The terminal receives the generated simulation information and performs standardized tests. The user specifies the test conditions and uses computing resources on the server to evaluate the autonomous driving software. The input is simulation information, and the output is test result data.

[0480] Step 5:

[0481] The server analyzes test results and generates feedback. It utilizes machine learning models (generative AI models) to extract insights from test results and provide concrete suggestions for performance improvement. The input is test result data, and the output is analysis results and feedback information.

[0482] Step 6:

[0483] The terminal presents the user with feedback from the server. The user then uses this feedback to consider ways to improve the autonomous driving software. This process involves using prompts to further analyze the insights gained from the generated AI model. The input is feedback information, and the output is a set of improvement suggestions formatted in a way that is easy for the user to understand.

[0484] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0485] This invention is a comprehensive system for improving efficiency and safety in the development of autonomous driving technology, and further aims to make the developer experience more personalized and optimized by incorporating an emotion engine that recognizes user emotions.

[0486] The server first collects real-world data and analyzes data such as traffic conditions, accident history, and environmental conditions. This data is processed to enable car driving simulations and converted into simulation datasets. The server then uses a scenario generation engine to create standard and customized driving scenarios.

[0487] Next, the user uploads the autonomous driving software via their terminal and connects to a benchmark test environment on the server. In this environment, the software's performance is tested based on generated scenarios. This provides detailed test results according to standardized evaluation criteria.

[0488] A key feature of this invention is the inclusion of an emotion engine, which allows the terminal to recognize emotions from the user's facial expressions, tone of voice, and operation patterns. This emotion information is fed back to the server, and the method of presenting test results is adjusted according to the user's emotions. For example, if the emotion engine determines that the user is stressed, the server provides clearly organized information and modifies the display procedure to facilitate user comprehension.

[0489] Furthermore, the results of the emotion engine are reflected in the feedback, supplementing it with specific advice and guidance that users need. This makes it easier for users to improve the software and enhances the efficiency and effectiveness of the entire development process.

[0490] For example, when a user tests a scenario in a specific situation (e.g., nighttime driving in an urban area), if the emotion engine recognizes the user's anxiety or tension, the server will simplify the presentation of information and clarify the points to emphasize, thereby reducing the user's workload. In this way, flexible support tailored to the user's emotional state can be provided, further refining the development of autonomous driving technology.

[0491] The following describes the processing flow.

[0492] Step 1:

[0493] The server collects real-world data such as traffic conditions and environmental conditions from various sensors and databases. This includes visibility data during nighttime driving and traffic density information in urban areas.

[0494] Step 2:

[0495] The server analyzes the collected data, removes noise, and converts it into a format optimized for simulation. During this process, missing data is filled in, and standardization is performed to ensure consistency.

[0496] Step 3:

[0497] The server uses the analyzed data to generate specific simulation data for car driving. This simulation data includes multiple driving scenarios, creating situations based on specific conditions and locations.

[0498] Step 4:

[0499] The user uploads the autonomous driving software to the server using their device. This operation allows the software to be evaluated in a benchmark test environment.

[0500] Step 5:

[0501] The server activates the emotion engine and collects facial expressions and voice data from the user's device via the camera and microphone. This prepares the system to recognize the user's emotional state in real time.

[0502] Step 6:

[0503] The server performs benchmark tests using simulation data and meticulously records the results. This includes data on whether the software handled the conditions well.

[0504] Step 7:

[0505] The server analyzes the test results and evaluates each performance metric. In this process, the emotion engine adjusts how the test results are presented based on the user's emotions as perceived.

[0506] Step 8:

[0507] The device displays adjusted test results to the user. The display includes highlights and diagrams to make it easy for the user to understand.

[0508] Step 9:

[0509] The emotion engine integrates the analysis results into feedback and generates specific improvement suggestions and advice on the server.

[0510] Step 10:

[0511] Users receive feedback on their devices, which is then used to improve the autonomous driving software. This ensures an efficient and appropriate development process.

[0512] (Example 2)

[0513] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0514] In developing autonomous driving technology, it is essential to utilize real-world data and conduct efficient and safe evaluations. Furthermore, providing appropriate feedback and information tailored to the user's emotional state is also crucial. Current technology struggles to fully meet these requirements, particularly in terms of system design that considers user emotions.

[0515] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0516] In this invention, the server includes means for collecting and analyzing the latest real-world information, means for generating simulated data for transportation equipment based on the collected information, and means for conducting standard tests using the generated simulated data. This makes it possible to recognize the user's emotions and provide information based on those emotions.

[0517] "Real-world information" refers to data about events and conditions that occur in the real world, including traffic conditions, accident history, and environmental conditions.

[0518] "Simulated data for transportation equipment" refers to simulation data used to reproduce the operation of transportation equipment based on collected real-world information.

[0519] A "reference test" is a standardized test method conducted to evaluate the performance of developed software or systems using simulated data for transportation equipment.

[0520] "Means of recognizing emotions" refers to technologies that analyze data such as the user's facial expressions, tone of voice, and operation patterns to identify their emotional state.

[0521] "Means of adjusting the method of information delivery" refers to technologies that optimize the way information is presented according to the user's emotions, aiming to provide information in an easy-to-understand and effective manner.

[0522] This invention is a system for supporting the efficient development of autonomous driving technology, and is configured as follows.

[0523] The server first collects the latest real-world information. This information includes traffic conditions, accident history, and environmental conditions. This collection is carried out using sensing technology, and the data is aggregated on the server. Next, the server generates simulated data for transportation equipment based on this real-world information. This simulated data is generated using a simulation execution engine and is used as the basis data for driving simulations.

[0524] The user uploads autonomous driving software to the server via a terminal. The server runs the software sent from the terminal in a standard test environment. This test is conducted based on generated simulated data and adheres to performance evaluation criteria. The terminal also has an emotion recognition function that analyzes the user's emotions in real time. This emotion data is sent to the server, which adjusts the way the test results are presented based on the user's emotions. This allows the user to receive evaluation results in an easy-to-understand format.

[0525] For example, if a user conducts a test simulating nighttime driving in an urban area, and the server detects tension from the user's tone of voice and facial expression, it will organize the information clearly and highlight key points. In this way, information is provided according to the user's emotional state, which can improve the quality of the development process.

[0526] An example of a prompt used in a generative AI model is the input, "Please explain in detail how to use emotion recognition technology to present test results of autonomous driving software in a way that matches the user's emotions."

[0527] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0528] Step 1:

[0529] The server automatically collects real-world information such as traffic conditions, accident history, and environmental conditions from external sensor systems. This data is stored in storage. As input, it receives data from multiple data sources (e.g., cameras, radar, traffic APIs), performs data cleaning, and converts it into well-formed data. As output, it generates an analyzable dataset suitable for driving simulations.

[0530] Step 2:

[0531] The server uses this dataset to generate simulated data for transportation equipment. This process utilizes a simulation execution engine to set simulation parameters and build scenarios. It receives the dataset generated in step 1 as input and uses a scenario generation algorithm to construct standard and customized simulated data. The output is a detailed operating scenario, resulting in a complete dataset for simulation.

[0532] Step 3:

[0533] The user uploads autonomous driving software from their device to the server. This is done through a user-friendly UI. The server runs the uploaded software within a standard test framework. The input is a software package sent by the user; the server decompresses this software and runs it in the test environment. The output generates software operation logs and performance metrics, which are used for evaluation in the next step.

[0534] Step 4:

[0535] The device recognizes emotions through the user's facial expressions, tone of voice, and operation patterns. This uses camera and microphone input, and an emotion recognition algorithm is executed in real time. As input, the user's real-time data is input to the device, and as output, an index indicating the user's emotional state is calculated and sent to the server.

[0536] Step 5:

[0537] The server adjusts how test results are presented based on the user's emotional state. It analyzes emotional data and summarizes and emphasizes the information presented. The server requires the emotional data obtained in step 4 and the test results from step 3 as input, and applies an optimization algorithm for presentation. As output, feedback and advice that takes the user's emotions into account are generated and presented in a way that is easy for the user to understand.

[0538] (Application Example 2)

[0539] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0540] In the development of autonomous driving technology, there is a need to improve development efficiency and quality by effectively utilizing real-world data while providing feedback that takes into account the emotional state of the developers. Conventional systems have not made adjustments that reflect such individualized emotions, which has sometimes increased the workload of developers.

[0541] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0542] In this invention, the server includes means for collecting and analyzing the latest real-world information, means for generating vehicle simulation information based on the collected information, means for conducting standard tests using the generated simulation information, means for evaluating the test results and providing the evaluation results to the user, and means for generating emotional information using an emotional recognition engine that recognizes the user's emotions and adjusting the information presentation means based on that emotional information. This makes it possible to adjust information according to the user's emotional state, providing personalized support and improving the efficiency of the entire development process.

[0543] "Latest real-world information" refers to various dynamic data that occurs in the real world, including traffic conditions, accident history, and environmental conditions.

[0544] "Vehicle simulation information" refers to a set of data generated based on real-world information to simulate the operation of autonomous vehicles in a virtual environment.

[0545] A "standard test" is a standardized test conducted using vehicle simulation data to evaluate the performance of autonomous driving technology.

[0546] An "emotion recognition engine" is a computer system that analyzes emotions from a user's facial expressions, voice, and operation patterns.

[0547] "Emotional information" refers to data that indicates the user's current emotional state, based on the results of analysis by an emotion recognition engine.

[0548] In the system that realizes this invention, a server first collects the latest real-world information such as traffic conditions and environmental conditions, and generates vehicle simulation information by analyzing it. Based on the generated simulation information, the server conducts standard tests and evaluates the results. The evaluated test results are provided to the user and displayed on the user's terminal.

[0549] The emotion recognition engine analyzes the user's facial expressions, voice, and operation patterns to generate emotional information. Based on this emotional information, the server adjusts how information is provided to the user. For example, if the user is stressed, the information presented is made more concise and key points are highlighted. Conversely, if the user is relaxed, more detailed information is provided.

[0550] The server and terminal systems utilize emotion recognition APIs such as "Microsoft Azure's Emotion API" and "Google Cloud's Vision AI" to analyze user emotions in real time. Additionally, "Google Maps API" and other services are used to provide appropriate assistance and guidance for navigation.

[0551] For example, if a user is traveling long distances in an autonomous vehicle and fatigue is detected from their facial expression, the emotion recognition engine will suggest a break and display the route to the nearest rest stop on the device. Furthermore, for users relaxing in their car on their way home from work, the system will support safer travel by displaying more detailed traffic conditions than usual.

[0552] Examples of prompts for a generative AI model are as follows:

[0553] "Estimate the user's emotions from their facial expressions and voice, and decide what information to provide based on their driving situation. If the user is stressed, provide concise information; if they are tired, encourage them to take a break; and if they are relaxed, provide detailed route guidance."

[0554] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0555] Step 1:

[0556] The server collects the latest real-world information from various sensors and databases. Input data includes traffic conditions, accident history, and environmental conditions. Based on this input, data analysis algorithms are used to process the information and convert it into vehicle simulation information. The output is this vehicle simulation information.

[0557] Step 2:

[0558] The server performs baseline tests using the generated vehicle simulation information. The tests simulate autonomous driving behavior in a virtual environment and evaluate its performance. The input is vehicle simulation information, and the test outputs evaluation data as a result.

[0559] Step 3:

[0560] The user's device activates an emotion recognition engine and captures the user's facial expressions and voice using the camera and microphone. This captured information is sent as input to an emotion recognition API for data analysis. The output is emotion information indicating the user's emotional state.

[0561] Step 4:

[0562] The server receives emotional information and adjusts how test results and driving assistance information are presented based on it. For example, if it determines that the user is stressed, it processes the information to be concise and emphasize key points. The input is emotional information and test results, and the output is information tailored to the user.

[0563] Step 5:

[0564] The user receives pre-configured information on their device and uses it to monitor the autonomous vehicle's operation or select rest stops. The input is pre-configured information from the server, and the output is the user's next selected action.

[0565] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0566] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0567] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0568] [Fourth Embodiment]

[0569] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0570] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0571] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0572] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0573] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0574] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0575] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0576] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0577] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0578] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0579] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0580] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0581] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0582] This invention provides a comprehensive system for the efficient and safe development of autonomous driving technology. This system integrates data collection, simulation data generation, benchmark testing, evaluation, and feedback.

[0583] The server collects real-world data, including traffic conditions and accident data, from various sensors and devices, and stores the relevant information in a database. The collected data is not used directly; the server analyzes it and converts it into a common data format. This analysis process optimizes the data for simulation.

[0584] Furthermore, the server generates detailed simulation data to realize a virtual driving simulation based on the accumulated analysis data. This generation process can include a wide variety of driving scenarios and can be customized to take into account vehicle types and specific driving conditions.

[0585] The generated simulation data is applied to a benchmark test environment prepared on the server. Users can upload their self-developed autonomous driving software to this environment from their terminal and receive performance evaluations based on standardized scenarios. The tests are automatically conducted by the server, and the results are collected.

[0586] The collected test results are analyzed by the server, and each performance metric is identified. These results are provided to the user via the terminal, allowing developers to improve and modify the software based on the evaluation. The server also generates specific guidance for performance improvement as feedback, providing advice to the user.

[0587] As a concrete example, users can use simulation data simulating urban traffic congestion to evaluate how autonomous driving software detects and avoids pedestrians and obstacles. Based on these results, users can tune the software's algorithms to build a safer and more efficient autonomous driving system.

[0588] Thus, by using this system, the development of autonomous driving technology can be standardized, safety can be improved, and development costs and time can be expected to be reduced.

[0589] The following describes the processing flow.

[0590] Step 1:

[0591] The server collects up-to-date real-world data on traffic conditions and accidents from various sensors and external databases. This includes road traffic flow, weather conditions, and pedestrian movement.

[0592] Step 2:

[0593] The server analyzes the collected data, performing noise reduction and imputing missing values. This formats the data for simulation. This data preparation is a crucial step in ensuring the accuracy of the simulation.

[0594] Step 3:

[0595] Based on the analyzed data, the server generates simulation data to realize a detailed car driving simulation. Here, various driving scenarios are considered, and detailed models are built for each region and vehicle type.

[0596] Step 4:

[0597] The user uploads the autonomous driving software to the server via their terminal. This allows the software to access the benchmark test environment prepared on the system, and the system is ready to begin testing.

[0598] Step 5:

[0599] The server uses the generated simulation data to evaluate the performance of the uploaded autonomous driving software based on benchmark test scenarios. The tests proceed automatically, and various performance metrics are measured.

[0600] Step 6:

[0601] The server collects and analyzes the test results. From the data obtained from the tests, the software's performance and areas for improvement are identified.

[0602] Step 7:

[0603] The terminal provides the user with the analysis results from the server. Based on these results, the user has the opportunity to improve the algorithms of the autonomous driving software and create new versions.

[0604] Step 8:

[0605] The server provides users with specific suggestions and guidelines for performance improvement as feedback. Users can use this as a reference to rethink their design in order to improve development efficiency and accuracy.

[0606] (Example 1)

[0607] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0608] In the development of autonomous driving technology, systems that are highly safe and efficient are required. However, integrating the collection of real-world data, the generation of simulation data, the conduct of benchmark tests, and the evaluation and provision of feedback on test results is technically complex and leads to increased development costs and time. The present invention aims to provide a system that streamlines and integrates these development processes, enabling developers to improve autonomous driving technology more efficiently and effectively.

[0609] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0610] In this invention, the server includes means for collecting real-world environmental information from various detection devices and equipment and storing it in an information resource; means for analyzing the collected data, converting it into a common format, and optimizing it for use in a virtual environment; and means for generating diverse driving scenarios aimed at realizing a virtual environment based on the optimized data. This enables integrated management of the development process of autonomous driving technology and efficient and safe improvement of system performance.

[0611] "Various detection devices and equipment" refers to hardware such as cameras, sensors, and GPS devices used to collect traffic environment and accident data in real time.

[0612] "Real-world environmental information" refers to data about real-world conditions such as traffic conditions, location information, the movement and location of obstacles, and weather.

[0613] "Information resources" refer to databases and storage mechanisms that store collected data and manage it for later analysis and use.

[0614] A "common format" refers to a unified data format that allows for consistent management of diverse data collected from different devices, facilitating subsequent analysis and use.

[0615] A "virtual environment" refers to a simulated driving space built on a computer to reproduce real-world traffic conditions and test driving software.

[0616] A "driving scenario" refers to a specific set of driving conditions used in the evaluation and development of autonomous driving systems, taking into account different road conditions, weather conditions, vehicle behavior, and other factors.

[0617] This invention provides a comprehensive system for efficiently developing autonomous driving technology. Specific embodiments of this system are described below.

[0618] First, the server receives real-world environmental information, such as traffic conditions and accident data, from various detection devices installed in the autonomous vehicle, including cameras, LiDAR sensors, and GPS devices. This information is stored in a database resource, preparing it for the next analysis step.

[0619] Next, the server analyzes the data stored in the information resources and converts it into a common format. This conversion process ensures that data in different formats can be managed consistently, making it easier to use in subsequent virtual environments. Data cleaning algorithms remove noise from the data and extract meaningful information at this stage.

[0620] Based on the analyzed data, the server generates driving scenarios. By utilizing the generation AI model, virtual driving scenarios based on various driving conditions can be created. For example, scenarios that take into account specific weather conditions and road conditions can be created. As a concrete example, it is possible to generate simulation data for urban traffic congestion.

[0621] Users upload autonomous driving software to a virtual environment on a server via their terminal and perform benchmark tests. These tests evaluate the software's performance based on standardized driving scenarios.

[0622] Test results are analyzed and collected by the server, and performance metrics are clearly displayed. Finally, feedback and specific advice generated based on the evaluation results are provided to the user via the terminal, and the user uses this to improve the software.

[0623] As an example of a prompt, inputting the command "Generate scenario data to evaluate a pedestrian detection algorithm in urban traffic congestion" into the generating AI model will generate the relevant simulation data.

[0624] Thus, by implementing the present invention, it is possible to streamline the development process of autonomous driving technology, thereby improving safety and reducing development costs.

[0625] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0626] Step 1:

[0627] The server receives traffic environment and accident data as input from various detection devices and stores it in its information resources. This process collects data in real time from cameras, LiDAR sensors, and GPS devices. The storage of this data in the information resources forms the foundation for subsequent analysis steps.

[0628] Step 2:

[0629] The server analyzes the accumulated data as input and converts it into a common format. In the specific data cleaning process, noise is removed, relevant information is integrated, and the data is processed into a unified format. This conversion makes data from different devices available in a consistent format. Optimized data is then prepared as output.

[0630] Step 3:

[0631] The server takes optimized data as input and generates driving scenarios using a generated AI model. In this process, inherent parameters are considered to simulate diverse driving conditions and situations. The output is provided as driving scenarios that can be used for testing in a virtual environment.

[0632] Step 4:

[0633] The user inputs the autonomous driving software into a virtual environment on the server via a terminal. The server then performs benchmark tests based on pre-generated driving scenarios. Specifically, it evaluates the performance of how the autonomous driving software functions within the virtual environment. The output obtained from this test consists of various performance indicators.

[0634] Step 5:

[0635] The server analyzes the test results as input and extracts each performance metric. This analysis generates specific improvement suggestions to provide to the user. The output consists of evaluation results and feedback based on them, which are provided to the user as advice via the terminal. This process allows the user to utilize the results for software tuning and improvement suggestions.

[0636] (Application Example 1)

[0637] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0638] Developing autonomous driving technology requires simulations and tests using vast amounts of real-world data, but this presents significant challenges in terms of cost and time. Furthermore, there is a lack of means for developers to receive real-time feedback and efficiently improve autonomous driving software.

[0639] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0640] In this invention, the server includes means for collecting and analyzing the latest environmental information, means for generating vehicle simulation information based on the collected information, and means for conducting standardized tests using the generated simulation information. This enables developers to efficiently and safely advance the development of autonomous driving technology.

[0641] "Environmental information" refers to real-world data such as traffic conditions, road conditions, and weather that are necessary for autonomous vehicles to operate.

[0642] "Analysis" refers to statistical or computational processing to break down collected environmental information data and find meaning in it.

[0643] "Simulation information" refers to data representing a hypothetical driving scenario generated based on analyzed environmental information.

[0644] A "standardized test" is a test designed to consistently evaluate the performance of autonomous driving software under similar conditions.

[0645] "Users" refers to developers and researchers who use this system to develop or improve autonomous driving software.

[0646] A "generative AI model" refers to a model formed by algorithms that use machine learning techniques to extract insights from data.

[0647] "Real-time evaluation" refers to a process that allows for immediate review of test results and feedback, enabling immediate action to be taken in the development process.

[0648] The system implementing this invention is configured in which a server and terminals cooperate using network communication. The server analyzes environmental information collected from various sensors and devices and generates a virtual driving scenario based on this information. The generated simulation information is used for standardized tests to evaluate the vehicle's autonomous driving software.

[0649] The terminal's role is to allow users (developers and researchers) to interactively review evaluation results and acquire data to improve autonomous driving software. The terminal receives real-time feedback from the server, providing users with visual and analytical information. The terminal can also utilize generative AI models to gain further insights from environmental information and test results.

[0650] The main technical elements of this system are as follows:

[0651] 1. Data Collection and Analysis: The server collects traffic conditions and surrounding environmental data via various sensors (cameras, LiDAR, etc.). This enables reliable simulations that mimic actual driving conditions.

[0652] 2. Simulation Generation: Based on the analyzed information, driving scenarios are created using simulation software such as Python or SimPy. This allows for pre-testing of various driving conditions and emergency responses.

[0653] 3. Test Implementation and Feedback Generation: Autonomous driving software is evaluated using a standardized test framework, and its performance is stored in a database. Test results are analyzed using NumPy and Pandas, allowing users to receive feedback for improvement.

[0654] As a concrete example, this system can be used to generate simulation data that mimics urban traffic congestion, and the pedestrian and obstacle detection performance of autonomous driving software can be evaluated. Based on the results of this test, the software algorithm can be improved to achieve safer and more efficient autonomous driving. An example of a prompt used in such a system evaluation process would be, "Test a driving algorithm to safely control the vehicle while maintaining a constant distance from pedestrians in an urban traffic congestion scenario, and provide suggestions for improvement."

[0655] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0656] Step 1:

[0657] The server collects environmental information from various sensors and devices. This information includes traffic conditions, road conditions, and weather information. This data is stored in a database for later analysis. The input is raw data from sensors, and the output is integrated environmental information data.

[0658] Step 2:

[0659] The server analyzes the collected environmental information. Using Python, it analyzes traffic conditions and the positional relationships of objects, and processes the data by converting it into a common format. The input is raw environmental information data, and the output is formatted, analyzed data.

[0660] Step 3:

[0661] The server generates simulation information using the analyzed data. It uses simulation software (e.g., SimPy) to create a virtual operating scenario that considers multiple operating conditions. The input is the analyzed data, and the output is the simulation information.

[0662] Step 4:

[0663] The terminal receives the generated simulation information and performs standardized tests. The user specifies the test conditions and uses computing resources on the server to evaluate the autonomous driving software. The input is simulation information, and the output is test result data.

[0664] Step 5:

[0665] The server analyzes test results and generates feedback. It utilizes machine learning models (generative AI models) to extract insights from test results and provide concrete suggestions for performance improvement. The input is test result data, and the output is analysis results and feedback information.

[0666] Step 6:

[0667] The terminal presents the user with feedback from the server. The user then uses this feedback to consider ways to improve the autonomous driving software. This process involves using prompts to further analyze the insights gained from the generated AI model. The input is feedback information, and the output is a set of improvement suggestions formatted in a way that is easy for the user to understand.

[0668] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0669] This invention is a comprehensive system for improving efficiency and safety in the development of autonomous driving technology, and further aims to make the developer experience more personalized and optimized by incorporating an emotion engine that recognizes user emotions.

[0670] The server first collects real-world data and analyzes data such as traffic conditions, accident history, and environmental conditions. This data is processed to enable car driving simulations and converted into simulation datasets. The server then uses a scenario generation engine to create standard and customized driving scenarios.

[0671] Next, the user uploads the autonomous driving software via their terminal and connects to a benchmark test environment on the server. In this environment, the software's performance is tested based on generated scenarios. This provides detailed test results according to standardized evaluation criteria.

[0672] A key feature of this invention is the inclusion of an emotion engine, which allows the terminal to recognize emotions from the user's facial expressions, tone of voice, and operation patterns. This emotion information is fed back to the server, and the method of presenting test results is adjusted according to the user's emotions. For example, if the emotion engine determines that the user is stressed, the server provides clearly organized information and modifies the display procedure to facilitate user comprehension.

[0673] Furthermore, the results of the emotion engine are reflected in the feedback, supplementing it with specific advice and guidance that users need. This makes it easier for users to improve the software and enhances the efficiency and effectiveness of the entire development process.

[0674] For example, when a user tests a scenario in a specific situation (e.g., nighttime driving in an urban area), if the emotion engine recognizes the user's anxiety or tension, the server will simplify the presentation of information and clarify the points to emphasize, thereby reducing the user's workload. In this way, flexible support tailored to the user's emotional state can be provided, further refining the development of autonomous driving technology.

[0675] The following describes the processing flow.

[0676] Step 1:

[0677] The server collects real-world data such as traffic conditions and environmental conditions from various sensors and databases. This includes visibility data during nighttime driving and traffic density information in urban areas.

[0678] Step 2:

[0679] The server analyzes the collected data, removes noise, and converts it into a format optimized for simulation. During this process, missing data is filled in, and standardization is performed to ensure consistency.

[0680] Step 3:

[0681] The server uses the analyzed data to generate specific simulation data for car driving. This simulation data includes multiple driving scenarios, creating situations based on specific conditions and locations.

[0682] Step 4:

[0683] The user uploads the autonomous driving software to the server using their device. This operation allows the software to be evaluated in a benchmark test environment.

[0684] Step 5:

[0685] The server activates the emotion engine and collects facial expressions and voice data from the user's device via the camera and microphone. This prepares the system to recognize the user's emotional state in real time.

[0686] Step 6:

[0687] The server performs benchmark tests using simulation data and meticulously records the results. This includes data on whether the software handled the conditions well.

[0688] Step 7:

[0689] The server analyzes the test results and evaluates each performance metric. In this process, the emotion engine adjusts how the test results are presented based on the user's emotions as perceived.

[0690] Step 8:

[0691] The device displays adjusted test results to the user. The display includes highlights and diagrams to make it easy for the user to understand.

[0692] Step 9:

[0693] The emotion engine integrates the analysis results into feedback and generates specific improvement suggestions and advice on the server.

[0694] Step 10:

[0695] Users receive feedback on their devices, which is then used to improve the autonomous driving software. This ensures an efficient and appropriate development process.

[0696] (Example 2)

[0697] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0698] In developing autonomous driving technology, it is essential to utilize real-world data and conduct efficient and safe evaluations. Furthermore, providing appropriate feedback and information tailored to the user's emotional state is also crucial. Current technology struggles to fully meet these requirements, particularly in terms of system design that considers user emotions.

[0699] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0700] In this invention, the server includes means for collecting and analyzing the latest real-world information, means for generating simulated data for transportation equipment based on the collected information, and means for conducting standard tests using the generated simulated data. This makes it possible to recognize the user's emotions and provide information based on those emotions.

[0701] "Real-world information" refers to data about events and conditions that occur in the real world, including traffic conditions, accident history, and environmental conditions.

[0702] "Simulated data for transportation equipment" refers to simulation data used to reproduce the operation of transportation equipment based on collected real-world information.

[0703] A "reference test" is a standardized test method conducted to evaluate the performance of developed software or systems using simulated data for transportation equipment.

[0704] "Means of recognizing emotions" refers to technologies that analyze data such as the user's facial expressions, tone of voice, and operation patterns to identify their emotional state.

[0705] "Means of adjusting the method of information delivery" refers to technologies that optimize the way information is presented according to the user's emotions, aiming to provide information in an easy-to-understand and effective manner.

[0706] This invention is a system for supporting the efficient development of autonomous driving technology, and is configured as follows.

[0707] The server first collects the latest real-world information. This information includes traffic conditions, accident history, and environmental conditions. This collection is carried out using sensing technology, and the data is aggregated on the server. Next, the server generates simulated data for transportation equipment based on this real-world information. This simulated data is generated using a simulation execution engine and is used as the basis data for driving simulations.

[0708] The user uploads autonomous driving software to the server via a terminal. The server runs the software sent from the terminal in a standard test environment. This test is conducted based on generated simulated data and adheres to performance evaluation criteria. The terminal also has an emotion recognition function that analyzes the user's emotions in real time. This emotion data is sent to the server, which adjusts the way the test results are presented based on the user's emotions. This allows the user to receive evaluation results in an easy-to-understand format.

[0709] For example, if a user conducts a test simulating nighttime driving in an urban area, and the server detects tension from the user's tone of voice and facial expression, it will organize the information clearly and highlight key points. In this way, information is provided according to the user's emotional state, which can improve the quality of the development process.

[0710] An example of a prompt used in a generative AI model is the input, "Please explain in detail how to use emotion recognition technology to present test results of autonomous driving software in a way that matches the user's emotions."

[0711] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0712] Step 1:

[0713] The server automatically collects real-world information such as traffic conditions, accident history, and environmental conditions from external sensor systems. This data is stored in storage. As input, it receives data from multiple data sources (e.g., cameras, radar, traffic APIs), performs data cleaning, and converts it into well-formed data. As output, it generates an analyzable dataset suitable for driving simulations.

[0714] Step 2:

[0715] The server uses this dataset to generate simulated data for transportation equipment. This process utilizes a simulation execution engine to set simulation parameters and build scenarios. It receives the dataset generated in step 1 as input and uses a scenario generation algorithm to construct standard and customized simulated data. The output is a detailed operating scenario, resulting in a complete dataset for simulation.

[0716] Step 3:

[0717] The user uploads autonomous driving software from their device to the server. This is done through a user-friendly UI. The server runs the uploaded software within a standard test framework. The input is a software package sent by the user; the server decompresses this software and runs it in the test environment. The output generates software operation logs and performance metrics, which are used for evaluation in the next step.

[0718] Step 4:

[0719] The device recognizes emotions through the user's facial expressions, tone of voice, and operation patterns. This uses camera and microphone input, and an emotion recognition algorithm is executed in real time. As input, the user's real-time data is input to the device, and as output, an index indicating the user's emotional state is calculated and sent to the server.

[0720] Step 5:

[0721] The server adjusts how test results are presented based on the user's emotional state. It analyzes emotional data and summarizes and emphasizes the information presented. The server requires the emotional data obtained in step 4 and the test results from step 3 as input, and applies an optimization algorithm for presentation. As output, feedback and advice that takes the user's emotions into account are generated and presented in a way that is easy for the user to understand.

[0722] (Application Example 2)

[0723] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0724] In the development of autonomous driving technology, there is a need to improve development efficiency and quality by effectively utilizing real-world data while providing feedback that takes into account the emotional state of the developers. Conventional systems have not made adjustments that reflect such individualized emotions, which has sometimes increased the workload of developers.

[0725] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0726] In this invention, the server includes means for collecting and analyzing the latest real-world information, means for generating vehicle simulation information based on the collected information, means for conducting standard tests using the generated simulation information, means for evaluating the test results and providing the evaluation results to the user, and means for generating emotional information using an emotional recognition engine that recognizes the user's emotions and adjusting the information presentation means based on that emotional information. This makes it possible to adjust information according to the user's emotional state, providing personalized support and improving the efficiency of the entire development process.

[0727] "Latest real-world information" refers to various dynamic data that occurs in the real world, including traffic conditions, accident history, and environmental conditions.

[0728] "Vehicle simulation information" refers to a set of data generated based on real-world information to simulate the operation of autonomous vehicles in a virtual environment.

[0729] A "standard test" is a standardized test conducted using vehicle simulation data to evaluate the performance of autonomous driving technology.

[0730] An "emotion recognition engine" is a computer system that analyzes emotions from a user's facial expressions, voice, and operation patterns.

[0731] "Emotional information" refers to data that indicates the user's current emotional state, based on the results of analysis by an emotion recognition engine.

[0732] In the system that realizes this invention, a server first collects the latest real-world information such as traffic conditions and environmental conditions, and generates vehicle simulation information by analyzing it. Based on the generated simulation information, the server conducts standard tests and evaluates the results. The evaluated test results are provided to the user and displayed on the user's terminal.

[0733] The emotion recognition engine analyzes the user's facial expressions, voice, and operation patterns to generate emotional information. Based on this emotional information, the server adjusts how information is provided to the user. For example, if the user is stressed, the information presented is made more concise and key points are highlighted. Conversely, if the user is relaxed, more detailed information is provided.

[0734] The server and terminal systems utilize emotion recognition APIs such as "Microsoft Azure's Emotion API" and "Google Cloud's Vision AI" to analyze user emotions in real time. Additionally, "Google Maps API" and other services are used to provide appropriate assistance and guidance for navigation.

[0735] For example, if a user is traveling long distances in an autonomous vehicle and fatigue is detected from their facial expression, the emotion recognition engine will suggest a break and display the route to the nearest rest stop on the device. Furthermore, for users relaxing in their car on their way home from work, the system will support safer travel by displaying more detailed traffic conditions than usual.

[0736] Examples of prompts for a generative AI model are as follows:

[0737] "Estimate the user's emotions from their facial expressions and voice, and decide what information to provide based on their driving situation. If the user is stressed, provide concise information; if they are tired, encourage them to take a break; and if they are relaxed, provide detailed route guidance."

[0738] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0739] Step 1:

[0740] The server collects the latest real-world information from various sensors and databases. Input data includes traffic conditions, accident history, and environmental conditions. Based on this input, data analysis algorithms are used to process the information and convert it into vehicle simulation information. The output is this vehicle simulation information.

[0741] Step 2:

[0742] The server performs baseline tests using the generated vehicle simulation information. The tests simulate autonomous driving behavior in a virtual environment and evaluate its performance. The input is vehicle simulation information, and the test outputs evaluation data as a result.

[0743] Step 3:

[0744] The user's device activates an emotion recognition engine and captures the user's facial expressions and voice using the camera and microphone. This captured information is sent as input to an emotion recognition API for data analysis. The output is emotion information indicating the user's emotional state.

[0745] Step 4:

[0746] The server receives emotional information and adjusts how test results and driving assistance information are presented based on it. For example, if it determines that the user is stressed, it processes the information to be concise and emphasize key points. The input is emotional information and test results, and the output is information tailored to the user.

[0747] Step 5:

[0748] The user receives pre-configured information on their device and uses it to monitor the autonomous vehicle's operation or select rest stops. The input is pre-configured information from the server, and the output is the user's next selected action.

[0749] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0750] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0751] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0752] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0753] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. In the upper and lower directions of the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. Also, the upper side of the concentric circles is where "pleasant" emotions are located, and the lower side is where "unpleasant" emotions are located. In this way, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0754] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0755] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0756] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0757] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0758] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0759] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0760] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0761] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0762] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0763] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0764] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0765] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0766] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0767] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0768] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0769] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0770] The following is further disclosed regarding the embodiments described above.

[0771] (Claim 1)

[0772] A means of collecting and analyzing the latest real-world data,

[0773] A means for generating automotive simulation data based on collected data,

[0774] A means of conducting benchmark tests using the generated simulation data,

[0775] A means of evaluating test results and providing those evaluation results to the user,

[0776] A system that includes this.

[0777] (Claim 2)

[0778] The system according to claim 1, comprising means for creating and providing multiple scenarios based on analyzed data.

[0779] (Claim 3)

[0780] The system according to claim 1, comprising means for generating feedback based on evaluation results and for the user to use it to improve autonomous driving software.

[0781] "Example 1"

[0782] (Claim 1)

[0783] A means of collecting real-world environmental information from various detection devices and equipment and storing it in information resources,

[0784] A means of analyzing the collected data, converting it into a common format, and optimizing it for use in a virtual environment,

[0785] A means for generating diverse operating scenarios aimed at realizing a virtual environment based on optimized data,

[0786] A means of conducting standardized tests in a virtual environment and evaluating their performance,

[0787] A means of generating improvement proposals through the analyzed evaluation results and providing them to the user,

[0788] A system that includes this.

[0789] (Claim 2)

[0790] The system according to claim 1, comprising means for creating multiple driving scenarios based on analyzed data, corresponding to driving conditions and vehicle type, and applying them to testing in a virtual environment.

[0791] (Claim 3)

[0792] The system according to claim 1, comprising means for generating improvement proposals obtained from evaluation results and providing the user with advice to promote the optimization of driving techniques in a virtual environment.

[0793] "Application Example 1"

[0794] (Claim 1)

[0795] A means of collecting and analyzing the latest environmental information,

[0796] A means for generating vehicle simulation information based on collected information,

[0797] A means of conducting standardized tests using the generated simulation information,

[0798] A means of analyzing test results and providing those analysis results to users,

[0799] A means of creating and providing multiple operating conditions based on user instructions,

[0800] A means to enable users to receive evaluations in real time and consider improvement measures using computing resources,

[0801] A system that includes this.

[0802] (Claim 2)

[0803] The system according to claim 1, comprising means for using prompt statements and providing insights using a generative AI model.

[0804] (Claim 3)

[0805] The system according to claim 1, comprising means for presenting improvement proposals based on evaluation results and generated feedback, and utilizing them in the development of autonomous driving software.

[0806] "Example 2 of combining an emotion engine"

[0807] (Claim 1)

[0808] A means of collecting and analyzing the latest real-world information,

[0809] A means for generating simulated data for transportation equipment based on collected information,

[0810] A means of conducting a standard test using the generated simulated data,

[0811] A means of evaluating test results and providing those evaluation results to the user,

[0812] A means of recognizing the user's emotions and adjusting the method of presenting evaluation results based on those emotions,

[0813] A system that includes this.

[0814] (Claim 2)

[0815] The system according to claim 1, comprising means for generating and providing multiple situations based on analyzed information.

[0816] (Claim 3)

[0817] The system according to claim 1, comprising means for generating opinions based on evaluation results and for the user to use them to improve software for controlling transportation equipment.

[0818] "Application example 2 when combining with an emotional engine"

[0819] (Claim 1)

[0820] A means of collecting and analyzing the latest real-world information,

[0821] A means for generating vehicle simulation information based on collected information,

[0822] A means of conducting a baseline test using the generated simulation information,

[0823] A means of evaluating test results and providing those evaluation results to users,

[0824] A means for generating emotional information using an emotional recognition engine that recognizes the user's emotions, and for adjusting the information presentation means based on that emotional information,

[0825] A system that includes this.

[0826] (Claim 2)

[0827] The system according to claim 1, comprising means for creating and providing multiple situations based on analyzed information.

[0828] (Claim 3)

[0829] The system according to claim 1, further comprising means for generating feedback based on evaluation results and for the user to use it to improve vehicle control software. [Explanation of Symbols]

[0830] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of collecting and analyzing the latest real-world data, A means for generating automotive simulation data based on collected data, A means of conducting benchmark tests using the generated simulation data, A means of evaluating test results and providing those evaluation results to the user, A system that includes this.

2. The system according to claim 1, comprising means for creating and providing multiple scenarios based on analyzed data.

3. The system according to claim 1, further comprising means for generating feedback based on evaluation results and for the user to use it to improve autonomous driving software.

Images