system

Abstract

To provide a system.SOLUTION: A system includes means for a user to input a prompt sentence, means for a terminal to transmit input data of the prompt sentence to a server, means for the server to analyze the data of the prompt sentence using a generative AI model, means for the server to generate a virtual object based on analysis results, means for the server to input the generated virtual object into a digital twin, means for the digital twin to perform a simulation, means for the server to transmit simulation results to the terminal, and means for the terminal to display the simulation results to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Autonomous driving AI testing needs to be extensive and thorough using a variety of realistic scenarios, but replicating and testing all of these scenarios in real-world environments is time-consuming and costly. [Means for solving the problem]

[0005] By using generative AI to generate virtual vehicles, people, and environments from natural language and video, and then inputting these into a digital twin, highly realistic and diverse simulations can be performed. This enables extensive and thorough testing of autonomous driving AI. Furthermore, it allows for more countermeasures for the various scenarios that autonomous vehicles may encounter on the road, significantly reducing the time and cost involved in testing. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment Example 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] In one embodiment of the present invention, generative AI receives and analyzes information such as natural language and video as input. Based on the results of the analysis, it generates virtual vehicles, people, and environments. The generated virtual vehicles, people, and environments are then fed into a digital twin, which uses them to perform highly realistic and diverse simulations. For example, it can recreate various scenarios that an autonomous vehicle may encounter, such as urban and suburban environments, various weather conditions, and traffic situations.

[0029] "Example 2"

[0030] In another embodiment of the present invention, the Generative AI generates scenarios for testing the autonomous driving AI. The generated scenarios are designed to cover a variety of scenarios that an autonomous vehicle may encounter on the road. This allows the autonomous driving AI to be extensively and thoroughly tested and improved or adjusted based on the results. For example, the autonomous vehicle can handle a variety of scenarios, such as turning right at an intersection, a pedestrian suddenly stepping out, or approaching another vehicle.

[0031] "Example 3"

[0032] In a further embodiment of the present invention, simulation using generative AI and digital twins can significantly reduce the time and cost associated with testing autonomous driving AI. Specifically, simulation testing can be performed more quickly and efficiently than testing in real-world environments, and scenarios can be repeatedly tested as needed. This accelerates the process of developing and improving autonomous driving AI, thereby improving the safety and reliability of autonomous vehicles.

[0033] The processing flow of each embodiment will be described below.

[0034] "Example 1"

[0035] Step 1: The generative AI receives information such as natural language and video as input.

[0036] Step 2: The generation AI analyzes the input information and generates virtual cars, people, and environments based on the results of the analysis.

[0037] Step 3: The generated virtual vehicles, humans, and environments are fed into a digital twin.

[0038] Step 4: The digital twin performs highly realistic and diverse simulations using virtual vehicles, people, and environments.

[0039] "Example 2"

[0040] Step 1: The generation AI generates scenarios for testing the autonomous driving AI.

[0041] Step 2: The generated scenarios are designed to cover a variety of scenarios that an autonomous vehicle may encounter on the road.

[0042] Step 3: The self-driving AI is extensively and thoroughly tested using generated scenarios.

[0043] Step 4: Based on the results of the testing, improvements and adjustments are made to the self-driving AI.

[0044] "Example 3"

[0045] Step 1: Using generative AI and digital twin simulations to significantly reduce the time and cost associated with testing autonomous driving AI.

[0046] Step 2: Simulation testing is fast and efficient, allowing scenarios to be tested repeatedly as needed.

[0047] Step 3: This will accelerate the process of developing and improving self-driving AI, thereby improving the safety and reliability of self-driving cars.

[0048] Example 1

[0049] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0050] In the development of autonomous driving technology, testing in actual road environments is time-consuming and costly, and it is difficult to ensure safety. Furthermore, to fully implement countermeasures for various scenarios, simulations under a variety of environments and conditions are required, but there is a lack of efficient means to carry this out.

[0051] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0052] In this invention, the server includes a means for a user to input a prompt sentence, a means for the terminal to send the input data to the server, a means for the server to analyze the data using a generative AI model, a means for the server to generate a virtual object based on the analysis results, a means for the server to input the generated virtual object into a digital twin, a means for the digital twin to perform a simulation, a means for the server to send the simulation results to the terminal, and a means for the terminal to display the simulation results to the user. This enables advanced and diverse simulations that reproduce actual road environments, reducing the time and cost in the development of autonomous driving technology, improving safety, and strengthening countermeasures for various scenarios.

[0053] A "user" is an entity that uses the system to input prompt statements and check the simulation results.

[0054] A "terminal" is a device through which a user enters prompts and communicates with a server.

[0055] A "server" is a computer system that has the ability to analyze data using generative AI models, generate virtual objects, and input them into a digital twin.

[0056] A "generative AI model" is an artificial intelligence model that analyzes information such as natural language and video and generates virtual objects.

[0057] A "prompt sentence" is an input sentence that describes in natural language the scenario that the user wants to simulate.

[0058] "Virtual objects" are digital data such as virtual cars, people, and environments generated by generative AI models.

[0059] A "digital twin" is a simulation system that uses virtual objects to recreate real-world environments and situations with a high degree of realism.

[0060] "Simulation" is the process of using a digital twin to recreate the behavior and interactions of virtual objects and test various scenarios.

[0061] "Simulation results" are data and information obtained as a result of a simulation performed by a digital twin.

[0062] This invention begins when a user inputs a prompt and the device sends the data to a server. The server analyzes the data using a generative AI model and generates a virtual object. The generated virtual object is then input into a digital twin, which then uses it to perform a simulation. The simulation results are then sent from the server to the device and ultimately displayed to the user.

[0063] Specifically, the user uses a terminal to input a prompt statement, which is a natural language description of the scenario they want to simulate. For example, the prompt statement might be, "I want to simulate an autonomous vehicle in an urban environment on a rainy day," or "I want to simulate an autonomous vehicle in a mountainous area on a snowy day."

[0064] The device sends the prompt text entered by the user to the server. At this time, the device appropriately converts the data format and sends it according to the communication protocol. The server inputs the received prompt text into a generative AI model (for example, a natural language processing model or an image generation model) for analysis. The generative AI model understands the content of the prompt text and extracts the necessary information.

[0065] The server generates virtual vehicles, people, and environments based on the analysis results. In this process, the server uses image generation models to create realistic virtual objects. The generated virtual objects are then input into the digital twin system, which then prepares the system for simulation using these virtual objects.

[0066] A digital twin uses virtual objects to perform simulations, such as recreating the behavior of an autonomous vehicle in an urban environment on a rainy day. The results show how the vehicle behaves in the rain and how it interacts with other vehicles and pedestrians.

[0067] The server receives the results of the simulation performed by the digital twin and sends them to the terminal. The simulation results include the movement of the car and changes in the environment. The terminal displays the simulation results received from the server to the user. The user can check the simulation results and re-enter prompt statements if necessary.

[0068] This system enables advanced and diverse simulations that replicate actual road environments, reducing the time and cost required to develop autonomous driving technology, improving safety, and strengthening countermeasures for a variety of scenarios.

[0069] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0070] Step 1:

[0071] The user enters a prompt statement.

[0072] Specifically, the user describes the scenario they want to simulate in natural language in the input field of the device. For example, they might enter, "I want to simulate an autonomous vehicle in an urban environment on a rainy day."

[0073] Input: The prompt text entered by the user

[0074] Output: The prompt text entered in the terminal

[0075] Step 2:

[0076] The terminal sends the input data to the server.

[0077] Specifically, the terminal converts the prompt text entered by the user into an appropriate data format and sends it to the server in accordance with the communication protocol.

[0078] Input: The prompt text entered into the terminal

[0079] Output: The prompt sent to the server

[0080] Step 3:

[0081] The server analyzes the data using a generative AI model.

[0082] Specifically, the server inputs the received prompt into a generative AI model (e.g., a natural language processing model) and analyzes the content of the prompt. The generative AI model then understands the content of the prompt and extracts the necessary information.

[0083] Input: The prompt sent to the server

[0084] Output: Analysis results (required information)

[0085] Step 4:

[0086] The server generates a virtual object based on the analysis results.

[0087] Specifically, the server generates virtual vehicles, people, and environments based on the analysis results, using image generation models to create realistic virtual objects.

[0088] Input: Analysis results (required information)

[0089] Output: Generated virtual object

[0090] Step 5:

[0091] The virtual objects generated by the server are then inserted into the digital twin.

[0092] Specifically, the server inputs the generated virtual object into the digital twin system and prepares for the simulation.

[0093] Input: Generated virtual object

[0094] Output: Virtual objects fed into the digital twin

[0095] Step 6:

[0096] The digital twin performs the simulation.

[0097] Specifically, the digital twin uses virtual objects to perform simulations, such as recreating the behavior of an autonomous vehicle in an urban environment on a rainy day.

[0098] Input: Virtual objects fed into the digital twin

[0099] Output: Simulation results

[0100] Step 7:

[0101] The server transmits the simulation results to the terminal.

[0102] Specifically, the server receives the results of the simulation performed by the digital twin and sends them to the terminal.

[0103] Input: Simulation results

[0104] Output: Simulation results sent to the terminal

[0105] Step 8:

[0106] The terminal displays the simulation results to the user.

[0107] Specifically, the terminal displays the simulation results received from the server to the user, who can then check them and re-enter the prompt sentence if necessary.

[0108] Input: Simulation results sent to the terminal

[0109] Output: Simulation results displayed to the user

[0110] (Application example 1)

[0111] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0112] In the development of autonomous vehicles, testing in real road environments is time-consuming and costly, and it is difficult to cover all scenarios. In addition, there is a lack of means for users to simulate specific scenarios and check the behavior of autonomous vehicles.

[0113] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0114] In this invention, the server includes means for generating virtual cars, people, and environments from natural language and video using a generative AI and inputting them into a digital twin, means for implementing highly realistic and diverse simulations using the digital twin, means for extensively and thoroughly testing autonomous driving AI using the simulations, means for a user to input natural language and video and implement a simulation using the generated virtual elements, and means for providing the results of the simulation to the user, thereby enabling the user to simulate various scenarios and check the behavior of an autonomous driving car.

[0115] "Generative AI" is an artificial intelligence technology that analyzes information such as natural language and video to generate virtual cars, people, and environments.

[0116] A "digital twin" is a digital model that recreates a real-world physical object or environment in a virtual space.

[0117] "Simulation" is the process of recreating various real-world scenarios using virtual vehicles, people, and environments to verify their behavior.

[0118] "Autonomous driving AI" is an artificial intelligence technology used to control and make decisions about autonomous vehicles.

[0119] A "user" is a person or organization that uses the system to input natural language or video and perform simulations.

[0120] "Virtual elements" refer to virtual cars, people, and environments generated by generative AI.

[0121] "Simulation results" refers to the data and information obtained when a simulation is performed.

[0122] A system for implementing this invention includes elements such as generative AI, digital twin, simulation, autonomous driving AI, user, virtual element, and simulation results.

[0123] The server uses generative AI to analyze information such as natural language and video to generate virtual vehicles, people, and environments. These virtual elements are then fed into a digital twin, which then uses them to perform highly realistic and diverse simulations. Simulations enable extensive and thorough testing of autonomous driving AI.

[0124] Users input natural language or video using a device such as a smartphone. The input information is sent to a server and analyzed by generative AI. As a result of the analysis, a virtual car, human, and environment are generated. The generated virtual elements are input into a digital twin, where a simulation is performed. The simulation results are provided to the user, who can simulate various scenarios and check the behavior of the autonomous vehicle.

[0125] For example, if a user requests, "I want to simulate an autonomous vehicle in an urban environment on a sunny day," the generative AI will recreate the urban environment on a sunny day and run a simulation in the digital twin. The simulation results will be provided to the user, who can then check the behavior of the autonomous vehicle.

[0126] The hardware used includes smartphones and servers. The software used includes Python and OpenAI (registered trademark) API. Data processing and calculation include information analysis using generative AI, simulation using digital twins, and provision of simulation results.

[0127] Examples of prompts include:

[0128] "I want to simulate an autonomous vehicle in an urban environment on a sunny day."

[0129] "I want to simulate an autonomous vehicle in a suburban environment on a rainy day."

[0130] In this way, users can simulate various scenarios and see how the autonomous vehicle will behave.

[0131] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0132] Step 1:

[0133] The user inputs natural language or video using a device such as a smartphone. The input information is sent from the device to the server. The input data is a prompt sentence, for example, "I would like to simulate an autonomous vehicle in an urban environment on a sunny day."

[0134] Step 2:

[0135] The server passes the received natural language and video information to the generation AI. The generation AI analyzes the input data and generates virtual cars, people, and environments. Specifically, it uses the OpenAI API to analyze the input prompt sentences and generate virtual elements. The output is data on the generated virtual cars, people, and environments.

[0136] Step 3:

[0137] The server inputs the generated virtual elements into a digital twin. A digital twin is a digital model that recreates physical objects and environments in a virtual space, and performs a simulation using the generated virtual elements. The input is the data of the generated virtual elements, and the output is the simulation results.

[0138] Step 4:

[0139] The server analyzes the results of the simulations performed by the digital twin and generates data to provide to users. Specifically, it organizes the simulation results and converts them into a format that is easy for users to understand. The input is the simulation result data, and the output is the organized data to provide to users.

[0140] Step 5:

[0141] The server sends the organized simulation results to the user's device, where the user can check the simulation results and evaluate the behavior of the autonomous vehicle. The input is the organized simulation result data, and the output is the simulation results displayed on the user's device.

[0142] In this way, users can simulate various scenarios and see how the autonomous vehicle will behave.

[0143] Example 2

[0144] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0145] When testing autonomous driving AI, it takes a lot of time and money to recreate real-world road conditions. Furthermore, it is difficult to completely recreate real-world road conditions, which often limits the scope of testing. As a result, there is a problem in that it is not possible to adequately prepare countermeasures for the diverse scenarios that autonomous driving AI may encounter in a real driving environment.

[0146] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0147] In this invention, the server includes means for generating virtual vehicles, people, and environments from natural language, video, etc. using a generative AI model and inputting them into a digital twin, means for implementing highly realistic and diverse simulations using the digital twin, means for extensively and thoroughly testing the autonomous driving AI using the simulations, means for generating scenarios by inputting prompt sentences, means for saving the generated scenarios, means for testing the autonomous driving AI using the saved scenarios, and means for recording the test results. This allows the autonomous driving AI to be tested so that it can handle a variety of scenarios, and allows for improvements and adjustments based on the test results.

[0148] A "generative AI model" is an artificial intelligence technology that generates virtual vehicles, people, and environments from input data such as natural language and video.

[0149] A "digital twin" is a digital model that recreates a real-world physical system or environment in a virtual space.

[0150] "Simulation" is the process of using a digital twin to virtually recreate and test real-world road conditions and driving scenarios.

[0151] "Autonomous driving AI" is an artificial intelligence technology used to control the driving of self-driving cars.

[0152] A "prompt" is an instruction entered into a generative AI model to generate a specific scenario.

[0153] A "scenario" is a hypothetical situational setting that describes a specific driving situation or environment that an autonomous vehicle might encounter.

[0154] "Saving" is the act of recording the generated scenario in a database or file system.

[0155] "Test results" are data on the behavior and performance of autonomous driving AI obtained when running a simulation.

[0156] This invention is a system that generates scenarios for testing autonomous driving AI using a generative AI model and performs simulations using a digital twin. Specific embodiments of this system are described below.

[0157] System configuration

[0158] Hardware

[0159] Server: Use a server with high-performance computing capabilities. Specifically, a server equipped with a GPU is preferable.

[0160] Device: The user uses a device such as a computer or tablet to enter the prompt.

[0161] software

[0162] Generative AI model: An artificial intelligence technology for generating virtual vehicles, people, and environments from input data such as natural language and video. Specifically, it uses machine learning libraries such as TENSORFLOW (registered trademark) and PyTorch.

[0163] Digital twin: Software for recreating real-world physical systems and environments in virtual space. Game engines such as Unity and Unreal Engine can be used.

[0164] Database: A database for saving the generated scenarios. Specifically, a database management system such as MySQL (registered trademark) or MongoDB is used.

[0165] System Operation

[0166] Entering a prompt statement

[0167] The user inputs a prompt sentence to the generative AI model using the terminal. For example, the user inputs the prompt sentence "Please generate a right turn scenario at an intersection."

[0168] Scenario Generation Execution

[0169] The server passes the received prompt sentence to the generative AI model, which then generates a detailed scenario based on the prompt sentence. For example, the generated scenario may include the shape of the intersection, the status of traffic lights, and the movements of other vehicles and pedestrians.

[0170] Saving a Scenario

[0171] The server saves the generated scenarios in JSON format, which are later used to test the autonomous driving AI.

[0172] Test run of the scenario

[0173] The server runs tests on the self-driving AI using the saved scenarios, and the test results are logged and later analyzed.

[0174] Specific examples

[0175] As a concrete example, the following scenario is generated:

[0176] A scenario in which an autonomous vehicle is turning right at an intersection and an oncoming vehicle is coming straight ahead.

[0177] Scenario where a pedestrian suddenly jumps out onto the crosswalk

[0178] Scenarios where other vehicles suddenly change lanes

[0179] Prompt Sentence Examples

[0180] "Generate a right turn scenario at an intersection"

[0181] "Generate a scenario where a pedestrian suddenly jumps out."

[0182] "Generate a scenario where another vehicle suddenly changes lanes."

[0183] This system allows the self-driving AI to be tested to handle a variety of scenarios, allowing it to be improved and adjusted based on the results.

[0184] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0185] Step 1:

[0186] The user inputs a prompt sentence to the generative AI model using a terminal. For example, the user inputs a prompt sentence such as "Please generate a right turn scenario at an intersection."

[0187] Input: prompt statement

[0188] Output: The prompt is sent to the server.

[0189] Specific operation: The user uses the terminal keyboard to enter a prompt sentence and clicks the send button.

[0190] Step 2:

[0191] The server passes the received prompt sentence to the generative AI model, which then generates a detailed scenario based on the prompt sentence.

[0192] Input: prompt statement

[0193] Output: Generated scenario

[0194] Specific operation: The server inputs the prompt sentence into the generative AI model, which then generates a scenario including details such as the shape of the intersection, the status of traffic lights, and the movement of other vehicles and pedestrians.

[0195] Step 3:

[0196] The server saves the generated scenarios in JSON format, which are later used to test the autonomous driving AI.

[0197] Input: Generated scenario

[0198] Output: JSON format scenario file

[0199] Specific operation: The server converts the generated scenario into JSON format and saves it in a database or file system.

[0200] Step 4:

[0201] The server runs tests on the self-driving AI using the saved scenarios, and the test results are logged and later analyzed.

[0202] Input: JSON format scenario file

[0203] Output: Test result log file

[0204] Specific operation: The server loads the saved scenario, executes the scenario for the autonomous driving AI, and records the test results in a log file.

[0205] (Application example 2)

[0206] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0207] As autonomous driving technology advances, there is an increasing need to test the performance of autonomous driving AI extensively and thoroughly. However, testing in real-world road environments is time-consuming and costly, and ensuring safety is also a challenge. Furthermore, there is a lack of a means to efficiently visualize generated scenarios and propose improvements to autonomous driving AI based on the results. To solve these challenges, a more efficient and comprehensive test system is needed.

[0208] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0209] In this invention, the server includes means for generating virtual cars, people, and environments from natural language, videos, etc. using a generative AI and inputting them into a digital twin, means for implementing highly realistic and diverse simulations using the digital twin, means for extensively and thoroughly testing the autonomous driving AI using the simulations, means for simulating the generated scenarios on a smartphone and visualizing the results, and means for proposing improvements to the autonomous driving AI based on the results of the scenarios, thereby enabling efficient and comprehensive testing of the performance of the autonomous driving AI and rapid identification of areas for improvement.

[0210] "Generative AI" is an artificial intelligence technology that generates virtual cars, people, and environments from natural language, videos, etc.

[0211] A "digital twin" is a digital model that recreates a real-world physical object or environment in a virtual space.

[0212] "Simulation" is a method of using digital twins to virtually recreate highly realistic and diverse situations and conduct experiments and tests.

[0213] "Autonomous driving AI" is an artificial intelligence technology that enables cars to drive autonomously.

[0214] A "smartphone" is a mobile information terminal that has the functions of a computer in addition to the functions of a mobile phone.

[0215] "Visualization" is a technique for visually displaying data and simulation results to make them easier to understand.

[0216] "Areas for improvement" are corrections and refinements that are identified based on the simulation results to improve the performance of the autonomous driving AI.

[0217] The system for implementing this invention includes elements of generative AI, digital twin, simulation, autonomous driving AI, smartphone, visualization, and improvement. Specific embodiments are described below.

[0218] The server uses generative AI to generate virtual cars, people, and environments from natural language and video. The generated virtual cars, people, and environments are then input into a digital twin, a digital model that recreates physical objects and environments in the real world in a virtual space.

[0219] The server then uses the digital twin to perform a variety of highly realistic simulations, a method of experimenting and testing in virtually recreated situations, used to extensively and thoroughly test the performance of autonomous driving AI.

[0220] The generated scenario is simulated on a smartphone, and the results are visualized. A smartphone is a mobile information terminal that has the functionality of a computer in addition to a mobile phone. Visualization is a technique for visually displaying data and simulation results to make them easier to understand.

[0221] Users can identify and propose improvements to the autonomous driving AI based on the simulation results. Improvements are corrections and improvements that can be made to improve the performance of the autonomous driving AI, as identified based on the simulation results.

[0222] For example, you can generate a scenario by inputting the following prompt into the generation AI:

[0223] "Generate a scenario in which an autonomous vehicle makes a right turn at an intersection."

[0224] "Generate a lane change scenario for an autonomous vehicle on a highway."

[0225] "Generate a scenario in which an autonomous vehicle drives through a residential area at night."

[0226] "Generate a scenario in which an autonomous vehicle drives through urban areas in the rain."

[0227] This allows users to test autonomous driving AI against a variety of scenarios and evaluate its performance efficiently and comprehensively.

[0228] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0229] Step 1:

[0230] The server uses generative AI to generate virtual cars, people, and environments from natural language, videos, etc.

[0231] Input: natural language and video data

[0232] Data processing: Generative AI models analyze input data and generate virtual cars, people, and environments.

[0233] Output: Digital data of virtual vehicles, humans, and environments

[0234] Step 2:

[0235] The server then inputs the generated virtual vehicles, people, and environments into the digital twin.

[0236] Input: Digital data of virtual vehicles, humans, and environments

[0237] Data processing: Integrating virtual vehicles, humans and environments into the digital twin.

[0238] Output: Virtual environment integrated into a digital twin

[0239] Step 3:

[0240] The server uses the digital twin to perform highly realistic and diverse simulations.

[0241] Input: Virtual environment integrated into the digital twin

[0242] Data calculation: The simulation engine calculates the behavior in the virtual environment and executes the scenario.

[0243] Output: Simulation result data

[0244] Step 4:

[0245] The server simulates the generated scenario on a smartphone and visualizes the results.

[0246] Input: Simulation result data

[0247] Data processing: Generate graphs and charts to visually display the simulation results.

[0248] Output: Visualized data displayed on a smartphone

[0249] Step 5:

[0250] Users will identify and propose improvements to the autonomous driving AI based on the simulation results.

[0251] Input: Visualized data displayed on a smartphone

[0252] Data Computation: Users analyze simulation results and identify areas for improvement.

[0253] Output: Proposal data for improvements to the autonomous driving AI

[0254] This allows users to test autonomous driving AI against a variety of scenarios and evaluate its performance efficiently and comprehensively.

[0255] Example 3

[0256] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0257] In the development of autonomous driving AI, testing in real-world environments is time-consuming and costly, making it difficult to conduct efficient testing. Furthermore, it is difficult to repeatedly test reproducible scenarios in real-world environments, making it difficult to adequately prepare for various scenarios.

[0258] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means. In this invention, the server includes: means for generating virtual objects, people, and environments from natural language, videos, etc. using a generative AI and inputting them into a digital twin; means for performing highly realistic and diverse simulations using the digital twin; means for extensively and thoroughly testing autonomous driving AI using the simulations; means for a user to set a test scenario using a terminal; means for the server to generate a simulation environment using a generative AI model; means for the server to execute a simulation on the digital twin platform; means for the server to analyze the test results; and means for the user to modify the scenario and retest. This significantly reduces the time and cost involved in testing and enables repeated testing of reproducible scenarios.

[0259] "Generative AI" is an artificial intelligence technology that generates virtual objects, people, and environments from input data such as natural language, images, and videos.

[0260] "Digital twin" is a technology that recreates real-world objects and environments in a virtual space and performs simulations and analyses.

[0261] A "simulation environment" is a virtual space for testing and analysis that includes virtual objects, people, and environments generated by generative AI.

[0262] "Autonomous driving AI" is an artificial intelligence technology for automating automobile driving.

[0263] A "terminal" is a device such as a computer or smartphone that is operated by a user.

[0264] A "server" is a high-performance computer system that runs generative AI models, manages simulations, and analyzes data.

[0265] A "test scenario" is a scenario that includes specific conditions or situations set up to evaluate the performance of autonomous driving AI.

[0266] A "generative AI model" is a specific implementation of generative AI, an algorithm or program for generating virtual objects, people, and environments from input data such as natural language, images, and videos.

[0267] A "prompt" is an instruction entered into a generative AI model to cause it to generate a specific output.

[0268] A "digital twin platform" is software and systems that realize digital twin technology.

[0269] MODE FOR CARRYING OUT THE INVENTION

[0270] This invention is a system that significantly reduces the time and cost involved in testing autonomous driving AI through simulations using generative AI and digital twins. This system involves a series of processes: a user sets up a test scenario using a terminal, a server generates a simulation environment using a generative AI model, runs the simulation on a digital twin platform, and analyzes the test results.

[0271] Hardware and software used

[0272] Hardware: High-performance servers (e.g., high-performance computer systems)

[0273] Software: Generative AI models (e.g., advanced generative AI algorithms), digital twin platforms (e.g., simulation software)

[0274] Data processing and calculation flow

[0275] 1. The user sets up a specific test scenario using the device. For example, the user selects "changing lanes on a highway in the rain" as the scenario. Detailed parameters such as traffic volume and weather conditions are entered through the device interface.

[0276] 2. Based on the scenario settings received from the user, the server uses a generative AI model to generate a detailed simulation environment. The generative AI model then builds a realistic simulation environment based on the input conditions.

[0277] 3. The server imports the generated simulation environment into the digital twin platform and runs autonomous driving AI tests. The simulation is performed in real time, and vehicle behavior and changes in the environment are recorded in detail.

[0278] 4. The server analyzes the simulation results and evaluates the performance and problems of the autonomous driving AI. The analysis results are provided to the user in the form of graphs and reports.

[0279] 5. The user modifies the scenario as needed based on the analysis results. After modification, the user runs the simulation again and checks the results. By repeating this process, the performance of the autonomous driving AI is optimized.

[0280] Specific examples

[0281] For example, if a user sets "changing lanes on a highway in the rain" as a test scenario, they would input the following prompt sentence into the generative AI model:

[0282] Example prompt sentence:

[0283] "Generate a lane change scenario on a highway in the rain. Specifically, simulate a lane change in heavy traffic and record the vehicle's behavior in detail."

[0284] When this prompt is entered, the generative AI model creates a detailed simulation environment and runs autonomous driving AI tests on the digital twin platform. The test results are analyzed by the server and fed back to the user.

[0285] The above is an embodiment of the present invention. This system significantly reduces the time and cost involved in testing, and enables repeatable testing of scenarios. The flow of the identification process in the third embodiment will be described with reference to FIG. 15.

[0286] Step 1:

[0287] The user sets up a test scenario using the device. Through the device interface, the user inputs detailed parameters such as specific traffic and weather conditions. For example, the user selects "changing lanes on a highway in rainy weather" as a scenario. The input data is then sent to the server.

[0288] Input: Detailed parameters of the test scenario (e.g., rain, highway, lane change)

[0289] Output: Scenario configuration data sent to the server

[0290] Specific behavior:

[0291] The user operates the GUI on the terminal to open the scenario setting screen.

[0292] The user inputs conditions such as "rainy weather," "highway," and "lane change."

[0293] Step 2:

[0294] Based on the scenario settings received from the user, the server uses a generative AI model to generate a detailed simulation environment. The generative AI model builds a realistic simulation environment based on the input conditions. A prompt statement is input into the generative AI model to generate the simulation environment.

[0295] Input: Scenario setting data, prompt text

[0296] Output: Generated simulation environment data

[0297] Specific behavior:

[0298] The server receives input data from the user.

[0299] The server inputs prompt statements into the generated AI model and generates a simulation environment.

[0300] Step 3:

[0301] The server then imports the generated simulation environment into the digital twin platform and runs autonomous driving AI tests. The simulation is performed in real time, and vehicle behavior and changes in the environment are recorded in detail.

[0302] Input: Generated simulation environment data

[0303] Output: Simulation execution result data

[0304] Specific behavior:

[0305] The server imports the generated simulation environment into the digital twin platform.

[0306] The server runs the autonomous driving AI in a simulated environment and collects data.

[0307] Step 4:

[0308] The server analyzes the simulation results and evaluates the performance and problems of the autonomous driving AI. The analysis results are provided to the user in the form of graphs and reports.

[0309] Input: Simulation execution result data

[0310] Output: Analysis result data (graphs, reports)

[0311] Specific behavior:

[0312] The server runs algorithms that analyze the simulation data.

[0313] The server visualizes the analysis results and generates a report.

[0314] Step 5:

[0315] The user modifies the scenario as necessary based on the analysis results. After modification, the user runs the simulation again and checks the results. By repeating this process, the performance of the autonomous driving AI is optimized.

[0316] Input: Analysis result data, modified scenario setting data

[0317] Output: New simulation run results data

[0318] Specific behavior:

[0319] The user uses the terminal to check the analysis results.

[0320] The user corrects the scenario and runs the test again.

[0321] (Application example 3)

[0322] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0323] In the development of autonomous driving AI, testing in real-world environments is time-consuming, costly, and difficult to ensure safety. Furthermore, extensive and thorough testing is required to adequately address a wide range of scenarios, which is also difficult to achieve in real-world environments. To address these challenges, a means is needed to efficiently and safely test autonomous driving AI.

[0324] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0325] In this invention, the server includes means for generating virtual cars, people, and environments from natural language, video, etc. using a generative AI and inputting them into a digital twin, means for implementing highly realistic and diverse simulations using the digital twin, means for extensively and thoroughly testing autonomous driving AI using the simulations, means for running simulation tests on a smartphone, and means for evaluating the simulation results and calculating safety and efficiency scores. This makes it possible to efficiently and safely test autonomous driving AI and quickly take measures for a variety of scenarios.

[0326] "Generative AI" is an artificial intelligence technology that generates virtual cars, people, and environments from data such as natural language and video.

[0327] "Digital twin" is a technology that recreates real-world physical environments and objects in virtual space.

[0328] "Simulation" is a method of recreating real-world environments and scenarios in a virtual space for testing and evaluation.

[0329] "Autonomous driving AI" is an artificial intelligence technology that enables autonomous driving of vehicles.

[0330] A "smartphone" is a mobile device that has the functions of a computer in addition to the functions of a mobile phone.

[0331] A "simulation test" is a test conducted using a simulation in a virtual space.

[0332] "Safety" is the ability of a system or technology to operate without accidents or failures.

[0333] "Efficiency" is the ability of a system or technology to achieve maximum results with minimum resources.

[0334] The "score" is a numerical representation of the evaluation result.

[0335] The system for implementing this invention utilizes generative AI and digital twin technology to perform simulation tests of autonomous driving AI. Specific embodiments are described below.

[0336] First, the server uses generative AI to generate virtual cars, people, and environments from data such as natural language and video. This generated data is then input into a digital twin, which recreates real-world physical environments and objects in a virtual space.

[0337] The digital twin is then subjected to a variety of highly realistic simulations, allowing the autonomous driving AI to be tested extensively and thoroughly. The simulations run on a smartphone, allowing users to set up various scenarios and repeatedly test them.

[0338] The simulation results are evaluated by the server and safety and efficiency scores are calculated, allowing users to evaluate the performance of their autonomous driving AI and make any necessary improvements.

[0339] Hardware and software used:

[0340] Hardware: Smartphone (iOS or ANDROID (registered trademark))

[0341] Software: Python, TensorFlow, Digital Twin Library

[0342] Data processing and calculation:

[0343] The server uses generative AI to generate virtual vehicles, people, and environments from natural language and video. The generated data is input into a digital twin, where a simulation is conducted in the virtual space. The simulation results are evaluated by the server, and a safety and efficiency score is calculated.

[0344] Examples:

[0345] For example, to test an urban driving scenario, the following prompt sentences are input to the generative AI model:

[0346] Example prompt sentence:

[0347] "Simulate urban driving scenarios. The scenarios include traffic lights, pedestrians, and other vehicles. The AI ​​model needs to drive safely and efficiently."

[0348] A user opens the AutoDrive Sim app on their smartphone and selects an urban driving scenario. The app loads the scenario and starts the simulation. When the simulation is finished, a safety and efficiency score is displayed, allowing the user to evaluate the AI ​​model's performance.

[0349] In this way, it becomes possible to efficiently and safely test autonomous driving AI and quickly develop countermeasures for a variety of scenarios.

[0350] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0351] Step 1:

[0352] The server uses generative AI to generate virtual cars, people, and environments from natural language and video.

[0353] Input: Natural language and video data

[0354] Data processing: Generative AI models are used to generate virtual vehicles, people, and environments from input data.

[0355] Output: Virtual vehicle, human, and environment data

[0356] Step 2:

[0357] The server inputs data about the generated virtual cars, people, and environment into the digital twin.

[0358] Input: Virtual vehicle, human, and environment data

[0359] Data processing: Input data into the digital twin and recreate it in virtual space.

[0360] Output: Virtual environment in the digital twin

[0361] Step 3:

[0362] The server uses the digital twin to perform highly realistic and diverse simulations.

[0363] Input: Virtual environment in the digital twin

[0364] Data processing: Using a simulation engine to run scenarios in a virtual environment.

[0365] Output: Simulation result data

[0366] Step 4:

[0367] The terminal (smartphone) executes a simulation test.

[0368] Input: Simulation scenario data

[0369] Data processing: Run simulations on your smartphone and collect the results.

[0370] Output: Simulation result data

[0371] Step 5:

[0372] The server evaluates the simulation results and calculates a safety and efficiency score.

[0373] Input: Simulation result data

[0374] Data processing: Analyze the results data and calculate safety and efficiency scores.

[0375] Output: Safety score, efficiency score

[0376] Step 6:

[0377] Users can check the simulation results on their smartphones and evaluate the performance of the AI ​​model.

[0378] Input: Safety score, Efficiency score

[0379] Data processing: Display scores through a smartphone interface.

[0380] Output: User's evaluation result

[0381] The above is the flow of processing of the program of the system that realizes the application example.

[0382] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0383] "Example 1"

[0384] In one embodiment of the present invention, the generation AI uses an emotion engine that recognizes the user's emotions to generate virtual cars, people, and environments that correspond to the user's emotions. Specifically, the emotion engine analyzes the user's emotions from their facial expressions, tone of voice, and choice of words, and transmits the results to the generation AI. Based on the analysis results, the generation AI generates, for example, a virtual environment with bright weather if the user is happy, or a virtual environment with rain if the user is sad.

[0385] "Example 2"

[0386] The emotion engine also adjusts the simulation scenario according to the user's emotions. For example, if the user is angry, the emotion engine communicates this information to the digital twin, and the digital twin prioritizes simulating scenarios that are likely to anger the user, such as traffic congestion or rude driver behavior.

[0387] "Example 3"

[0388] Furthermore, the emotion engine adjusts the testing priorities of the autonomous driving AI based on the user's emotions. For example, if the user is feeling scared, the emotion engine will convey that information to the autonomous driving AI, and the autonomous driving AI will prioritize testing responses to accidents and dangerous situations. This enables more effective testing based on the user's emotions.

[0389] The processing flow of each embodiment will be described below.

[0390] "Example 1"

[0391] Step 1: The emotion engine analyzes the user's emotions based on their facial expressions, tone of voice, and choice of words.

[0392] Step 2: The emotion engine communicates the analysis results to the generative AI.

[0393] Step 3: Based on the analysis results, the AI ​​generates virtual cars, people, and environments according to the user's emotions. For example, if the user is happy, it generates a virtual environment with bright weather, and if the user is sad, it generates a virtual environment with rain.

[0394] "Example 2"

[0395] Step 1: The emotion engine analyzes the user's emotions.

[0396] Step 2: The emotion engine communicates the analysis results to the digital twin.

[0397] Step 3: The digital twin adjusts the simulation scenarios according to the user's emotions. For example, if the user is angry, it will prioritize scenarios that are likely to anger the user, such as traffic congestion or rude driver behavior.

[0398] "Example 3"

[0399] Step 1: The emotion engine analyzes the user's emotions.

[0400] Step 2: The emotion engine communicates the analysis results to the autonomous driving AI.

[0401] Step 3: The self-driving AI adjusts test priorities based on the user's emotions. For example, if the user is feeling scared, it will prioritize testing responses to accidents and dangerous situations.

[0402] Example 1

[0403] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0404] In the development of autonomous driving technology, testing in real road environments is time-consuming and costly, and it is difficult to cover all scenarios. Furthermore, it is not possible to provide a simulation environment that responds to the user's emotions, making it difficult to improve the user experience.

[0405] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0406] In this invention, the server includes: a means for generating virtual cars, people, and environments from natural language, video, etc. using a generative AI and inputting them into a digital twin; a means for conducting highly realistic and diverse simulations using the digital twin; a means for extensively and thoroughly testing autonomous driving AI using the simulation; a means for generating virtual cars, people, and environments based on a user's emotions using an emotion engine that recognizes the user's emotions; a means for the emotion engine to analyze the user's emotions from their facial expressions, tone of voice, choice of words, etc. and communicate the results to the generative AI; and a means for the generative AI to generate a virtual environment based on the analysis results. This reduces the time and cost required for testing in a real road environment and enables all scenarios to be covered. Furthermore, providing a simulation environment that responds to the user's emotions improves the user experience.

[0407] "Generative AI" is an artificial intelligence technology that analyzes information such as natural language and videos to generate virtual cars, people, and environments.

[0408] "Digital twin" is a technology that recreates physical objects and environments in a virtual space and performs simulations in real time.

[0409] "Simulation" is the process of replicating the behavior of autonomous vehicles and other elements in a virtual environment for testing and evaluation.

[0410] "Autonomous driving AI" is an artificial intelligence technology used to control and make decisions about autonomous vehicles.

[0411] An "emotion engine" is a technology that analyzes emotions from a user's facial expressions, tone of voice, choice of words, etc., and transmits the results to other systems.

[0412] "User emotions" refers to the psychological state that a user expresses through facial expressions, tone of voice, choice of words, etc.

[0413] A "virtual environment" is a simulated setting that includes elements such as cars, people, and the environment in a digital space generated by generative AI.

[0414] MODE FOR CARRYING OUT THE INVENTION

[0415] This invention is a system that uses a generative AI model to analyze information such as natural language and video to generate virtual cars, people, and environments. The generated virtual environment is input into a digital twin, where realistic and diverse simulations are performed. Furthermore, an emotion engine that recognizes the user's emotions can be used to generate a virtual environment that corresponds to the user's emotions.

[0416] Hardware and software used

[0417] The server uses Google Cloud's natural language processing API and OpenAI's GPT-4 as generative AI models. Game engines such as Unity and Unreal Engine are used to implement the digital twin. The emotion engine includes software for analyzing the user's facial expressions, tone of voice, and word choice.

[0418] Data processing and calculation

[0419] The server receives natural language and video input from the user and analyzes it using a generative AI model. Based on the analysis results, virtual cars, people, and environments are generated. The generated data is input into a digital twin, where a simulation is carried out in real time. The emotion engine analyzes the user's emotions and conveys the results to the generative AI. Based on the analysis results, the generative AI generates a virtual environment that corresponds to the user's emotions.

[0420] Specific examples

[0421] If a user inputs into the system, "I would like to simulate autonomous driving in an urban environment on a sunny day," the server will analyze this natural language and generate an urban environment on a sunny day. The generated environment is then input into the digital twin, and the simulation begins.

[0422] Furthermore, when using the emotion engine, emotions are analyzed from the user's facial expressions, tone of voice, choice of words, etc. For example, if a user says, "I'm in a good mood today," the emotion engine will interpret this as the user being happy, and the generation AI will generate a virtual environment with bright weather.

[0423] Prompt Sentence Examples

[0424] "I want to simulate autonomous driving in an urban environment on a sunny day."

[0425] "I want to simulate autonomous driving in a suburban environment on a rainy day."

[0426] "I'm feeling good today, so I want to see a bright weather simulation."

[0427] In this way, by having the server, terminal, and user work together to execute system processing, it is possible to reduce the time and cost required for testing in a real road environment and cover all scenarios.In addition, by providing a simulation environment that responds to the user's emotions, it is possible to improve the user experience.

[0428] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0429] Step 1:

[0430] The user provides input

[0431] The user inputs text into the system interface, such as "I would like to simulate autonomous driving in an urban environment on a sunny day," or speaks a similar message using voice input. The input data can be text data or voice data.

[0432] Step 2:

[0433] The server receives the input

[0434] The server receives text and voice input from the user. In the case of voice input, it first converts it into text using a speech recognition API. The input data is saved as text data.

[0435] Step 3:

[0436] The server parses the input

[0437] The server then sends the received text data to Google Cloud's natural language processing API and obtains the analysis results. The analysis uses natural language processing technology to understand the user's request. The analysis results include information about the simulation environment the user desires.

[0438] Step 4:

[0439] The server creates a virtual environment

[0440] The server uses Unity to generate a sunny urban environment based on the analysis results. Specifically, it creates a 3D model of the city and sets the weather conditions. The generated virtual environment data is then input into the digital twin.

[0441] Step 5:

[0442] The server populates the digital twin with a virtual environment

[0443] The server sends the generated urban environment data to the digital twin platform, which prepares the simulation in real time.Digital twin is a platform for simulating virtual environments in real time.

[0444] Step 6:

[0445] The server runs the simulation

[0446] The server then runs a simulation of the autonomous vehicle on the digital twin, simulating, for example, how the vehicle behaves when passing through an intersection and interacts with other vehicles and pedestrians. Simulation results are generated in real time.

[0447] Step 7:

[0448] The terminal displays the simulation results.

[0449] The device displays the simulation results sent from the server in real time, allowing the user to see how the autonomous vehicle navigates through an urban environment on the device screen. The displayed data includes the progress and results of the simulation.

[0450] In this way, by having the server, terminal, and user work together to execute system processing, it is possible to reduce the time and cost required for testing in a real road environment and cover all scenarios.In addition, by providing a simulation environment that responds to the user's emotions, it is possible to improve the user experience.

[0451] (Application example 1)

[0452] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0453] In the development of autonomous vehicles, testing in real road environments is not only time-consuming and costly, but can also be dangerous. Furthermore, it is difficult to reproduce various scenarios and weather conditions, limiting the scope of testing. Furthermore, it is not possible to provide a real-time simulation environment that responds to user emotions, so there is a need to improve the user experience.

[0454] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0455] In this invention, the server includes: means for generating virtual cars, people, and environments from natural language, video, etc. using a generative AI and inputting them into a digital twin; means for implementing highly realistic and diverse simulations using the digital twin; means for extensively and thoroughly testing autonomous driving AI using the simulations; means for analyzing user emotions using an emotion engine and generating a virtual environment based on the analysis results; and means for displaying the simulation results in real time using a smartphone or head-mounted display and allowing the user to operate an autonomous driving vehicle in the virtual environment. This makes it possible to reproduce various scenarios and weather conditions and provide a real-time simulation environment that responds to the user's emotions.

[0456] "Generative AI" is an artificial intelligence that analyzes information such as natural language and videos, and generates virtual cars, people, and environments.

[0457] "Digital twin" is a technology that recreates and simulates physical objects and environments in a virtual space.

[0458] The "emotion engine" is a system that analyzes emotions from the user's facial expressions, tone of voice, choice of words, etc.

[0459] The "virtual environment" is a simulated environment within the digital twin generated by generative AI.

[0460] "Autonomous driving AI" is artificial intelligence used to control the operation of autonomous vehicles.

[0461] A "smartphone" is a portable information terminal that, in addition to the functions of a mobile phone, can also connect to the Internet and run applications.

[0462] A "head-mounted display" is a device worn on the user's head that displays images in the user's field of vision.

[0463] "Simulation" is the process of recreating and testing the behavior of autonomous vehicles in a virtual environment.

[0464] "Real-time" refers to processing and display occurring immediately without delay.

[0465] "User" refers to a person who uses the system.

[0466] The system for implementing this invention uses generative AI, a digital twin, an emotion engine, a smartphone, and a head-mounted display (HMD). The specific configuration and operation of the system are described below.

[0467] System configuration

[0468] 1. Generative AI: Analyzes information such as natural language and video to generate virtual cars, people, and environments.

[0469] 2. Digital Twin: Reproducing and simulating real-world physical objects and environments in a virtual space.

[0470] 3. Emotion engine: Analyzes emotions from the user's facial expressions, tone of voice, choice of words, etc.

[0471] 4. Smartphone: A mobile information terminal that has the functionality of a mobile phone and can also connect to the Internet and run applications.

[0472] 5. Head-mounted display (HMD): A device worn on the user's head that displays images in the user's field of vision.

[0473] System Operation

[0474] 1. Data entry and analysis:

[0475] The server receives natural language and video input from the user.

[0476] Generative AI analyzes this data and generates virtual cars, people, and environments.

[0477] 2. Emotion analysis:

[0478] The server uses an emotion engine to analyze the user's facial expressions and tone of voice.

[0479] The emotion engine communicates the analysis results to the generative AI.

[0480] 3. Create a virtual environment:

[0481] The generative AI generates a virtual environment based on the analysis results from the emotion engine.

[0482] For example, if the user is happy, a bright virtual environment is generated, and if the user is sad, a rainy virtual environment is generated.

[0483] 4. Run the simulation:

[0484] The server inputs the generated virtual environment into the digital twin and performs a simulation.

[0485] The digital twin simulates the behavior of an autonomous vehicle in a generated virtual environment in real time.

[0486] 5. Viewing and manipulating results:

[0487] Using a smartphone or HMD, users can check the simulation results in real time.

[0488] Users can operate autonomous vehicles in a virtual environment and test them in various scenarios.

[0489] Specific examples

[0490] When a user simulates an autonomous vehicle using their smartphone, a camera captures their facial expressions and an emotion engine detects "happiness." The generative AI generates a sunny urban environment and runs the simulation in the digital twin. The user can view the simulation results in real time on their smartphone screen and control the vehicle's behavior.

[0491] Prompt Sentence Examples

[0492] "If the user is happy, generate an urban environment with sunny weather and run a simulation of an autonomous vehicle."

[0493] In this way, the embodiment of the invention can reproduce various scenarios and weather conditions and provide a real-time simulation environment that responds to the user's emotions.

[0494] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0495] Step 1:

[0496] The server receives natural language and video input from the user. The input data is information about the scenario and environment the user wants to use in the simulation. The server then passes this data to the generation AI.

[0497] Step 2:

[0498] The generative AI analyzes the natural language and video it receives to generate virtual cars, people, and environments. Specifically, it uses natural language processing technology to analyze text data and video analysis technology to analyze video data. The generated virtual cars, people, and environments are then input into the digital twin.

[0499] Step 3:

[0500] The server uses an emotion engine to analyze the user's facial expressions and tone of voice. The input data is the user's facial image and voice data. The emotion engine analyzes this data and identifies the user's emotional state. The analysis results are transmitted to the generation AI.

[0501] Step 4:

[0502] The generative AI generates a virtual environment based on the analysis results from the emotion engine. For example, if the user is happy, it generates a virtual environment with bright weather, and if they are sad, it generates a virtual environment with rain. The generated virtual environment is then input into the digital twin.

[0503] Step 5:

[0504] The server inputs the generated virtual environment into the digital twin and performs a simulation. The digital twin simulates the behavior of the autonomous vehicle in the generated virtual environment in real time. The input data is the initial state of the virtual environment and the autonomous vehicle. The output data is the simulation results.

[0505] Step 6:

[0506] The terminal (smartphone or HMD) receives the simulation results sent from the server and displays them in real time. The user can check the simulation results through the terminal and operate the autonomous vehicle in the virtual environment. The input data are the simulation results, and the output data are the user's operation instructions.

[0507] Step 7:

[0508] The user operates the autonomous vehicle in the virtual environment using a terminal. The user's operation instructions are sent to the server and reflected in the digital twin, thereby updating the simulation in real time. The input data are the user's operation instructions, and the output data are the updated simulation results.

[0509] Example 2

[0510] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0511] When testing autonomous driving AI, it takes a lot of time and money to recreate real-world road conditions. It is also difficult to adjust scenarios based on user emotions, which reduces the accuracy and efficiency of testing. Furthermore, conducting extensive and thorough testing requires the generation of diverse scenarios, which requires advanced technology.

[0512] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0513] In this invention, the server includes: means for generating virtual cars, humans, and environments from natural language, video, etc. using a generative AI model and inputting them into a digital twin; means for performing highly realistic and diverse simulations using the digital twin; means for extensive and thorough testing of an autonomous driving AI using the simulation; means for a user to input a prompt; means for a terminal to send the prompt to the server; means for the server to generate a scenario using the generative AI model; means for the server to send the generated scenario to the terminal; means for the terminal to display the scenario to the user; means for the user to adjust the scenario using an emotion engine; means for the server to regenerate the scenario based on information from the emotion engine; means for the server to send the regenerated scenario to the terminal; and means for the terminal to display the regenerated scenario to the user. This enables scenario adjustment according to the user's emotions, enabling extensive and thorough testing to be performed efficiently.

[0514] A "generative AI model" is an artificial intelligence model that generates virtual cars, people, and environments from input data such as natural language and video.

[0515] A "digital twin" is a digital model that recreates a real-world physical object or environment in a virtual space.

[0516] "Simulation" is the process of virtually recreating various real-world situations using digital twins for testing and analysis.

[0517] "Autonomous driving AI" is an artificial intelligence system that controls the driving of autonomous vehicles.

[0518] A "prompt" is an instruction that a user inputs to a generative AI model, and is text that prompts the generation of a scenario.

[0519] A "terminal" is a device that allows a user to input prompts and check scenarios.

[0520] A "server" is a computer system that runs generative AI models and generates and transmits scenarios.

[0521] The "emotion engine" is a system that analyzes the user's emotional information and adjusts the scenario based on that information.

[0522] "Regeneration" is the process of regenerating a scenario based on information from the emotion engine.

[0523] This invention is a system for efficiently and effectively testing autonomous driving AI. Specific embodiments of this system will be described below.

[0524] First, the user inputs a prompt using the terminal. A prompt is an instruction for the generative AI model to generate a scenario. For example, a prompt such as "Please generate a scenario for turning right at an intersection" can be input.

[0525] The device sends the input prompt to the server. This communication uses protocols such as HTTP requests. The server generates a scenario based on the received prompt using a generative AI model. For example, GPT-4 is used as the generative AI model.

[0526] The server sends the generated scenario to the terminal. The terminal displays the received scenario to the user. The user checks the displayed scenario and adjusts it as necessary using the emotion engine. The emotion engine is a system that analyzes the user's emotion information and regenerates the scenario based on that information.

[0527] For example, if a user types "I'm angry," the emotion engine relays that information to the digital twin, which then prioritizes simulating scenarios that are likely to anger the user, such as traffic congestion or rude drivers.

[0528] The server regenerates the scenario based on the information received from the emotion engine. The regenerated scenario is then sent back to the device, which displays it to the user. This allows the user to adjust the scenario according to the emotion, enabling efficient and extensive testing.

[0529] As a concrete example, consider the case where a user wants to generate a scenario for testing an autonomous vehicle. If the user inputs "Please generate a scenario for turning right at an intersection," the server will generate the following scenario:

[0530] Examples:

[0531] Scenario 1: A self-driving car is about to turn right at an intersection when an oncoming vehicle suddenly appears.

[0532] Scenario 2: When turning right at an intersection, a pedestrian suddenly jumps into the crosswalk.

[0533] Scenario 3: Another vehicle is approaching while turning right, posing a risk of collision.

[0534] Example prompt sentence:

[0535] "Generate a right turn scenario at an intersection."

[0536] "Create a scenario where a pedestrian suddenly jumps out."

[0537] "We want to test close-communication scenarios with other vehicles."

[0538] In this way, users can efficiently generate test scenarios for their autonomous driving AI and even adjust them according to emotions.

[0539] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0540] Step 1:

[0541] The user enters a prompt statement.

[0542] The user inputs a prompt sentence such as "Please generate a scenario for turning right at an intersection" into an input field on the terminal. The input prompt sentence is sent to the terminal as an instruction to generate a scenario.

[0543] Step 2:

[0544] The terminal sends the prompt to the server.

[0545] The terminal sends the prompt text entered by the user to the server as an HTTP POST request. This request includes the prompt text. The server receives this request and analyzes the prompt text.

[0546] Step 3:

[0547] The server generates scenarios using generative AI models.

[0548] The server generates a scenario using a generative AI model (e.g., GPT-4) based on the received prompt. The generative AI model creates an appropriate scenario based on the prompt. For example, when generating a "right turn scenario at an intersection," it generates a scenario in which an oncoming vehicle suddenly appears.

[0549] Step 4:

[0550] The server sends the generated scenario to the terminal.

[0551] The server sends the generated scenario to the terminal in JSON format. This communication also uses protocols such as HTTP requests. The terminal receives this request and analyzes the scenario data.

[0552] Step 5:

[0553] The terminal displays the scenario to the user.

[0554] The terminal displays the received scenario to the user. The user checks the scenario on the screen. For example, if the generated scenario is "An oncoming vehicle suddenly appears while turning right at an intersection," the details are displayed.

[0555] Step 6:

[0556] The user reviews the scenario and adjusts it using the emotion engine if necessary.

[0557] The user reviews the displayed scenario and adjusts it as needed using the emotion engine. For example, if the user types "I'm angry," the emotion engine relays that information to the digital twin.

[0558] Step 7:

[0559] The server regenerates the scenario based on the information from the emotion engine.

[0560] The server regenerates scenarios based on the information received from the emotion engine. For example, if the user is angry, it will prioritize scenarios that are likely to anger the user, such as traffic congestion or rude drivers.

[0561] Step 8:

[0562] The server transmits the regenerated scenario to the terminal.

[0563] The server sends the regenerated scenario to the terminal in JSON format. This communication also uses protocols such as HTTP requests. The terminal receives this request and analyzes the regenerated scenario data.

[0564] Step 9:

[0565] The terminal displays the regenerated scenario to the user.

[0566] The terminal displays the regenerated scenario received from the server to the user. The user checks the regenerated scenario. For example, if the regenerated scenario is "a scenario involving traffic congestion and rude behavior by drivers," details of the scenario are displayed.

[0567] (Application example 2)

[0568] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0569] As autonomous driving technology advances, testing and validation of autonomous driving AI is becoming increasingly important. However, testing in real-world road environments is time-consuming, costly, and poses safety issues. Furthermore, it is difficult to conduct tests that closely resemble real-world driving conditions because it is not possible to adjust scenarios based on user emotions.

[0570] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0571] In this invention, the server includes means for generating virtual cars, people, and environments from natural language, video, etc. using a generative AI and inputting them into a digital twin, means for implementing highly realistic and diverse simulations using the digital twin, means for extensively and thoroughly testing an autonomous driving AI using the simulations, means for adjusting the simulation scenario according to a user's emotions, and means for displaying the generated scenario on a smartphone. This enables testing of autonomous driving AI in realistic scenarios according to a user's emotions, enabling more realistic testing while reducing time and costs.

[0572] "Generative AI" is an artificial intelligence technology that generates virtual cars, people, and environments from natural language, videos, etc.

[0573] A "digital twin" is a digital model that recreates a real-world physical object in a virtual space.

[0574] "Simulation" is the process of using digital twins to recreate various situations in a virtual environment for testing and verification.

[0575] "Autonomous driving AI" is an artificial intelligence technology that enables cars to drive autonomously.

[0576] "User emotion" is the user's psychological state that is taken into account to adjust the simulation scenario.

[0577] A "scenario" is a specific situation or environment that is reproduced in a simulation.

[0578] A "smartphone" is a portable information terminal that can run a variety of applications in addition to the functions of a mobile phone.

[0579] Systems for implementing this invention include generative AI, digital twins, simulations, autonomous driving AI, user emotions, scenarios, and smartphones.

[0580] The server uses generative AI to generate virtual cars, people, and environments from natural language and video, and then inputs these into a digital twin. A digital twin is a digital model that recreates a real-world physical object in a virtual space, enabling highly realistic and diverse simulations.

[0581] Simulation is the process of using a digital twin to recreate various situations in a virtual environment for testing and validation, allowing for extensive and thorough testing of autonomous driving AI.

[0582] The system further includes means for adjusting the simulation scenario in response to the user's emotions, which are psychological states that are taken into consideration for adjusting the simulation scenario, so that if the user is angry, for example, a scenario in which the user encounters traffic congestion or rude drivers is generated.

[0583] The generated scenario is displayed on a smartphone, a mobile information terminal that can run a variety of applications in addition to the functions of a mobile phone, allowing the user to check the scenario and receive feedback on the test results in real time.

[0584] For example, if the user is "angry," the generated scenario will be "encountering heavy traffic and rude drivers." An example of a prompt sentence is "Generate a driving scenario where the autonomous vehicle encounters heavy traffic and rude drivers."

[0585] In this way, it becomes possible to test autonomous driving AI in realistic scenarios that respond to user emotions, enabling more realistic testing while reducing time and costs.

[0586] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0587] Step 1:

[0588] The server acquires the user's emotions. As input, it receives the user's emotion data (e.g., "angry," "happy," etc.) and passes the emotion data to the next step as output. Specifically, it either randomly selects the user's emotion or receives direct input from the user.

[0589] Step 2:

[0590] The server creates a scenario generation prompt based on the acquired emotional data. It receives the user's emotional data as input and obtains the generated prompt as output. Specifically, it performs conditional branching to generate a prompt according to the emotion. For example, if the emotion is "angry," it generates the prompt "Generate a driving scenario where the autonomous vehicle encounters heavy traffic and rude drivers."

[0591] Step 3:

[0592] The server uses the generated prompt sentence to request the generative AI model to generate a scenario. It receives the prompt sentence as input and obtains the generated scenario as output. Specifically, it sends the prompt sentence using the OpenAI API and receives the generated scenario.

[0593] Step 4:

[0594] The server sends the generated scenario to the smartphone. It receives the generated scenario as input and obtains scenario data to be displayed on the smartphone as output. Specifically, it sends the scenario data to the smartphone application so that the user can check it.

[0595] Step 5:

[0596] The user checks the scenario displayed on the smartphone and receives feedback on the test results in real time. The system receives the scenario displayed on the smartphone as input and sends feedback data to the server as output. Specifically, the user checks the scenario and enters feedback as necessary.

[0597] Step 6:

[0598] The server receives feedback data from users and analyzes the simulation results. It receives the feedback data as input and obtains the analysis results as output. Specifically, it analyzes the feedback data and identifies areas for improvement and adjustment of the autonomous driving AI.

[0599] Example 3

[0600] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0601] In the development of autonomous driving AI, testing in real-world environments is time-consuming and costly, and it is difficult to reproduce dangerous situations. Furthermore, because it is not possible to adjust test priorities based on user emotions, tests do not adequately consider user anxiety and fear. Therefore, efficient testing methods are needed to improve the safety and reliability of autonomous driving AI.

[0602] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means. In this invention, the server includes: means for generating virtual cars, people, and environments from natural language, videos, etc. using a generation AI and inputting them into a digital twin; means for implementing highly realistic and diverse simulations using the digital twin; means for extensively and thoroughly testing the autonomous driving AI using the simulations; means for adjusting test priorities using an emotion engine that collects and analyzes user emotion data; and means for setting simulation scenarios based on the analysis results from the emotion engine. This enables effective testing according to user emotions, improving the safety and reliability of the autonomous driving AI.

[0603] "Generative AI" is an artificial intelligence technology that generates virtual cars, people, and environments from data such as natural language and video.

[0604] A "digital twin" is a digital model that recreates a real-world physical environment or object in a virtual space.

[0605] "Simulation" is the process of using a digital twin to recreate various scenarios in a virtual environment for testing and evaluation.

[0606] "Autonomous driving AI" is an artificial intelligence technology for automating automobile driving.

[0607] The "emotion engine" is a system for collecting and analyzing user emotional data.

[0608] "Emotional data" is information that indicates the user's emotional state, and is collected through questionnaires, voice input, biometric sensors, etc.

[0609] A "simulation scenario" is a scenario that sets out specific situations or conditions that are reproduced within a simulation.

[0610] The "analysis result" is the result obtained by the emotion engine analyzing the user's emotion data.

[0611] This invention is a system that significantly reduces the time and cost involved in testing autonomous driving AI through simulations using generative AI and digital twins. It also has the function of adjusting test priorities based on user emotions using an emotion engine.

[0612] System configuration

[0613] The system consists of the following main components:

[0614] 1. Generative AI Models

[0615] 2. Digital Twin

[0616] 3. Emotion Engine

[0617] 4. Server

[0618] 5. Terminal

[0619] Hardware and software used

[0620] Generative AI model: Used to generate virtual cars, people, and environments from natural language, videos, etc. Specifically, a general generative AI model (e.g., GPT-4) is used.

[0621] Digital Twin: A digital model that recreates a real-world environment in a virtual space, faithfully recreating the physical environment and traffic conditions.

[0622] Emotion engine: A system for collecting and analyzing user emotional data. Emotion data is collected through questionnaires, voice input, biometric sensors, etc.

[0623] Server: Runs simulations using generative AI models and digital twins, and sets up simulation scenarios based on data from the emotion engine.

[0624] Terminal: A device through which a user inputs emotion data, such as a smartphone or tablet.

[0625] Program processing

[0626] 1. The user inputs emotion data

[0627] Users use a device to input emotion data, which can be input in multiple ways, such as through a questionnaire, voice input, or even data collection using biometric sensors.

[0628] 2. The device sends emotion data to the emotion engine.

[0629] The device transmits the emotion data entered by the user to the emotion engine in real time via an internet connection.

[0630] 3. The emotion engine analyzes the emotion data

[0631] The emotion engine analyzes the received emotion data and identifies the user's emotional state. For example, if the user inputs "I'm scared of driving on rainy days," the emotion engine identifies the emotion "fear."

[0632] 4. The emotion engine sends the analysis results to the server

[0633] The emotion engine sends the analysis results to the server, which include the user's emotional state and its details.

[0634] 5. The server generates the digital twin

[0635] The server uses the generative AI model to generate a digital twin that mimics the real-world environment, a virtual reproduction of the physical environment and traffic conditions.

[0636] 6. The server sets up the simulation scenario

[0637] The server sets a simulation scenario based on the analysis results received from the emotion engine. For example, if the user feels that "driving on a rainy day is scary," the server sets a simulation scenario for rainy weather.

[0638] 7. The server runs simulation tests of the autonomous driving AI.

[0639] The server runs simulation tests of the autonomous driving AI based on the set simulation scenario. The simulation is performed on a digital twin, faithfully reproducing the real-world environment.

[0640] 8. The server records the test results and repeats the scenario as needed.

[0641] The server records the results of the simulation test. If necessary, the same scenario can be repeatedly tested and the results compared and analyzed to evaluate the performance of the autonomous driving AI and identify areas for improvement.

[0642] Specific examples

[0643] If a user feels scared of driving on rainy days, the emotion engine analyzes this information and sends it to the server, which then sets up a rainy weather simulation scenario on the digital twin and tests the autonomous driving AI.

[0644] Prompt Sentence Examples

[0645] Example prompts to be input to the generative AI model:

[0646] "Please generate a simulation scenario for autonomous driving in urban areas in rainy weather, especially including situations with high risk of sudden braking and skidding."

[0647] In this way, effective testing according to the user's emotions becomes possible, and the safety and reliability of the autonomous driving AI are improved. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0648] Step 1:

[0649] The user inputs emotional data. The user inputs emotional data using a device. Emotional data can be input in multiple ways, such as through a questionnaire, voice input, or even data collection using biometric sensors. For example, the user opens a dedicated app on their smartphone and inputs, "I'm afraid of driving on rainy days." The input data is saved on the device in text format or as voice data.

[0650] Step 2:

[0651] The device sends the emotion data to the emotion engine. The device sends the emotion data entered by the user to the emotion engine in real time. This is done via an internet connection. Specifically, the device sends the emotion data to the emotion engine using the HTTPS protocol. The input data is sent to the emotion engine in text format or as voice data.

[0652] Step 3:

[0653] The emotion engine analyzes the emotion data. The emotion engine analyzes the received emotion data and identifies the user's emotional state. For example, from the input "I'm afraid of driving on rainy days," it identifies the emotion "fear." The emotion engine analyzes the text data using natural language processing technology and extracts the emotional state. The analysis results are generated in JSON format.

[0654] Step 4:

[0655] The emotion engine sends the analysis results to the server. The emotion engine sends the analysis results to the server. The analysis results include the user's emotional state and its details. Specifically, the emotion engine sends the analysis results to the server in JSON format. An internet connection is used for transmission.

[0656] Step 5:

[0657] The server generates a digital twin. Using a generative AI model, the server generates a digital twin that mimics a real-world environment. A digital twin is a virtual reproduction of a physical environment and traffic conditions. For example, the server inputs a prompt statement to the generative AI model, such as "Please generate a simulation scenario for autonomous driving in urban areas on rainy days," and generates a digital twin. The generated digital twin is saved on the server as a virtual environment.

[0658] Step 6:

[0659] The server sets the simulation scenario. The server sets the simulation scenario based on the analysis results received from the emotion engine. For example, if the user feels "afraid of driving on rainy days," the server sets a simulation scenario for rainy weather. Specifically, the server sets a scenario on the digital twin that includes situations with a high risk of sudden braking and skidding. The set scenario is saved on the server as input data for the simulation.

[0660] Step 7:

[0661] The server runs simulation tests of the autonomous driving AI. The server runs simulation tests of the autonomous driving AI based on the set simulation scenario. The simulation is performed on the digital twin and faithfully reproduces the real-world environment. For example, the server reproduces a scenario of an urban area on rainy weather on the digital twin to test the behavior of the autonomous driving AI. The results of the simulation are recorded on the server.

[0662] Step 8:

[0663] The server records the test results and repeats the scenario as necessary. The server records the results of the simulation test in a database. If necessary, the same scenario is repeatedly tested and the results are compared and analyzed. For example, the server collects data such as stopping distance during sudden braking and the frequency of skidding, and compares the results of multiple tests. This allows the performance of the autonomous driving AI to be evaluated and areas for improvement identified.

[0664] (Application example 3)

[0665] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0666] In the development of autonomous driving AI, testing in real-world environments is time-consuming, costly, and potentially dangerous. Furthermore, it is difficult to adjust test priorities based on user sentiment, making it difficult to conduct effective testing to ensure user safety and reliability.

[0667] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes: means for generating virtual cars, people, and environments from natural language, videos, etc. using a generative AI and inputting them into a digital twin; means for implementing highly realistic and diverse simulations using the digital twin; means for extensive and thorough testing of the autonomous driving AI using the simulations; and means for analyzing user emotions using an emotion engine and for the generative AI to provide optimal test scenarios. This enables effective testing based on user emotions, reducing the time and cost required for developing autonomous driving AI and improving safety and reliability.

[0668] "Generative AI" is an artificial intelligence technology that generates virtual cars, people, and environments from data such as natural language and video.

[0669] "Digital twin" is a technology for recreating real-world physical objects in a virtual space and performing simulations.

[0670] "Simulation" is a method of simulating real-world situations in a virtual environment for testing and analysis.

[0671] "Autonomous driving AI" is an artificial intelligence technology that enables cars to drive autonomously.

[0672] An "emotion engine" is a technology that analyzes the user's emotions and adjusts the system's behavior based on that information.

[0673] A "test scenario" is a plan or scenario for setting specific situations or conditions in a simulation and testing a system under those conditions.

[0674] As an embodiment of the present invention, an emotion-responsive automated driving test simulator will be described as an example.

[0675] The server uses generative AI to generate virtual vehicles, people, and environments from natural language, video, and other data, and then inputs them into the digital twin. The digital twin recreates real-world physical objects in a virtual space, allowing for highly realistic and diverse simulations. This allows for extensive and thorough testing of autonomous driving AI.

[0676] The server then uses an emotion engine to analyze the user's emotions. The emotion engine uses the smartphone's camera and microphone to analyze the user's facial expressions and voice to detect the user's emotions in real time. The detected emotion data is sent to the generative AI and used to generate optimal test scenarios.

[0677] For example, if the user is feeling fear, the emotion engine will convey that information to the generative AI, which will then generate scenarios that prioritize testing responses to accidents and dangerous situations. This enables effective testing based on user emotions, improving the safety and reliability of autonomous driving AI.

[0678] The hardware used includes a smartphone (camera, microphone), and the software used includes emotion recognition software (e.g., Microsoft® Azure® Face API) and a Python program.

[0679] As a concrete example, consider a scenario in which a user uses a smartphone to conduct a simulation test of an autonomous vehicle. When the user uses the smartphone to conduct a simulation test of an autonomous vehicle, an emotion engine analyzes the user's emotions, and a generative AI is designed to provide the optimal test scenario.

[0680] Example prompt sentence:

[0681] "Design an application where a user uses a smartphone to simulate a self-driving vehicle and an emotion engine analyzes the user's emotions, and a generative AI provides the optimal test scenario."

[0682] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0683] Step 1:

[0684] The server uses the smartphone's camera and microphone to collect the user's facial expressions and voice in real time. The input is the user's facial expression and voice data, and the output is to send this data to the emotion engine. Specifically, the smartphone's camera captures the user's face and the microphone records the user's voice.

[0685] Step 2:

[0686] The server uses an emotion engine to analyze the collected facial and voice data and detect the user's emotions. The input is the facial and voice data collected in step 1, and the output is the detected user emotion (e.g., joy, sadness, anger, fear, etc.). Specifically, emotion recognition software (e.g., Microsoft Azure Face API) analyzes the facial expression data, and voice analysis software analyzes the tone of voice.

[0687] Step 3:

[0688] The server sends the detected user emotion data to the generation AI. The input is the user emotion data detected in step 2, and the output is the emotion data sent to the generation AI. Specifically, the emotion engine sends the emotion data to the generation AI's API.

[0689] Step 4:

[0690] The server uses generative AI to generate an optimal test scenario based on the user's emotions. The input is the user's emotion data sent in step 3, and the output is the generated test scenario. Specifically, the generative AI model analyzes the emotion data and generates an appropriate scenario (e.g., emergency braking, pedestrian crossing, etc.).

[0691] Step 5:

[0692] The server inputs the generated test scenario into the digital twin and performs a simulation. The input is the test scenario generated in step 4, and the output is the simulation result. Specifically, the digital twin creates a virtual environment and simulates the behavior of the autonomous driving AI based on the scenario.

[0693] Step 6:

[0694] The server analyzes the simulation results and evaluates the performance of the autonomous driving AI. The input is the simulation results obtained in Step 5, and the output is the evaluation results. Specifically, the server analyzes the simulation results and evaluates performance indicators such as the AI's reaction time and accuracy.

[0695] Step 7:

[0696] The server feeds back the evaluation results to the user. The input is the evaluation results obtained in step 6, and the output is the feedback information provided to the user. Specifically, the server visually displays the evaluation results and informs the user of areas for improvement and success.

[0697] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0698] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0699] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0700] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0701] [Second embodiment]

[0702] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0703] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0704] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0705] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0706] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0707] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0708] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0709] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0710] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program.

[0711] The processor 28 executes the specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0712] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0713] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0714] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0715] "Example 1"

[0716] In one embodiment of the present invention, generative AI receives and analyzes information such as natural language and video as input. Based on the results of the analysis, it generates virtual vehicles, people, and environments. The generated virtual vehicles, people, and environments are then fed into a digital twin, which uses them to perform highly realistic and diverse simulations. For example, it can recreate various scenarios that an autonomous vehicle may encounter, such as urban and suburban environments, various weather conditions, and traffic situations.

[0717] "Example 2"

[0718] In another embodiment of the present invention, the Generative AI generates scenarios for testing the autonomous driving AI. The generated scenarios are designed to cover a variety of scenarios that an autonomous vehicle may encounter on the road. This allows the autonomous driving AI to be extensively and thoroughly tested and improved or adjusted based on the results. For example, the autonomous vehicle can handle a variety of scenarios, such as turning right at an intersection, a pedestrian suddenly stepping out, or approaching another vehicle.

[0719] "Example 3"

[0720] In a further embodiment of the present invention, simulation using generative AI and digital twins can significantly reduce the time and cost associated with testing autonomous driving AI. Specifically, simulation testing can be performed more quickly and efficiently than testing in real-world environments, and scenarios can be repeatedly tested as needed. This accelerates the process of developing and improving autonomous driving AI, thereby improving the safety and reliability of autonomous vehicles.

[0721] The processing flow of each embodiment will be described below.

[0722] "Example 1"

[0723] Step 1: The generative AI receives information such as natural language and video as input.

[0724] Step 2: The generation AI analyzes the input information and generates virtual cars, people, and environments based on the results of the analysis.

[0725] Step 3: The generated virtual vehicles, humans, and environments are fed into a digital twin.

[0726] Step 4: The digital twin performs highly realistic and diverse simulations using virtual vehicles, people, and environments.

[0727] "Example 2"

[0728] Step 1: The generation AI generates scenarios for testing the autonomous driving AI.

[0729] Step 2: The generated scenarios represent various scenarios that an autonomous vehicle may encounter on the road.

[0730] It is designed to cover a wide range of scenarios.

[0731] Step 3: The self-driving AI is extensively and thoroughly tested using generated scenarios.

[0732] Step 4: Based on the results of the testing, improvements and adjustments are made to the self-driving AI.

[0733] "Example 3"

[0734] Step 1: Using generative AI and digital twin simulations to significantly reduce the time and cost associated with testing autonomous driving AI.

[0735] Step 2: Simulation testing is fast and efficient, allowing scenarios to be tested repeatedly as needed.

[0736] Step 3: This will accelerate the process of developing and improving self-driving AI, thereby improving the safety and reliability of self-driving cars.

[0737] Example 1

[0738] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0739] In the development of autonomous driving technology, testing in actual road environments is time-consuming and costly, and it is difficult to ensure safety. Furthermore, to fully implement countermeasures for various scenarios, simulations under a variety of environments and conditions are required, but there is a lack of efficient means to carry this out.

[0740] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0741] In this invention, the server includes a means for a user to input a prompt sentence, a means for the terminal to send the input data to the server, a means for the server to analyze the data using a generative AI model, a means for the server to generate a virtual object based on the analysis results, a means for the server to input the generated virtual object into a digital twin, a means for the digital twin to perform a simulation, a means for the server to send the simulation results to the terminal, and a means for the terminal to display the simulation results to the user. This enables advanced and diverse simulations that reproduce actual road environments, reducing the time and cost in the development of autonomous driving technology, improving safety, and strengthening countermeasures for various scenarios.

[0742] A "user" is an entity that uses the system to input prompt statements and check the simulation results.

[0743] A "terminal" is a device through which a user enters prompts and communicates with a server.

[0744] A "server" is a computer system that has the ability to analyze data using generative AI models, generate virtual objects, and input them into a digital twin.

[0745] A "generative AI model" is an artificial intelligence model that analyzes information such as natural language and video and generates virtual objects.

[0746] A "prompt sentence" is an input sentence that describes in natural language the scenario that the user wants to simulate.

[0747] "Virtual objects" are digital data such as virtual cars, people, and environments generated by generative AI models.

[0748] A "digital twin" is a simulation system that uses virtual objects to recreate real-world environments and situations with a high degree of realism.

[0749] "Simulation" is the process of using a digital twin to recreate the behavior and interactions of virtual objects and test various scenarios.

[0750] "Simulation results" are data and information obtained as a result of a simulation performed by a digital twin.

[0751] This invention begins when a user inputs a prompt and the device sends the data to a server. The server analyzes the data using a generative AI model and generates a virtual object. The generated virtual object is then input into a digital twin, which then uses it to perform a simulation. The simulation results are then sent from the server to the device and ultimately displayed to the user.

[0752] Specifically, the user uses a terminal to input a prompt statement, which is a natural language description of the scenario they want to simulate. For example, the prompt statement might be, "I want to simulate an autonomous vehicle in an urban environment on a rainy day," or "I want to simulate an autonomous vehicle in a mountainous area on a snowy day."

[0753] The device sends the prompt text entered by the user to the server. At this time, the device appropriately converts the data format and sends it according to the communication protocol. The server inputs the received prompt text into a generative AI model (for example, a natural language processing model or an image generation model) for analysis. The generative AI model understands the content of the prompt text and extracts the necessary information.

[0754] The server generates virtual vehicles, people, and environments based on the analysis results. In this process, the server uses image generation models to create realistic virtual objects. The generated virtual objects are then input into the digital twin system, which then prepares the system for simulation using these virtual objects.

[0755] A digital twin uses virtual objects to perform simulations, such as recreating the behavior of an autonomous vehicle in an urban environment on a rainy day. The results show how the vehicle behaves in the rain and how it interacts with other vehicles and pedestrians.

[0756] The server receives the results of the simulation performed by the digital twin and sends them to the terminal. The simulation results include the movement of the car and changes in the environment. The terminal displays the simulation results received from the server to the user. The user can check the simulation results and re-enter prompt statements if necessary.

[0757] This system enables advanced and diverse simulations that replicate actual road environments, reducing the time and cost required to develop autonomous driving technology, improving safety, and strengthening countermeasures for a variety of scenarios.

[0758] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0759] Step 1:

[0760] The user enters a prompt statement.

[0761] Specifically, the user describes the scenario they want to simulate in natural language in the input field of the device. For example, they might enter, "I want to simulate an autonomous vehicle in an urban environment on a rainy day."

[0762] Input: The prompt text entered by the user

[0763] Output: The prompt text entered in the terminal

[0764] Step 2:

[0765] The terminal sends the input data to the server.

[0766] Specifically, the terminal converts the prompt text entered by the user into an appropriate data format and sends it to the server in accordance with the communication protocol.

[0767] Input: The prompt text entered into the terminal

[0768] Output: The prompt sent to the server

[0769] Step 3:

[0770] The server analyzes the data using a generative AI model.

[0771] Specifically, the server inputs the received prompt into a generative AI model (e.g., a natural language processing model) and analyzes the content of the prompt. The generative AI model then understands the content of the prompt and extracts the necessary information.

[0772] Input: The prompt sent to the server

[0773] Output: Analysis results (required information)

[0774] Step 4:

[0775] The server generates a virtual object based on the analysis results.

[0776] Specifically, the server generates virtual vehicles, people, and environments based on the analysis results, using image generation models to create realistic virtual objects.

[0777] Input: Analysis results (required information)

[0778] Output: Generated virtual object

[0779] Step 5:

[0780] The virtual objects generated by the server are then inserted into the digital twin.

[0781] Specifically, the server inputs the generated virtual object into the digital twin system and prepares for the simulation.

[0782] Input: Generated virtual object

[0783] Output: Virtual objects fed into the digital twin

[0784] Step 6:

[0785] The digital twin performs the simulation.

[0786] Specifically, the digital twin uses virtual objects to perform simulations, such as recreating the behavior of an autonomous vehicle in an urban environment on a rainy day.

[0787] Input: Virtual objects fed into the digital twin

[0788] Output: Simulation results

[0789] Step 7:

[0790] The server transmits the simulation results to the terminal.

[0791] Specifically, the server receives the results of the simulation performed by the digital twin and sends them to the terminal.

[0792] Input: Simulation results

[0793] Output: Simulation results sent to the terminal

[0794] Step 8:

[0795] The terminal displays the simulation results to the user.

[0796] Specifically, the terminal displays the simulation results received from the server to the user, who can then check them and re-enter the prompt sentence if necessary.

[0797] Input: Simulation results sent to the terminal

[0798] Output: Simulation results displayed to the user

[0799] (Application example 1)

[0800] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0801] In the development of autonomous vehicles, testing in real road environments is time-consuming and costly, and it is difficult to cover all scenarios. In addition, there is a lack of means for users to simulate specific scenarios and check the behavior of autonomous vehicles.

[0802] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0803] In this invention, the server includes means for generating virtual cars, people, and environments from natural language and video using a generative AI and inputting them into a digital twin, means for implementing highly realistic and diverse simulations using the digital twin, means for extensively and thoroughly testing autonomous driving AI using the simulations, means for a user to input natural language and video and implement a simulation using the generated virtual elements, and means for providing the results of the simulation to the user, thereby enabling the user to simulate various scenarios and check the behavior of an autonomous driving car.

[0804] "Generative AI" is an artificial intelligence technology that analyzes information such as natural language and video to generate virtual cars, people, and environments.

[0805] A "digital twin" is a digital model that recreates a real-world physical object or environment in a virtual space.

[0806] "Simulation" is the process of recreating various real-world scenarios using virtual vehicles, people, and environments to verify their behavior.

[0807] "Autonomous driving AI" is an artificial intelligence technology used to control and make decisions about autonomous vehicles.

[0808] A "user" is a person or organization that uses the system to input natural language or video and perform simulations.

[0809] "Virtual elements" refer to virtual cars, people, and environments generated by generative AI.

[0810] "Simulation results" refers to the data and information obtained when a simulation is performed.

[0811] A system for implementing this invention includes elements such as generative AI, digital twin, simulation, autonomous driving AI, user, virtual element, and simulation results.

[0812] The server uses generative AI to analyze information such as natural language and video to generate virtual vehicles, people, and environments. These virtual elements are then fed into a digital twin, which then uses them to perform highly realistic and diverse simulations. Simulations enable extensive and thorough testing of autonomous driving AI.

[0813] Users input natural language or video using a device such as a smartphone. The input information is sent to a server and analyzed by generative AI. As a result of the analysis, a virtual car, human, and environment are generated. The generated virtual elements are input into a digital twin, where a simulation is performed. The simulation results are provided to the user, who can simulate various scenarios and check the behavior of the autonomous vehicle.

[0814] For example, if a user requests, "I want to simulate an autonomous vehicle in an urban environment on a sunny day," the generative AI will recreate the urban environment on a sunny day and run a simulation in the digital twin. The simulation results will be provided to the user, who can then check the behavior of the autonomous vehicle.

[0815] The hardware used includes smartphones and servers. The software used includes Python and OpenAI API. Data processing and calculation include information analysis using generative AI, simulation using digital twins, and provision of simulation results.

[0816] Examples of prompts include:

[0817] "I want to simulate an autonomous vehicle in an urban environment on a sunny day."

[0818] "I want to simulate an autonomous vehicle in a suburban environment on a rainy day."

[0819] In this way, users can simulate various scenarios and see how the autonomous vehicle will behave.

[0820] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0821] Step 1:

[0822] The user inputs natural language or video using a device such as a smartphone. The input information is sent from the device to the server. The input data is a prompt sentence, for example, "I would like to simulate an autonomous vehicle in an urban environment on a sunny day."

[0823] Step 2:

[0824] The server passes the received natural language and video information to the generation AI. The generation AI analyzes the input data and generates virtual cars, people, and environments. Specifically, it uses the OpenAI API to analyze the input prompt sentences and generate virtual elements. The output is data on the generated virtual cars, people, and environments.

[0825] Step 3:

[0826] The server inputs the generated virtual elements into a digital twin. A digital twin is a digital model that recreates physical objects and environments in a virtual space, and performs a simulation using the generated virtual elements. The input is the data of the generated virtual elements, and the output is the simulation results.

[0827] Step 4:

[0828] The server analyzes the results of the simulations performed by the digital twin and generates data to provide to users. Specifically, it organizes the simulation results and converts them into a format that is easy for users to understand. The input is the simulation result data, and the output is the organized data to provide to users.

[0829] Step 5:

[0830] The server sends the organized simulation results to the user's device, where the user can check the simulation results and evaluate the behavior of the autonomous vehicle. The input is the organized simulation result data, and the output is the simulation results displayed on the user's device.

[0831] In this way, users can simulate various scenarios and see how the autonomous vehicle will behave.

[0832] Example 2

[0833] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0834] When testing autonomous driving AI, it takes a lot of time and money to recreate real-world road conditions. Furthermore, it is difficult to completely recreate real-world road conditions, which often limits the scope of testing. As a result, there is a problem in that it is not possible to adequately prepare countermeasures for the diverse scenarios that autonomous driving AI may encounter in a real driving environment.

[0835] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0836] In this invention, the server includes means for generating virtual vehicles, people, and environments from natural language, video, etc. using a generative AI model and inputting them into a digital twin, means for implementing highly realistic and diverse simulations using the digital twin, means for extensively and thoroughly testing the autonomous driving AI using the simulations, means for generating scenarios by inputting prompt sentences, means for saving the generated scenarios, means for testing the autonomous driving AI using the saved scenarios, and means for recording the test results. This allows the autonomous driving AI to be tested so that it can handle a variety of scenarios, and allows for improvements and adjustments based on the test results.

[0837] A "generative AI model" is an artificial intelligence technology that generates virtual vehicles, people, and environments from input data such as natural language and video.

[0838] A "digital twin" is a digital model that recreates a real-world physical system or environment in a virtual space.

[0839] "Simulation" is the process of using a digital twin to virtually recreate and test real-world road conditions and driving scenarios.

[0840] "Autonomous driving AI" is an artificial intelligence technology used to control the driving of self-driving cars.

[0841] A "prompt" is an instruction entered into a generative AI model to generate a specific scenario.

[0842] A "scenario" is a hypothetical situational setting that describes a specific driving situation or environment that an autonomous vehicle might encounter.

[0843] "Saving" is the act of recording the generated scenario in a database or file system.

[0844] "Test results" are data on the behavior and performance of autonomous driving AI obtained when running a simulation.

[0845] This invention is a system that generates scenarios for testing autonomous driving AI using a generative AI model and performs simulations using a digital twin. Specific embodiments of this system are described below.

[0846] System configuration

[0847] Hardware

[0848] Server: Use a server with high-performance computing capabilities. Specifically, a server equipped with a GPU is preferable.

[0849] Device: The user uses a device such as a computer or tablet to enter the prompt.

[0850] software

[0851] Generative AI model: An artificial intelligence technology for generating virtual vehicles, people, and environments from input data such as natural language and video. Specifically, it uses machine learning libraries such as TensorFlow and PyTorch.

[0852] Digital twin: Software for recreating real-world physical systems and environments in virtual space. Game engines such as Unity and Unreal Engine can be used.

[0853] Database: A database for saving the generated scenarios. Specifically, a database management system such as MySQL or MongoDB is used.

[0854] System Operation

[0855] Entering a prompt statement

[0856] The user inputs a prompt sentence to the generative AI model using the terminal. For example, the user inputs the prompt sentence "Please generate a right turn scenario at an intersection."

[0857] Scenario Generation Execution

[0858] The server passes the received prompt sentence to the generative AI model, which then generates a detailed scenario based on the prompt sentence. For example, the generated scenario may include the shape of the intersection, the status of traffic lights, and the movements of other vehicles and pedestrians.

[0859] Saving a Scenario

[0860] The server saves the generated scenarios in JSON format, which are later used to test the autonomous driving AI.

[0861] Test run of the scenario

[0862] The server runs tests on the self-driving AI using the saved scenarios, and the test results are logged and later analyzed.

[0863] Specific examples

[0864] As a concrete example, the following scenario is generated:

[0865] A scenario in which an autonomous vehicle is turning right at an intersection and an oncoming vehicle is coming straight ahead.

[0866] Scenario where a pedestrian suddenly jumps out onto the crosswalk

[0867] Scenarios where other vehicles suddenly change lanes

[0868] Prompt Sentence Examples

[0869] "Generate a right turn scenario at an intersection"

[0870] "Generate a scenario where a pedestrian suddenly jumps out."

[0871] "Generate a scenario where another vehicle suddenly changes lanes."

[0872] This system allows the self-driving AI to be tested to handle a variety of scenarios, allowing it to be improved and adjusted based on the results.

[0873] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0874] Step 1:

[0875] The user inputs a prompt sentence to the generative AI model using a terminal. For example, the user inputs a prompt sentence such as "Please generate a right turn scenario at an intersection."

[0876] Input: prompt statement

[0877] Output: The prompt is sent to the server.

[0878] Specific operation: The user uses the terminal keyboard to enter a prompt sentence and clicks the send button.

[0879] Step 2:

[0880] The server passes the received prompt sentence to the generative AI model, which then generates a detailed scenario based on the prompt sentence.

[0881] Input: prompt statement

[0882] Output: Generated scenario

[0883] Specific operation: The server inputs the prompt sentence into the generative AI model, which then generates a scenario including details such as the shape of the intersection, the status of traffic lights, and the movement of other vehicles and pedestrians.

[0884] Step 3:

[0885] The server saves the generated scenarios in JSON format, which are later used to test the autonomous driving AI.

[0886] Input: Generated scenario

[0887] Output: JSON format scenario file

[0888] Specific operation: The server converts the generated scenario into JSON format and saves it in a database or file system.

[0889] Step 4:

[0890] The server runs tests on the self-driving AI using the saved scenarios, and the test results are logged and later analyzed.

[0891] Input: JSON format scenario file

[0892] Output: Test result log file

[0893] Specific operation: The server loads the saved scenario, executes the scenario for the autonomous driving AI, and records the test results in a log file.

[0894] (Application example 2)

[0895] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0896] As autonomous driving technology advances, there is an increasing need to test the performance of autonomous driving AI extensively and thoroughly. However, testing in real-world road environments is time-consuming and costly, and ensuring safety is also a challenge. Furthermore, there is a lack of a means to efficiently visualize generated scenarios and propose improvements to autonomous driving AI based on the results. To solve these challenges, a more efficient and comprehensive test system is needed.

[0897] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0898] In this invention, the server includes means for generating virtual cars, people, and environments from natural language, videos, etc. using a generative AI and inputting them into a digital twin, means for implementing highly realistic and diverse simulations using the digital twin, means for extensively and thoroughly testing the autonomous driving AI using the simulations, means for simulating the generated scenarios on a smartphone and visualizing the results, and means for proposing improvements to the autonomous driving AI based on the results of the scenarios, thereby enabling efficient and comprehensive testing of the performance of the autonomous driving AI and rapid identification of areas for improvement.

[0899] "Generative AI" is an artificial intelligence technology that generates virtual cars, people, and environments from natural language, videos, etc.

[0900] A "digital twin" is a digital model that recreates a real-world physical object or environment in a virtual space.

[0901] "Simulation" is a method of using digital twins to virtually recreate highly realistic and diverse situations and conduct experiments and tests.

[0902] "Autonomous driving AI" is an artificial intelligence technology that enables cars to drive autonomously.

[0903] A "smartphone" is a mobile information terminal that has the functions of a computer in addition to the functions of a mobile phone.

[0904] "Visualization" is a technique for visually displaying data and simulation results to make them easier to understand.

[0905] "Areas for improvement" are corrections and refinements that are identified based on the simulation results to improve the performance of the autonomous driving AI.

[0906] The system for implementing this invention includes elements of generative AI, digital twin, simulation, autonomous driving AI, smartphone, visualization, and improvement. Specific embodiments are described below.

[0907] The server uses generative AI to generate virtual cars, people, and environments from natural language and video. The generated virtual cars, people, and environments are then input into a digital twin, a digital model that recreates physical objects and environments in the real world in a virtual space.

[0908] The server then uses the digital twin to perform a variety of highly realistic simulations, a method of experimenting and testing in virtually recreated situations, used to extensively and thoroughly test the performance of autonomous driving AI.

[0909] The generated scenario is simulated on a smartphone, and the results are visualized. A smartphone is a mobile information terminal that has the functionality of a computer in addition to a mobile phone. Visualization is a technique for visually displaying data and simulation results to make them easier to understand.

[0910] Users can identify and propose improvements to the autonomous driving AI based on the simulation results. Improvements are corrections and improvements that can be made to improve the performance of the autonomous driving AI, as identified based on the simulation results.

[0911] For example, you can generate a scenario by inputting the following prompt into the generation AI:

[0912] "Generate a scenario in which an autonomous vehicle makes a right turn at an intersection."

[0913] "Generate a lane change scenario for an autonomous vehicle on a highway."

[0914] "Generate a scenario in which an autonomous vehicle drives through a residential area at night."

[0915] "Generate a scenario in which an autonomous vehicle drives through urban areas in the rain."

[0916] This allows users to test autonomous driving AI against a variety of scenarios and evaluate its performance efficiently and comprehensively.

[0917] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0918] Step 1:

[0919] The server uses generative AI to generate virtual cars, people, and environments from natural language, videos, etc.

[0920] Input: natural language and video data

[0921] Data processing: Generative AI models analyze input data and generate virtual cars, people, and environments.

[0922] Output: Digital data of virtual vehicles, humans, and environments

[0923] Step 2:

[0924] The server then inputs the generated virtual vehicles, people, and environments into the digital twin.

[0925] Input: Digital data of virtual vehicles, humans, and environments

[0926] Data processing: Integrating virtual vehicles, humans and environments into the digital twin.

[0927] Output: Virtual environment integrated into a digital twin

[0928] Step 3:

[0929] The server uses the digital twin to perform highly realistic and diverse simulations.

[0930] Input: Virtual environment integrated into the digital twin

[0931] Data calculation: The simulation engine calculates the behavior in the virtual environment and executes the scenario.

[0932] Output: Simulation result data

[0933] Step 4:

[0934] The server simulates the generated scenario on a smartphone and visualizes the results.

[0935] Input: Simulation result data

[0936] Data processing: Generate graphs and charts to visually display the simulation results.

[0937] Output: Visualized data displayed on a smartphone

[0938] Step 5:

[0939] Users will identify and propose improvements to the autonomous driving AI based on the simulation results.

[0940] Input: Visualized data displayed on a smartphone

[0941] Data Computation: Users analyze simulation results and identify areas for improvement.

[0942] Output: Proposal data for improvements to the autonomous driving AI

[0943] This allows users to test autonomous driving AI against a variety of scenarios and evaluate its performance efficiently and comprehensively.

[0944] Example 3

[0945] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0946] In the development of autonomous driving AI, testing in real-world environments is time-consuming and costly, making it difficult to conduct efficient testing. Furthermore, it is difficult to repeatedly test reproducible scenarios in real-world environments, making it difficult to adequately prepare for various scenarios.

[0947] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means. In this invention, the server includes: means for generating virtual objects, people, and environments from natural language, videos, etc. using a generative AI and inputting them into a digital twin; means for performing highly realistic and diverse simulations using the digital twin; means for extensively and thoroughly testing autonomous driving AI using the simulations; means for a user to set a test scenario using a terminal; means for the server to generate a simulation environment using a generative AI model; means for the server to execute a simulation on the digital twin platform; means for the server to analyze the test results; and means for the user to modify the scenario and retest. This significantly reduces the time and cost involved in testing and enables repeated testing of reproducible scenarios.

[0948] "Generative AI" is an artificial intelligence technology that generates virtual objects, people, and environments from input data such as natural language, images, and videos.

[0949] "Digital twin" is a technology that recreates real-world objects and environments in a virtual space and performs simulations and analyses.

[0950] A "simulation environment" is a virtual space for testing and analysis that includes virtual objects, people, and environments generated by generative AI.

[0951] "Autonomous driving AI" is an artificial intelligence technology for automating automobile driving.

[0952] A "terminal" is a device such as a computer or smartphone that is operated by a user.

[0953] A "server" is a high-performance computer system that runs generative AI models, manages simulations, and analyzes data.

[0954] A "test scenario" is a scenario that includes specific conditions or situations set up to evaluate the performance of autonomous driving AI.

[0955] A "generative AI model" is a specific implementation of generative AI, an algorithm or program for generating virtual objects, people, and environments from input data such as natural language, images, and videos.

[0956] A "prompt" is an instruction entered into a generative AI model to cause it to generate a specific output.

[0957] A "digital twin platform" is software and systems that realize digital twin technology.

[0958] MODE FOR CARRYING OUT THE INVENTION

[0959] This invention is a system that significantly reduces the time and cost involved in testing autonomous driving AI through simulations using generative AI and digital twins. This system involves a series of processes: a user sets up a test scenario using a terminal, a server generates a simulation environment using a generative AI model, runs the simulation on a digital twin platform, and analyzes the test results.

[0960] Hardware and software used

[0961] Hardware: High-performance servers (e.g., high-performance computer systems)

[0962] Software: Generative AI models (e.g., advanced generative AI algorithms), digital twin platforms (e.g., simulation software)

[0963] Data processing and calculation flow

[0964] 1. The user sets up a specific test scenario using the device. For example, the user selects "changing lanes on a highway in the rain" as the scenario. Detailed parameters such as traffic volume and weather conditions are entered through the device interface.

[0965] 2. Based on the scenario settings received from the user, the server uses a generative AI model to generate a detailed simulation environment. The generative AI model then builds a realistic simulation environment based on the input conditions.

[0966] 3. The server imports the generated simulation environment into the digital twin platform and runs autonomous driving AI tests. The simulation is performed in real time, and vehicle behavior and changes in the environment are recorded in detail.

[0967] 4. The server analyzes the simulation results and evaluates the performance and problems of the autonomous driving AI. The analysis results are provided to the user in the form of graphs and reports.

[0968] 5. The user modifies the scenario as needed based on the analysis results. After modification, the user runs the simulation again and checks the results. By repeating this process, the performance of the autonomous driving AI is optimized.

[0969] Specific examples

[0970] For example, if a user sets "changing lanes on a highway in the rain" as a test scenario, they would input the following prompt sentence into the generative AI model:

[0971] Example prompt sentence:

[0972] "Generate a lane change scenario on a highway in the rain. Specifically, simulate a lane change in heavy traffic and record the vehicle's behavior in detail."

[0973] When this prompt is entered, the generative AI model creates a detailed simulation environment and runs autonomous driving AI tests on the digital twin platform. The test results are analyzed by the server and fed back to the user.

[0974] The above is an embodiment of the present invention. This system significantly reduces the time and cost involved in testing, and enables repeatable testing of scenarios. The flow of the identification process in the third embodiment will be described with reference to FIG. 15.

[0975] Step 1:

[0976] The user sets up a test scenario using the device. Through the device interface, the user inputs detailed parameters such as specific traffic and weather conditions. For example, the user selects "changing lanes on a highway in rainy weather" as a scenario. The input data is then sent to the server.

[0977] Input: Detailed parameters of the test scenario (e.g., rain, highway, lane change)

[0978] Output: Scenario configuration data sent to the server

[0979] Specific behavior:

[0980] The user operates the GUI on the terminal to open the scenario setting screen.

[0981] The user inputs conditions such as "rainy weather," "highway," and "lane change."

[0982] Step 2:

[0983] Based on the scenario settings received from the user, the server uses a generative AI model to generate a detailed simulation environment. The generative AI model builds a realistic simulation environment based on the input conditions. A prompt statement is input into the generative AI model to generate the simulation environment.

[0984] Input: Scenario setting data, prompt text

[0985] Output: Generated simulation environment data

[0986] Specific behavior:

[0987] The server receives input data from the user.

[0988] The server inputs prompt statements into the generated AI model and generates a simulation environment.

[0989] Step 3:

[0990] The server then imports the generated simulation environment into the digital twin platform and runs autonomous driving AI tests. The simulation is performed in real time, and vehicle behavior and changes in the environment are recorded in detail.

[0991] Input: Generated simulation environment data

[0992] Output: Simulation execution result data

[0993] Specific behavior:

[0994] The server imports the generated simulation environment into the digital twin platform.

[0995] The server runs the autonomous driving AI in a simulated environment and collects data.

[0996] Step 4:

[0997] The server analyzes the simulation results and evaluates the performance and problems of the autonomous driving AI. The analysis results are provided to the user in the form of graphs and reports.

[0998] Input: Simulation execution result data

[0999] Output: Analysis result data (graphs, reports)

[1000] Specific behavior:

[1001] The server runs algorithms that analyze the simulation data.

[1002] The server visualizes the analysis results and generates a report.

[1003] Step 5:

[1004] The user modifies the scenario as necessary based on the analysis results. After modification, the user runs the simulation again and checks the results. By repeating this process, the performance of the autonomous driving AI is optimized.

[1005] Input: Analysis result data, modified scenario setting data

[1006] Output: New simulation run results data

[1007] Specific behavior:

[1008] The user uses the terminal to check the analysis results.

[1009] The user corrects the scenario and runs the test again.

[1010] (Application example 3)

[1011] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1012] In the development of autonomous driving AI, testing in real-world environments is time-consuming, costly, and difficult to ensure safety. Furthermore, extensive and thorough testing is required to adequately address a wide range of scenarios, which is also difficult to achieve in real-world environments. To address these challenges, a means is needed to efficiently and safely test autonomous driving AI.

[1013] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1014] In this invention, the server includes means for generating virtual cars, people, and environments from natural language, video, etc. using a generative AI and inputting them into a digital twin, means for implementing highly realistic and diverse simulations using the digital twin, means for extensively and thoroughly testing autonomous driving AI using the simulations, means for running simulation tests on a smartphone, and means for evaluating the simulation results and calculating safety and efficiency scores. This makes it possible to efficiently and safely test autonomous driving AI and quickly take measures for a variety of scenarios.

[1015] "Generative AI" is an artificial intelligence technology that generates virtual cars, people, and environments from data such as natural language and video.

[1016] "Digital twin" is a technology that recreates real-world physical environments and objects in virtual space.

[1017] "Simulation" is a method of recreating real-world environments and scenarios in a virtual space for testing and evaluation.

[1018] "Autonomous driving AI" is an artificial intelligence technology that enables autonomous driving of vehicles.

[1019] A "smartphone" is a mobile device that has the functions of a computer in addition to the functions of a mobile phone.

[1020] A "simulation test" is a test conducted using a simulation in a virtual space.

[1021] "Safety" is the ability of a system or technology to operate without accidents or failures.

[1022] "Efficiency" is the ability of a system or technology to achieve maximum results with minimum resources.

[1023] The "score" is a numerical representation of the evaluation result.

[1024] The system for implementing this invention utilizes generative AI and digital twin technology to perform simulation tests of autonomous driving AI. Specific embodiments are described below.

[1025] First, the server uses generative AI to generate virtual cars, people, and environments from data such as natural language and video. This generated data is then input into a digital twin, which recreates real-world physical environments and objects in a virtual space.

[1026] The digital twin is then subjected to a variety of highly realistic simulations, allowing the autonomous driving AI to be tested extensively and thoroughly. The simulations run on a smartphone, allowing users to set up various scenarios and repeatedly test them.

[1027] The simulation results are evaluated by the server and safety and efficiency scores are calculated, allowing users to evaluate the performance of their autonomous driving AI and make any necessary improvements.

[1028] Hardware and software used:

[1029] Hardware: Smartphone (iOS or Android)

[1030] Software: Python, TensorFlow, Digital Twin Library

[1031] Data processing and calculation:

[1032] The server uses generative AI to generate virtual vehicles, people, and environments from natural language and video. The generated data is input into a digital twin, where a simulation is conducted in the virtual space. The simulation results are evaluated by the server, and a safety and efficiency score is calculated.

[1033] Examples:

[1034] For example, to test an urban driving scenario, the following prompt sentences are input to the generative AI model:

[1035] Example prompt sentence:

[1036] "Simulate urban driving scenarios. The scenarios include traffic lights, pedestrians, and other vehicles. The AI ​​model needs to drive safely and efficiently."

[1037] A user opens the AutoDrive Sim app on their smartphone and selects an urban driving scenario. The app loads the scenario and starts the simulation. When the simulation is finished, a safety and efficiency score is displayed, allowing the user to evaluate the AI ​​model's performance.

[1038] In this way, it becomes possible to efficiently and safely test autonomous driving AI and quickly develop countermeasures for a variety of scenarios.

[1039] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1040] Step 1:

[1041] The server uses generative AI to generate virtual cars, people, and environments from natural language and video.

[1042] Input: Natural language and video data

[1043] Data processing: Generative AI models are used to generate virtual vehicles, people, and environments from input data.

[1044] Output: Virtual vehicle, human, and environment data

[1045] Step 2:

[1046] The server inputs data about the generated virtual cars, people, and environment into the digital twin.

[1047] Input: Virtual vehicle, human, and environment data

[1048] Data processing: Input data into the digital twin and recreate it in virtual space.

[1049] Output: Virtual environment in the digital twin

[1050] Step 3:

[1051] The server uses the digital twin to perform highly realistic and diverse simulations.

[1052] Input: Virtual environment in the digital twin

[1053] Data processing: Using a simulation engine to run scenarios in a virtual environment.

[1054] Output: Simulation result data

[1055] Step 4:

[1056] The terminal (smartphone) executes a simulation test.

[1057] Input: Simulation scenario data

[1058] Data processing: Run simulations on your smartphone and collect the results.

[1059] Output: Simulation result data

[1060] Step 5:

[1061] The server evaluates the simulation results and calculates a safety and efficiency score.

[1062] Input: Simulation result data

[1063] Data processing: Analyze the results data and calculate safety and efficiency scores.

[1064] Output: Safety score, efficiency score

[1065] Step 6:

[1066] Users can check the simulation results on their smartphones and evaluate the performance of the AI ​​model.

[1067] Input: Safety score, Efficiency score

[1068] Data processing: Display scores through a smartphone interface.

[1069] Output: User's evaluation result

[1070] The above is the flow of processing of the program of the system that realizes the application example.

[1071] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1072] "Example 1"

[1073] In one embodiment of the present invention, the generation AI uses an emotion engine that recognizes the user's emotions to generate virtual cars, people, and environments that correspond to the user's emotions. Specifically, the emotion engine analyzes the user's emotions from their facial expressions, tone of voice, and choice of words, and transmits the results to the generation AI. Based on the analysis results, the generation AI generates, for example, a virtual environment with bright weather if the user is happy, or a virtual environment with rain if the user is sad.

[1074] "Example 2"

[1075] The emotion engine also adjusts the simulation scenario according to the user's emotions. For example, if the user is angry, the emotion engine communicates this information to the digital twin, and the digital twin prioritizes simulating scenarios that are likely to anger the user, such as traffic congestion or rude driver behavior.

[1076] "Example 3"

[1077] Furthermore, the emotion engine adjusts the testing priorities of the autonomous driving AI based on the user's emotions. For example, if the user is feeling scared, the emotion engine will convey that information to the autonomous driving AI, and the autonomous driving AI will prioritize testing responses to accidents and dangerous situations. This enables more effective testing based on the user's emotions.

[1078] The processing flow of each embodiment will be described below.

[1079] "Example 1"

[1080] Step 1: The emotion engine analyzes the user's emotions based on their facial expressions, tone of voice, and choice of words.

[1081] Step 2: The emotion engine communicates the analysis results to the generative AI.

[1082] Step 3: Based on the analysis results, the AI ​​generates virtual cars, people, and environments according to the user's emotions. For example, if the user is happy, it generates a virtual environment with bright weather, and if the user is sad, it generates a virtual environment with rain.

[1083] "Example 2"

[1084] Step 1: The emotion engine analyzes the user's emotions.

[1085] Step 2: The emotion engine communicates the analysis results to the digital twin.

[1086] Step 3: The digital twin adjusts the simulation scenarios according to the user's emotions. For example, if the user is angry, it will prioritize scenarios that are likely to anger the user, such as traffic congestion or rude driver behavior.

[1087] "Example 3"

[1088] Step 1: The emotion engine analyzes the user's emotions.

[1089] Step 2: The emotion engine communicates the analysis results to the autonomous driving AI.

[1090] Step 3: The self-driving AI adjusts test priorities based on the user's emotions. For example, if the user is feeling scared, it will prioritize testing responses to accidents and dangerous situations.

[1091] Example 1

[1092] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1093] In the development of autonomous driving technology, testing in real road environments is time-consuming and costly, and it is difficult to cover all scenarios. Furthermore, it is not possible to provide a simulation environment that responds to the user's emotions, making it difficult to improve the user experience.

[1094] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1095] In this invention, the server includes: a means for generating virtual cars, people, and environments from natural language, video, etc. using a generative AI and inputting them into a digital twin; a means for conducting highly realistic and diverse simulations using the digital twin; a means for extensively and thoroughly testing autonomous driving AI using the simulation; a means for generating virtual cars, people, and environments based on a user's emotions using an emotion engine that recognizes the user's emotions; a means for the emotion engine to analyze the user's emotions from their facial expressions, tone of voice, choice of words, etc. and communicate the results to the generative AI; and a means for the generative AI to generate a virtual environment based on the analysis results. This reduces the time and cost required for testing in a real road environment and enables all scenarios to be covered. Furthermore, providing a simulation environment that responds to the user's emotions improves the user experience.

[1096] "Generative AI" is an artificial intelligence technology that analyzes information such as natural language and videos to generate virtual cars, people, and environments.

[1097] "Digital twin" is a technology that recreates physical objects and environments in a virtual space and performs simulations in real time.

[1098] "Simulation" is the process of replicating the behavior of autonomous vehicles and other elements in a virtual environment for testing and evaluation.

[1099] "Autonomous driving AI" is an artificial intelligence technology used to control and make decisions about autonomous vehicles.

[1100] An "emotion engine" is a technology that analyzes emotions from a user's facial expressions, tone of voice, choice of words, etc., and transmits the results to other systems.

[1101] "User emotions" refers to the psychological state that a user expresses through facial expressions, tone of voice, choice of words, etc.

[1102] A "virtual environment" is a simulated setting that includes elements such as cars, people, and the environment in a digital space generated by generative AI.

[1103] MODE FOR CARRYING OUT THE INVENTION

[1104] This invention is a system that uses a generative AI model to analyze information such as natural language and video to generate virtual cars, people, and environments. The generated virtual environment is input into a digital twin, where realistic and diverse simulations are performed. Furthermore, an emotion engine that recognizes the user's emotions can be used to generate a virtual environment that corresponds to the user's emotions.

[1105] Hardware and software used

[1106] The server uses Google Cloud's natural language processing API and OpenAI's GPT-4 as generative AI models. Game engines such as Unity and Unreal Engine are used to implement the digital twin. The emotion engine includes software for analyzing the user's facial expressions, tone of voice, and word choice.

[1107] Data processing and calculation

[1108] The server receives natural language and video input from the user and analyzes it using a generative AI model. Based on the analysis results, virtual cars, people, and environments are generated. The generated data is input into a digital twin, where a simulation is carried out in real time. The emotion engine analyzes the user's emotions and conveys the results to the generative AI. Based on the analysis results, the generative AI generates a virtual environment that corresponds to the user's emotions.

[1109] Specific examples

[1110] If a user inputs into the system, "I would like to simulate autonomous driving in an urban environment on a sunny day," the server will analyze this natural language and generate an urban environment on a sunny day. The generated environment is then input into the digital twin, and the simulation begins.

[1111] Furthermore, when using the emotion engine, emotions are analyzed from the user's facial expressions, tone of voice, choice of words, etc. For example, if a user says, "I'm in a good mood today," the emotion engine will interpret this as the user being happy, and the generation AI will generate a virtual environment with bright weather.

[1112] Prompt Sentence Examples

[1113] "I want to simulate autonomous driving in an urban environment on a sunny day."

[1114] "I want to simulate autonomous driving in a suburban environment on a rainy day."

[1115] "I'm feeling good today, so I want to see a bright weather simulation."

[1116] In this way, by having the server, terminal, and user work together to execute system processing, it is possible to reduce the time and cost required for testing in a real road environment and cover all scenarios.In addition, by providing a simulation environment that responds to the user's emotions, it is possible to improve the user experience.

[1117] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1118] Step 1:

[1119] The user provides input

[1120] The user inputs text into the system interface, such as "I would like to simulate autonomous driving in an urban environment on a sunny day," or speaks a similar message using voice input. The input data can be text data or voice data.

[1121] Step 2:

[1122] The server receives the input

[1123] The server receives text and voice input from the user. In the case of voice input, it first converts it into text using a speech recognition API. The input data is saved as text data.

[1124] Step 3:

[1125] The server parses the input

[1126] The server then sends the received text data to Google Cloud's natural language processing API and obtains the analysis results. The analysis uses natural language processing technology to understand the user's request. The analysis results include information about the simulation environment the user desires.

[1127] Step 4:

[1128] The server creates a virtual environment

[1129] The server uses Unity to generate a sunny urban environment based on the analysis results. Specifically, it creates a 3D model of the city and sets the weather conditions. The generated virtual environment data is then input into the digital twin.

[1130] Step 5:

[1131] The server populates the digital twin with a virtual environment

[1132] The server sends the generated urban environment data to the digital twin platform, which prepares the simulation in real time.Digital twin is a platform for simulating virtual environments in real time.

[1133] Step 6:

[1134] The server runs the simulation

[1135] The server then runs a simulation of the autonomous vehicle on the digital twin, simulating, for example, how the vehicle behaves when passing through an intersection and interacts with other vehicles and pedestrians. Simulation results are generated in real time.

[1136] Step 7:

[1137] The terminal displays the simulation results.

[1138] The device displays the simulation results sent from the server in real time, allowing the user to see how the autonomous vehicle navigates through an urban environment on the device screen. The displayed data includes the progress and results of the simulation.

[1139] In this way, by having the server, terminal, and user work together to execute system processing, it is possible to reduce the time and cost required for testing in a real road environment and cover all scenarios.In addition, by providing a simulation environment that responds to the user's emotions, it is possible to improve the user experience.

[1140] (Application example 1)

[1141] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1142] In the development of autonomous vehicles, testing in real road environments is not only time-consuming and costly, but can also be dangerous. Furthermore, it is difficult to reproduce various scenarios and weather conditions, limiting the scope of testing. Furthermore, it is not possible to provide a real-time simulation environment that responds to user emotions, so there is a need to improve the user experience.

[1143] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1144] In this invention, the server includes: means for generating virtual cars, people, and environments from natural language, video, etc. using a generative AI and inputting them into a digital twin; means for implementing highly realistic and diverse simulations using the digital twin; means for extensively and thoroughly testing autonomous driving AI using the simulations; means for analyzing user emotions using an emotion engine and generating a virtual environment based on the analysis results; and means for displaying the simulation results in real time using a smartphone or head-mounted display and allowing the user to operate an autonomous driving vehicle in the virtual environment. This makes it possible to reproduce various scenarios and weather conditions and provide a real-time simulation environment that responds to the user's emotions.

[1145] "Generative AI" is an artificial intelligence that analyzes information such as natural language and videos, and generates virtual cars, people, and environments.

[1146] "Digital twin" is a technology that recreates and simulates physical objects and environments in a virtual space.

[1147] The "emotion engine" is a system that analyzes emotions from the user's facial expressions, tone of voice, choice of words, etc.

[1148] The "virtual environment" is a simulated environment within the digital twin generated by generative AI.

[1149] "Autonomous driving AI" is artificial intelligence used to control the operation of autonomous vehicles.

[1150] A "smartphone" is a portable information terminal that, in addition to the functions of a mobile phone, can also connect to the Internet and run applications.

[1151] A "head-mounted display" is a device worn on the user's head that displays images in the user's field of vision.

[1152] "Simulation" is the process of recreating and testing the behavior of autonomous vehicles in a virtual environment.

[1153] "Real-time" refers to processing and display occurring immediately without delay.

[1154] "User" refers to a person who uses the system.

[1155] The system for implementing this invention uses generative AI, a digital twin, an emotion engine, a smartphone, and a head-mounted display (HMD). The specific configuration and operation of the system are described below.

[1156] System configuration

[1157] 1. Generative AI: Analyzes information such as natural language and video to generate virtual cars, people, and environments.

[1158] 2. Digital Twin: Reproducing and simulating real-world physical objects and environments in a virtual space.

[1159] 3. Emotion engine: Analyzes emotions from the user's facial expressions, tone of voice, choice of words, etc.

[1160] 4. Smartphone: A mobile information terminal that has the functionality of a mobile phone and can also connect to the Internet and run applications.

[1161] 5. Head-mounted display (HMD): A device worn on the user's head that displays images in the user's field of vision.

[1162] System Operation

[1163] 1. Data entry and analysis:

[1164] The server receives natural language and video input from the user.

[1165] Generative AI analyzes this data and generates virtual cars, people, and environments.

[1166] 2. Emotion analysis:

[1167] The server uses an emotion engine to analyze the user's facial expressions and tone of voice.

[1168] The emotion engine communicates the analysis results to the generative AI.

[1169] 3. Create a virtual environment:

[1170] The generative AI generates a virtual environment based on the analysis results from the emotion engine.

[1171] For example, if the user is happy, a bright virtual environment is generated, and if the user is sad, a rainy virtual environment is generated.

[1172] 4. Run the simulation:

[1173] The server inputs the generated virtual environment into the digital twin and performs a simulation.

[1174] The digital twin simulates the behavior of an autonomous vehicle in a generated virtual environment in real time.

[1175] 5. Viewing and manipulating results:

[1176] Using a smartphone or HMD, users can check the simulation results in real time.

[1177] Users can operate autonomous vehicles in a virtual environment and test them in various scenarios.

[1178] Specific examples

[1179] When a user simulates an autonomous vehicle using their smartphone, a camera captures their facial expressions and an emotion engine detects "happiness." The generative AI generates a sunny urban environment and runs the simulation in the digital twin. The user can view the simulation results in real time on their smartphone screen and control the vehicle's behavior.

[1180] Prompt Sentence Examples

[1181] "If the user is happy, generate an urban environment with sunny weather and run a simulation of an autonomous vehicle."

[1182] In this way, the embodiment of the invention can reproduce various scenarios and weather conditions and provide a real-time simulation environment that responds to the user's emotions.

[1183] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1184] Step 1:

[1185] The server receives natural language and video input from the user. The input data is information about the scenario and environment the user wants to use in the simulation. The server then passes this data to the generation AI.

[1186] Step 2:

[1187] The generative AI analyzes the natural language and video it receives to generate virtual cars, people, and environments. Specifically, it uses natural language processing technology to analyze text data and video analysis technology to analyze video data. The generated virtual cars, people, and environments are then input into the digital twin.

[1188] Step 3:

[1189] The server uses an emotion engine to analyze the user's facial expressions and tone of voice. The input data is the user's facial image and voice data. The emotion engine analyzes this data and identifies the user's emotional state. The analysis results are transmitted to the generation AI.

[1190] Step 4:

[1191] The generative AI generates a virtual environment based on the analysis results from the emotion engine. For example, if the user is happy, it generates a virtual environment with bright weather, and if they are sad, it generates a virtual environment with rain. The generated virtual environment is then input into the digital twin.

[1192] Step 5:

[1193] The server inputs the generated virtual environment into the digital twin and performs a simulation. The digital twin simulates the behavior of the autonomous vehicle in the generated virtual environment in real time. The input data is the initial state of the virtual environment and the autonomous vehicle. The output data is the simulation results.

[1194] Step 6:

[1195] The terminal (smartphone or HMD) receives the simulation results sent from the server and displays them in real time. The user can check the simulation results through the terminal and operate the autonomous vehicle in the virtual environment. The input data are the simulation results, and the output data are the user's operation instructions.

[1196] Step 7:

[1197] The user operates the autonomous vehicle in the virtual environment using a terminal. The user's operation instructions are sent to the server and reflected in the digital twin, thereby updating the simulation in real time. The input data are the user's operation instructions, and the output data are the updated simulation results.

[1198] Example 2

[1199] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1200] When testing autonomous driving AI, it takes a lot of time and money to recreate real-world road conditions. It is also difficult to adjust scenarios based on user emotions, which reduces the accuracy and efficiency of testing. Furthermore, conducting extensive and thorough testing requires the generation of diverse scenarios, which requires advanced technology.

[1201] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1202] In this invention, the server includes: means for generating virtual cars, humans, and environments from natural language, video, etc. using a generative AI model and inputting them into a digital twin; means for performing highly realistic and diverse simulations using the digital twin; means for extensive and thorough testing of an autonomous driving AI using the simulation; means for a user to input a prompt; means for a terminal to send the prompt to the server; means for the server to generate a scenario using the generative AI model; means for the server to send the generated scenario to the terminal; means for the terminal to display the scenario to the user; means for the user to adjust the scenario using an emotion engine; means for the server to regenerate the scenario based on information from the emotion engine; means for the server to send the regenerated scenario to the terminal; and means for the terminal to display the regenerated scenario to the user. This enables scenario adjustment according to the user's emotions, enabling extensive and thorough testing to be performed efficiently.

[1203] A "generative AI model" is an artificial intelligence model that generates virtual cars, people, and environments from input data such as natural language and video.

[1204] A "digital twin" is a digital model that recreates a real-world physical object or environment in a virtual space.

[1205] "Simulation" is the process of virtually recreating various real-world situations using digital twins for testing and analysis.

[1206] "Autonomous driving AI" is an artificial intelligence system that controls the driving of autonomous vehicles.

[1207] A "prompt" is an instruction that a user inputs to a generative AI model, and is text that prompts the generation of a scenario.

[1208] A "terminal" is a device that allows a user to input prompts and check scenarios.

[1209] A "server" is a computer system that runs generative AI models and generates and transmits scenarios.

[1210] The "emotion engine" is a system that analyzes the user's emotional information and adjusts the scenario based on that information.

[1211] "Regeneration" is the process of regenerating a scenario based on information from the emotion engine.

[1212] This invention is a system for efficiently and effectively testing autonomous driving AI. Specific embodiments of this system will be described below.

[1213] First, the user inputs a prompt using the terminal. A prompt is an instruction for the generative AI model to generate a scenario. For example, a prompt such as "Please generate a scenario for turning right at an intersection" can be input.

[1214] The device sends the input prompt to the server. This communication uses protocols such as HTTP requests. The server generates a scenario based on the received prompt using a generative AI model. For example, GPT-4 is used as the generative AI model.

[1215] The server sends the generated scenario to the terminal. The terminal displays the received scenario to the user. The user checks the displayed scenario and adjusts it as necessary using the emotion engine. The emotion engine is a system that analyzes the user's emotion information and regenerates the scenario based on that information.

[1216] For example, if a user types "I'm angry," the emotion engine relays that information to the digital twin, which then prioritizes simulating scenarios that are likely to anger the user, such as traffic congestion or rude drivers.

[1217] The server regenerates the scenario based on the information received from the emotion engine. The regenerated scenario is then sent back to the device, which displays it to the user. This allows the user to adjust the scenario according to the emotion, enabling efficient and extensive testing.

[1218] As a concrete example, consider the case where a user wants to generate a scenario for testing an autonomous vehicle. If the user inputs "Please generate a scenario for turning right at an intersection," the server will generate the following scenario:

[1219] Examples:

[1220] Scenario 1: A self-driving car is about to turn right at an intersection when an oncoming vehicle suddenly appears.

[1221] Scenario 2: When turning right at an intersection, a pedestrian suddenly jumps into the crosswalk.

[1222] Scenario 3: Another vehicle is approaching while turning right, posing a risk of collision.

[1223] Example prompt sentence:

[1224] "Generate a right turn scenario at an intersection."

[1225] "Create a scenario where a pedestrian suddenly jumps out."

[1226] "We want to test close-communication scenarios with other vehicles."

[1227] In this way, users can efficiently generate test scenarios for their autonomous driving AI and even adjust them according to emotions.

[1228] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1229] Step 1:

[1230] The user enters a prompt statement.

[1231] The user inputs a prompt sentence such as "Please generate a scenario for turning right at an intersection" into an input field on the terminal. The input prompt sentence is sent to the terminal as an instruction to generate a scenario.

[1232] Step 2:

[1233] The terminal sends the prompt to the server.

[1234] The terminal sends the prompt text entered by the user to the server as an HTTP POST request. This request includes the prompt text. The server receives this request and analyzes the prompt text.

[1235] Step 3:

[1236] The server generates scenarios using generative AI models.

[1237] The server generates a scenario using a generative AI model (e.g., GPT-4) based on the received prompt. The generative AI model creates an appropriate scenario based on the prompt. For example, when generating a "right turn scenario at an intersection," it generates a scenario in which an oncoming vehicle suddenly appears.

[1238] Step 4:

[1239] The server sends the generated scenario to the terminal.

[1240] The server sends the generated scenario to the terminal in JSON format. This communication also uses protocols such as HTTP requests. The terminal receives this request and analyzes the scenario data.

[1241] Step 5:

[1242] The terminal displays the scenario to the user.

[1243] The terminal displays the received scenario to the user. The user checks the scenario on the screen. For example, if the generated scenario is "An oncoming vehicle suddenly appears while turning right at an intersection," the details are displayed.

[1244] Step 6:

[1245] The user reviews the scenario and adjusts it using the emotion engine if necessary.

[1246] The user reviews the displayed scenario and adjusts it as needed using the emotion engine. For example, if the user types "I'm angry," the emotion engine relays that information to the digital twin.

[1247] Step 7:

[1248] The server regenerates the scenario based on the information from the emotion engine.

[1249] The server regenerates scenarios based on the information received from the emotion engine. For example, if the user is angry, it will prioritize scenarios that are likely to anger the user, such as traffic congestion or rude drivers.

[1250] Step 8:

[1251] The server transmits the regenerated scenario to the terminal.

[1252] The server sends the regenerated scenario to the terminal in JSON format. This communication also uses protocols such as HTTP requests. The terminal receives this request and analyzes the regenerated scenario data.

[1253] Step 9:

[1254] The terminal displays the regenerated scenario to the user.

[1255] The terminal displays the regenerated scenario received from the server to the user. The user checks the regenerated scenario. For example, if the regenerated scenario is "a scenario involving traffic congestion and rude behavior by drivers," details of the scenario are displayed.

[1256] (Application example 2)

[1257] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1258] As autonomous driving technology advances, testing and validation of autonomous driving AI is becoming increasingly important. However, testing in real-world road environments is time-consuming, costly, and poses safety issues. Furthermore, it is difficult to conduct tests that closely resemble real-world driving conditions because it is not possible to adjust scenarios based on user emotions.

[1259] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1260] In this invention, the server includes means for generating virtual cars, people, and environments from natural language, video, etc. using a generative AI and inputting them into a digital twin, means for implementing highly realistic and diverse simulations using the digital twin, means for extensively and thoroughly testing an autonomous driving AI using the simulations, means for adjusting the simulation scenario according to a user's emotions, and means for displaying the generated scenario on a smartphone. This enables testing of autonomous driving AI in realistic scenarios according to a user's emotions, enabling more realistic testing while reducing time and costs.

[1261] "Generative AI" is an artificial intelligence technology that generates virtual cars, people, and environments from natural language, videos, etc.

[1262] A "digital twin" is a digital model that recreates a real-world physical object in a virtual space.

[1263] "Simulation" is the process of using digital twins to recreate various situations in a virtual environment for testing and verification.

[1264] "Autonomous driving AI" is an artificial intelligence technology that enables cars to drive autonomously.

[1265] "User emotion" is the user's psychological state that is taken into account to adjust the simulation scenario.

[1266] A "scenario" is a specific situation or environment that is reproduced in a simulation.

[1267] A "smartphone" is a portable information terminal that can run a variety of applications in addition to the functions of a mobile phone.

[1268] Systems for implementing this invention include generative AI, digital twins, simulations, autonomous driving AI, user emotions, scenarios, and smartphones.

[1269] The server uses generative AI to generate virtual cars, people, and environments from natural language and video, and then inputs these into a digital twin. A digital twin is a digital model that recreates a real-world physical object in a virtual space, enabling highly realistic and diverse simulations.

[1270] Simulation is the process of using a digital twin to recreate various situations in a virtual environment for testing and validation, allowing for extensive and thorough testing of autonomous driving AI.

[1271] The system further includes means for adjusting the simulation scenario in response to the user's emotions, which are psychological states that are taken into consideration for adjusting the simulation scenario, so that if the user is angry, for example, a scenario in which the user encounters traffic congestion or rude drivers is generated.

[1272] The generated scenario is displayed on a smartphone, a mobile information terminal that can run a variety of applications in addition to the functions of a mobile phone, allowing the user to check the scenario and receive feedback on the test results in real time.

[1273] For example, if the user is "angry," the generated scenario will be "encountering heavy traffic and rude drivers." An example of a prompt sentence is "Generate a driving scenario where the autonomous vehicle encounters heavy traffic and rude drivers."

[1274] In this way, it becomes possible to test autonomous driving AI in realistic scenarios that respond to user emotions, enabling more realistic testing while reducing time and costs.

[1275] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1276] Step 1:

[1277] The server acquires the user's emotions. As input, it receives the user's emotion data (e.g., "angry," "happy," etc.) and passes the emotion data to the next step as output. Specifically, it either randomly selects the user's emotion or receives direct input from the user.

[1278] Step 2:

[1279] The server creates a scenario generation prompt based on the acquired emotional data. It receives the user's emotional data as input and obtains the generated prompt as output. Specifically, it performs conditional branching to generate a prompt according to the emotion. For example, if the emotion is "angry," it generates the prompt "Generate a driving scenario where the autonomous vehicle encounters heavy traffic and rude drivers."

[1280] Step 3:

[1281] The server uses the generated prompt sentence to request the generative AI model to generate a scenario. It receives the prompt sentence as input and obtains the generated scenario as output. Specifically, it sends the prompt sentence using the OpenAI API and receives the generated scenario.

[1282] Step 4:

[1283] The server sends the generated scenario to the smartphone. It receives the generated scenario as input and obtains scenario data to be displayed on the smartphone as output. Specifically, it sends the scenario data to the smartphone application so that the user can check it.

[1284] Step 5:

[1285] The user checks the scenario displayed on the smartphone and receives feedback on the test results in real time. The system receives the scenario displayed on the smartphone as input and sends feedback data to the server as output. Specifically, the user checks the scenario and enters feedback as necessary.

[1286] Step 6:

[1287] The server receives feedback data from users and analyzes the simulation results. It receives the feedback data as input and obtains the analysis results as output. Specifically, it analyzes the feedback data and identifies areas for improvement and adjustment of the autonomous driving AI.

[1288] Example 3

[1289] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1290] In the development of autonomous driving AI, testing in real-world environments is time-consuming and costly, and it is difficult to reproduce dangerous situations. Furthermore, because it is not possible to adjust test priorities based on user emotions, tests do not adequately consider user anxiety and fear. Therefore, efficient testing methods are needed to improve the safety and reliability of autonomous driving AI.

[1291] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means. In this invention, the server includes: means for generating virtual cars, people, and environments from natural language, videos, etc. using a generation AI and inputting them into a digital twin; means for implementing highly realistic and diverse simulations using the digital twin; means for extensively and thoroughly testing the autonomous driving AI using the simulations; means for adjusting test priorities using an emotion engine that collects and analyzes user emotion data; and means for setting simulation scenarios based on the analysis results from the emotion engine. This enables effective testing according to user emotions, improving the safety and reliability of the autonomous driving AI.

[1292] "Generative AI" is an artificial intelligence technology that generates virtual cars, people, and environments from data such as natural language and video.

[1293] A "digital twin" is a digital model that recreates a real-world physical environment or object in a virtual space.

[1294] "Simulation" is the process of using a digital twin to recreate various scenarios in a virtual environment for testing and evaluation.

[1295] "Autonomous driving AI" is an artificial intelligence technology for automating automobile driving.

[1296] The "emotion engine" is a system for collecting and analyzing user emotional data.

[1297] "Emotional data" is information that indicates the user's emotional state, and is collected through questionnaires, voice input, biometric sensors, etc.

[1298] A "simulation scenario" is a scenario that sets out specific situations or conditions that are reproduced within a simulation.

[1299] The "analysis result" is the result obtained by the emotion engine analyzing the user's emotion data.

[1300] This invention is a system that significantly reduces the time and cost involved in testing autonomous driving AI through simulations using generative AI and digital twins. It also has the function of adjusting test priorities based on user emotions using an emotion engine.

[1301] System configuration

[1302] The system consists of the following main components:

[1303] 1. Generative AI Models

[1304] 2. Digital Twin

[1305] 3. Emotion Engine

[1306] 4. Server

[1307] 5. Terminal

[1308] Hardware and software used

[1309] Generative AI model: Used to generate virtual cars, people, and environments from natural language, videos, etc. Specifically, a general generative AI model (e.g., GPT-4) is used.

[1310] Digital Twin: A digital model that recreates a real-world environment in a virtual space, faithfully recreating the physical environment and traffic conditions.

[1311] Emotion engine: A system for collecting and analyzing user emotional data. Emotion data is collected through questionnaires, voice input, biometric sensors, etc.

[1312] Server: Runs simulations using generative AI models and digital twins, and sets up simulation scenarios based on data from the emotion engine.

[1313] Terminal: A device through which a user inputs emotion data, such as a smartphone or tablet.

[1314] Program processing

[1315] 1. The user inputs emotion data

[1316] Users use a device to input emotion data, which can be input in multiple ways, such as through a questionnaire, voice input, or even data collection using biometric sensors.

[1317] 2. The device sends emotion data to the emotion engine.

[1318] The device transmits the emotion data entered by the user to the emotion engine in real time via an internet connection.

[1319] 3. The emotion engine analyzes the emotion data

[1320] The emotion engine analyzes the received emotion data and identifies the user's emotional state. For example, if the user inputs "I'm scared of driving on rainy days," the emotion engine identifies the emotion "fear."

[1321] 4. The emotion engine sends the analysis results to the server

[1322] The emotion engine sends the analysis results to the server, which include the user's emotional state and its details.

[1323] 5. The server generates the digital twin

[1324] The server uses the generative AI model to generate a digital twin that mimics the real-world environment, a virtual reproduction of the physical environment and traffic conditions.

[1325] 6. The server sets up the simulation scenario

[1326] The server sets a simulation scenario based on the analysis results received from the emotion engine. For example, if the user feels that "driving on a rainy day is scary," the server sets a simulation scenario for rainy weather.

[1327] 7. The server runs simulation tests of the autonomous driving AI.

[1328] The server runs simulation tests of the autonomous driving AI based on the set simulation scenario. The simulation is performed on a digital twin, faithfully reproducing the real-world environment.

[1329] 8. The server records the test results and repeats the scenario as needed.

[1330] The server records the results of the simulation test. If necessary, the same scenario can be repeatedly tested and the results compared and analyzed to evaluate the performance of the autonomous driving AI and identify areas for improvement.

[1331] Specific examples

[1332] If a user feels scared of driving on rainy days, the emotion engine analyzes this information and sends it to the server, which then sets up a rainy weather simulation scenario on the digital twin and tests the autonomous driving AI.

[1333] Prompt Sentence Examples

[1334] Example prompts to be input to the generative AI model:

[1335] "Please generate a simulation scenario for autonomous driving in urban areas in rainy weather, especially including situations with high risk of sudden braking and skidding."

[1336] In this way, effective testing according to the user's emotions becomes possible, and the safety and reliability of the autonomous driving AI are improved. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1337] Step 1:

[1338] The user inputs emotional data. The user inputs emotional data using a device. Emotional data can be input in multiple ways, such as through a questionnaire, voice input, or even data collection using biometric sensors. For example, the user opens a dedicated app on their smartphone and inputs, "I'm afraid of driving on rainy days." The input data is saved on the device in text format or as voice data.

[1339] Step 2:

[1340] The device sends the emotion data to the emotion engine. The device sends the emotion data entered by the user to the emotion engine in real time. This is done via an internet connection. Specifically, the device sends the emotion data to the emotion engine using the HTTPS protocol. The input data is sent to the emotion engine in text format or as voice data.

[1341] Step 3:

[1342] The emotion engine analyzes the emotion data. The emotion engine analyzes the received emotion data and identifies the user's emotional state. For example, from the input "I'm afraid of driving on rainy days," it identifies the emotion "fear." The emotion engine analyzes the text data using natural language processing technology and extracts the emotional state. The analysis results are generated in JSON format.

[1343] Step 4:

[1344] The emotion engine sends the analysis results to the server. The emotion engine sends the analysis results to the server. The analysis results include the user's emotional state and its details. Specifically, the emotion engine sends the analysis results to the server in JSON format. An internet connection is used for transmission.

[1345] Step 5:

[1346] The server generates a digital twin. Using a generative AI model, the server generates a digital twin that mimics a real-world environment. A digital twin is a virtual reproduction of a physical environment and traffic conditions. For example, the server inputs a prompt statement to the generative AI model, such as "Please generate a simulation scenario for autonomous driving in urban areas on rainy days," and generates a digital twin. The generated digital twin is saved on the server as a virtual environment.

[1347] Step 6:

[1348] The server sets the simulation scenario. The server sets the simulation scenario based on the analysis results received from the emotion engine. For example, if the user feels "afraid of driving on rainy days," the server sets a simulation scenario for rainy weather. Specifically, the server sets a scenario on the digital twin that includes situations with a high risk of sudden braking and skidding. The set scenario is saved on the server as input data for the simulation.

[1349] Step 7:

[1350] The server runs simulation tests of the autonomous driving AI. The server runs simulation tests of the autonomous driving AI based on the set simulation scenario. The simulation is performed on the digital twin and faithfully reproduces the real-world environment. For example, the server reproduces a scenario of an urban area on rainy weather on the digital twin to test the behavior of the autonomous driving AI. The results of the simulation are recorded on the server.

[1351] Step 8:

[1352] The server records the test results and repeats the scenario as necessary. The server records the results of the simulation test in a database. If necessary, the same scenario is repeatedly tested and the results are compared and analyzed. For example, the server collects data such as stopping distance during sudden braking and the frequency of skidding, and compares the results of multiple tests. This allows the performance of the autonomous driving AI to be evaluated and areas for improvement identified.

[1353] (Application example 3)

[1354] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1355] In the development of autonomous driving AI, testing in real-world environments is time-consuming, costly, and potentially dangerous. Furthermore, it is difficult to adjust test priorities based on user sentiment, making it difficult to conduct effective testing to ensure user safety and reliability.

[1356] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 3 is realized by the following means. In this invention, the server includes: means for generating virtual cars, people, and environments from natural language, videos, etc. using a generative AI and inputting them into a digital twin; means for implementing highly realistic and diverse simulations using the digital twin; means for extensive and thorough testing of the autonomous driving AI using the simulations; and means for analyzing user emotions using an emotion engine and for the generative AI to provide optimal test scenarios. This enables effective testing based on user emotions, reducing the time and cost required for developing autonomous driving AI and improving safety and reliability.

[1357] "Generative AI" is an artificial intelligence technology that generates virtual cars, people, and environments from data such as natural language and video.

[1358] "Digital twin" is a technology for recreating real-world physical objects in a virtual space and performing simulations.

[1359] "Simulation" is a method of simulating real-world situations in a virtual environment for testing and analysis.

[1360] "Autonomous driving AI" is an artificial intelligence technology that enables cars to drive autonomously.

[1361] An "emotion engine" is a technology that analyzes the user's emotions and adjusts the system's behavior based on that information.

[1362] A "test scenario" is a plan or scenario for setting specific situations or conditions in a simulation and testing a system under those conditions.

[1363] As an embodiment of the present invention, an emotion-responsive automated driving test simulator will be described as an example.

[1364] The server uses generative AI to generate virtual vehicles, people, and environments from natural language, video, and other data, and then inputs them into the digital twin. The digital twin recreates real-world physical objects in a virtual space, allowing for highly realistic and diverse simulations. This allows for extensive and thorough testing of autonomous driving AI.

[1365] The server then uses an emotion engine to analyze the user's emotions. The emotion engine uses the smartphone's camera and microphone to analyze the user's facial expressions and voice to detect the user's emotions in real time. The detected emotion data is sent to the generative AI and used to generate optimal test scenarios.

[1366] For example, if the user is feeling fear, the emotion engine will convey that information to the generative AI, which will then generate scenarios that prioritize testing responses to accidents and dangerous situations. This enables effective testing based on user emotions, improving the safety and reliability of autonomous driving AI.

[1367] The hardware used includes a smartphone (camera, microphone), and the software used includes emotion recognition software (e.g., Microsoft Azure Face API) and a Python program.

[1368] As a concrete example, consider a scenario in which a user uses a smartphone to conduct a simulation test of an autonomous vehicle. When the user uses the smartphone to conduct a simulation test of an autonomous vehicle, an emotion engine analyzes the user's emotions, and a generative AI is designed to provide the optimal test scenario.

[1369] Example prompt sentence:

[1370] "Design an application where a user uses a smartphone to simulate a self-driving vehicle and an emotion engine analyzes the user's emotions, and a generative AI provides the optimal test scenario."

[1371] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1372] Step 1:

[1373] The server uses the smartphone's camera and microphone to collect the user's facial expressions and voice in real time. The input is the user's facial expression and voice data, and the output is to send this data to the emotion engine. Specifically, the smartphone's camera captures the user's face and the microphone records the user's voice.

[1374] Step 2:

[1375] The server uses an emotion engine to analyze the collected facial and voice data and detect the user's emotions. The input is the facial and voice data collected in step 1, and the output is the detected user emotion (e.g., joy, sadness, anger, fear, etc.). Specifically, emotion recognition software (e.g., Microsoft Azure Face API) analyzes the facial expression data, and voice analysis software analyzes the tone of voice.

[1376] Step 3:

[1377] The server sends the detected user emotion data to the generation AI. The input is the user emotion data detected in step 2, and the output is the emotion data sent to the generation AI. Specifically, the emotion engine sends the emotion data to the generation AI's API.

[1378] Step 4:

[1379] The server uses generative AI to generate an optimal test scenario based on the user's emotions. The input is the user's emotion data sent in step 3, and the output is the generated test scenario. Specifically, the generative AI model analyzes the emotion data and generates an appropriate scenario (e.g., emergency braking, pedestrian crossing, etc.).

[1380] Step 5:

[1381] The server inputs the generated test scenario into the digital twin and performs a simulation. The input is the test scenario generated in step 4, and the output is the simulation result. Specifically, the digital twin creates a virtual environment and simulates the behavior of the autonomous driving AI based on the scenario.

[1382] Step 6:

[1383] The server analyzes the simulation results and evaluates the performance of the autonomous driving AI. The input is the simulation results obtained in Step 5, and the output is the evaluation results. Specifically, the server analyzes the simulation results and evaluates performance indicators such as the AI's reaction time and accuracy.

[1384] Step 7:

[1385] The server feeds back the evaluation results to the user. The input is the evaluation results obtained in step 6, and the output is the feedback information provided to the user. Specifically, the server visually displays the evaluation results and informs the user of areas for improvement and success.

[1386] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1387] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1388] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1389] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1390] [Third embodiment]

[1391] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1392] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1393] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1394] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1395] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1396] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1397] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1398] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1399] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1400] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1401] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1402] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1403] "Example 1"

[1404] In one embodiment of the present invention, generative AI receives and analyzes information such as natural language and video as input. Based on the results of the analysis, it generates virtual vehicles, people, and environments. The generated virtual vehicles, people, and environments are then fed into a digital twin, which uses them to perform highly realistic and diverse simulations. For example, it can recreate various scenarios that an autonomous vehicle may encounter, such as urban and suburban environments, various weather conditions, and traffic situations.

[1405] "Example 2"

[1406] In another embodiment of the present invention, the Generative AI generates scenarios for testing the autonomous driving AI. The generated scenarios are designed to cover a variety of scenarios that an autonomous vehicle may encounter on the road. This allows the autonomous driving AI to be extensively and thoroughly tested and improved or adjusted based on the results. For example, the autonomous vehicle can handle a variety of scenarios, such as turning right at an intersection, a pedestrian suddenly stepping out, or approaching another vehicle.

[1407] "Example 3"

[1408] In a further embodiment of the present invention, simulation using generative AI and digital twins can significantly reduce the time and cost associated with testing autonomous driving AI. Specifically, simulation testing can be performed more quickly and efficiently than testing in real-world environments, and scenarios can be repeatedly tested as needed. This accelerates the process of developing and improving autonomous driving AI, thereby improving the safety and reliability of autonomous vehicles.

[1409] The processing flow of each embodiment will be described below.

[1410] "Example 1"

[1411] Step 1: The generative AI receives information such as natural language and video as input.

[1412] Step 2: The generation AI analyzes the input information and generates virtual cars, people, and environments based on the results of the analysis.

[1413] Step 3: The generated virtual vehicles, humans, and environments are fed into a digital twin.

[1414] Step 4: The digital twin performs highly realistic and diverse simulations using virtual vehicles, people, and environments.

[1415] "Example 2"

[1416] Step 1: The generation AI generates scenarios for testing the autonomous driving AI.

[1417] Step 2: The generated scenarios are designed to cover a variety of scenarios that an autonomous vehicle may encounter on the road.

[1418] Step 3: The self-driving AI is extensively and thoroughly tested using generated scenarios.

[1419] Step 4: Based on the results of the testing, improvements and adjustments are made to the self-driving AI.

[1420] "Example 3"

[1421] Step 1: Using generative AI and digital twin simulations to significantly reduce the time and cost associated with testing autonomous driving AI.

[1422] Step 2: Simulation testing is fast and efficient, allowing scenarios to be tested repeatedly as needed.

[1423] Step 3: This will accelerate the process of developing and improving self-driving AI, thereby improving the safety and reliability of self-driving cars.

[1424] Example 1

[1425] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1426] In the development of autonomous driving technology, testing in actual road environments is time-consuming and costly, and it is difficult to ensure safety. Furthermore, to fully implement countermeasures for various scenarios, simulations under a variety of environments and conditions are required, but there is a lack of efficient means to carry this out.

[1427] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1428] In this invention, the server includes a means for a user to input a prompt sentence, a means for the terminal to send the input data to the server, a means for the server to analyze the data using a generative AI model, a means for the server to generate a virtual object based on the analysis results, a means for the server to input the generated virtual object into a digital twin, a means for the digital twin to perform a simulation, a means for the server to send the simulation results to the terminal, and a means for the terminal to display the simulation results to the user. This enables advanced and diverse simulations that reproduce actual road environments, reducing the time and cost in the development of autonomous driving technology, improving safety, and strengthening countermeasures for various scenarios.

[1429] A "user" is an entity that uses the system to input prompt statements and check the simulation results.

[1430] A "terminal" is a device through which a user enters prompts and communicates with a server.

[1431] A "server" is a computer system that has the ability to analyze data using generative AI models, generate virtual objects, and input them into a digital twin.

[1432] A "generative AI model" is an artificial intelligence model that analyzes information such as natural language and video and generates virtual objects.

[1433] A "prompt sentence" is an input sentence that describes in natural language the scenario that the user wants to simulate.

[1434] "Virtual objects" are digital data such as virtual cars, people, and environments generated by generative AI models.

[1435] A "digital twin" is a simulation system that uses virtual objects to recreate real-world environments and situations with a high degree of realism.

[1436] "Simulation" is the process of using a digital twin to recreate the behavior and interactions of virtual objects and test various scenarios.

[1437] "Simulation results" are data and information obtained as a result of a simulation performed by a digital twin.

[1438] This invention begins when a user inputs a prompt and the device sends the data to a server. The server analyzes the data using a generative AI model and generates a virtual object. The generated virtual object is then input into a digital twin, which then uses it to perform a simulation. The simulation results are then sent from the server to the device and ultimately displayed to the user.

[1439] Specifically, the user uses a terminal to input a prompt statement, which is a natural language description of the scenario they want to simulate. For example, the prompt statement might be, "I want to simulate an autonomous vehicle in an urban environment on a rainy day," or "I want to simulate an autonomous vehicle in a mountainous area on a snowy day."

[1440] The device sends the prompt text entered by the user to the server. At this time, the device appropriately converts the data format and sends it according to the communication protocol. The server inputs the received prompt text into a generative AI model (for example, a natural language processing model or an image generation model) for analysis. The generative AI model understands the content of the prompt text and extracts the necessary information.

[1441] The server generates virtual vehicles, people, and environments based on the analysis results. In this process, the server uses image generation models to create realistic virtual objects. The generated virtual objects are then input into the digital twin system, which then prepares the system for simulation using these virtual objects.

[1442] A digital twin uses virtual objects to perform simulations, such as recreating the behavior of an autonomous vehicle in an urban environment on a rainy day. The results show how the vehicle behaves in the rain and how it interacts with other vehicles and pedestrians.

[1443] The server receives the results of the simulation performed by the digital twin and sends them to the terminal. The simulation results include the movement of the car and changes in the environment. The terminal displays the simulation results received from the server to the user. The user can check the simulation results and re-enter prompt statements if necessary.

[1444] This system enables advanced and diverse simulations that replicate actual road environments, reducing the time and cost required to develop autonomous driving technology, improving safety, and strengthening countermeasures for a variety of scenarios.

[1445] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1446] Step 1:

[1447] The user enters a prompt statement.

[1448] Specifically, the user describes the scenario they want to simulate in natural language in the input field of the device. For example, they might enter, "I want to simulate an autonomous vehicle in an urban environment on a rainy day."

[1449] Input: The prompt text entered by the user

[1450] Output: The prompt text entered in the terminal

[1451] Step 2:

[1452] The terminal sends the input data to the server.

[1453] Specifically, the terminal converts the prompt text entered by the user into an appropriate data format and sends it to the server in accordance with the communication protocol.

[1454] Input: The prompt text entered into the terminal

[1455] Output: The prompt sent to the server

[1456] Step 3:

[1457] The server analyzes the data using a generative AI model.

[1458] Specifically, the server inputs the received prompt into a generative AI model (e.g., a natural language processing model) and analyzes the content of the prompt. The generative AI model then understands the content of the prompt and extracts the necessary information.

[1459] Input: The prompt sent to the server

[1460] Output: Analysis results (required information)

[1461] Step 4:

[1462] The server generates a virtual object based on the analysis results.

[1463] Specifically, the server generates virtual vehicles, people, and environments based on the analysis results, using image generation models to create realistic virtual objects.

[1464] Input: Analysis results (required information)

[1465] Output: Generated virtual object

[1466] Step 5:

[1467] The virtual objects generated by the server are then inserted into the digital twin.

[1468] Specifically, the server inputs the generated virtual object into the digital twin system and prepares for the simulation.

[1469] Input: Generated virtual object

[1470] Output: Virtual objects fed into the digital twin

[1471] Step 6:

[1472] The digital twin performs the simulation.

[1473] Specifically, the digital twin uses virtual objects to perform simulations, such as recreating the behavior of an autonomous vehicle in an urban environment on a rainy day.

[1474] Input: Virtual objects fed into the digital twin

[1475] Output: Simulation results

[1476] Step 7:

[1477] The server transmits the simulation results to the terminal.

[1478] Specifically, the server receives the results of the simulation performed by the digital twin and sends them to the terminal.

[1479] Input: Simulation results

[1480] Output: Simulation results sent to the terminal

[1481] Step 8:

[1482] The terminal displays the simulation results to the user.

[1483] Specifically, the terminal displays the simulation results received from the server to the user, who can then check them and re-enter the prompt sentence if necessary.

[1484] Input: Simulation results sent to the terminal

[1485] Output: Simulation results displayed to the user

[1486] (Application example 1)

[1487] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1488] In the development of autonomous vehicles, testing in real road environments is time-consuming and costly, and it is difficult to cover all scenarios. In addition, there is a lack of means for users to simulate specific scenarios and check the behavior of autonomous vehicles.

[1489] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1490] In this invention, the server includes means for generating virtual cars, people, and environments from natural language and video using a generative AI and inputting them into a digital twin, means for implementing highly realistic and diverse simulations using the digital twin, means for extensively and thoroughly testing autonomous driving AI using the simulations, means for a user to input natural language and video and implement a simulation using the generated virtual elements, and means for providing the results of the simulation to the user, thereby enabling the user to simulate various scenarios and check the behavior of an autonomous driving car.

[1491] "Generative AI" is an artificial intelligence technology that analyzes information such as natural language and video to generate virtual cars, people, and environments.

[1492] A "digital twin" is a digital model that recreates a real-world physical object or environment in a virtual space.

[1493] "Simulation" is the process of recreating various real-world scenarios using virtual vehicles, people, and environments to verify their behavior.

[1494] "Autonomous driving AI" is an artificial intelligence technology used to control and make decisions about autonomous vehicles.

[1495] A "user" is a person or organization that uses the system to input natural language or video and perform simulations.

[1496] "Virtual elements" refer to virtual cars, people, and environments generated by generative AI.

[1497] "Simulation results" refers to the data and information obtained when a simulation is performed.

[1498] A system for implementing this invention includes elements such as generative AI, digital twin, simulation, autonomous driving AI, user, virtual element, and simulation results.

[1499] The server uses generative AI to analyze information such as natural language and video to generate virtual vehicles, people, and environments. These virtual elements are then fed into a digital twin, which then uses them to perform highly realistic and diverse simulations. Simulations enable extensive and thorough testing of autonomous driving AI.

[1500] Users input natural language or video using a device such as a smartphone. The input information is sent to a server and analyzed by generative AI. As a result of the analysis, a virtual car, human, and environment are generated. The generated virtual elements are input into a digital twin, where a simulation is performed. The simulation results are provided to the user, who can simulate various scenarios and check the behavior of the autonomous vehicle.

[1501] For example, if a user requests, "I want to simulate an autonomous vehicle in an urban environment on a sunny day," the generative AI will recreate the urban environment on a sunny day and run a simulation in the digital twin. The simulation results will be provided to the user, who can then check the behavior of the autonomous vehicle.

[1502] The hardware used includes smartphones and servers. The software used includes Python and OpenAI API. Data processing and calculation include information analysis using generative AI, simulation using digital twins, and provision of simulation results.

[1503] Examples of prompts include:

[1504] "I want to simulate an autonomous vehicle in an urban environment on a sunny day."

[1505] "I want to simulate an autonomous vehicle in a suburban environment on a rainy day."

[1506] In this way, users can simulate various scenarios and see how the autonomous vehicle will behave.

[1507] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1508] Step 1:

[1509] The user inputs natural language or video using a device such as a smartphone. The input information is sent from the device to the server. The input data is a prompt sentence, for example, "I would like to simulate an autonomous vehicle in an urban environment on a sunny day."

[1510] Step 2:

[1511] The server passes the received natural language and video information to the generation AI. The generation AI analyzes the input data and generates virtual cars, people, and environments. Specifically, it uses the OpenAI API to analyze the input prompt sentences and generate virtual elements. The output is data on the generated virtual cars, people, and environments.

[1512] Step 3:

[1513] The server inputs the generated virtual elements into a digital twin. A digital twin is a digital model that recreates physical objects and environments in a virtual space, and performs a simulation using the generated virtual elements. The input is the data of the generated virtual elements, and the output is the simulation results.

[1514] Step 4:

[1515] The server analyzes the results of the simulations performed by the digital twin and generates data to provide to users. Specifically, it organizes the simulation results and converts them into a format that is easy for users to understand. The input is the simulation result data, and the output is the organized data to provide to users.

[1516] Step 5:

[1517] The server sends the organized simulation results to the user's device, where the user can check the simulation results and evaluate the behavior of the autonomous vehicle. The input is the organized simulation result data, and the output is the simulation results displayed on the user's device.

[1518] In this way, users can simulate various scenarios and see how the autonomous vehicle will behave.

[1519] Example 2

[1520] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1521] When testing autonomous driving AI, it takes a lot of time and money to recreate real-world road conditions. Furthermore, it is difficult to completely recreate real-world road conditions, which often limits the scope of testing. As a result, there is a problem in that it is not possible to adequately prepare countermeasures for the diverse scenarios that autonomous driving AI may encounter in a real driving environment.

[1522] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1523] In this invention, the server includes means for generating virtual vehicles, people, and environments from natural language, video, etc. using a generative AI model and inputting them into a digital twin, means for implementing highly realistic and diverse simulations using the digital twin, means for extensively and thoroughly testing the autonomous driving AI using the simulations, means for generating scenarios by inputting prompt sentences, means for saving the generated scenarios, means for testing the autonomous driving AI using the saved scenarios, and means for recording the test results. This allows the autonomous driving AI to be tested so that it can handle a variety of scenarios, and allows for improvements and adjustments based on the test results.

[1524] A "generative AI model" is an artificial intelligence technology that generates virtual vehicles, people, and environments from input data such as natural language and video.

[1525] A "digital twin" is a digital model that recreates a real-world physical system or environment in a virtual space.

[1526] "Simulation" is the process of using a digital twin to virtually recreate and test real-world road conditions and driving scenarios.

[1527] "Autonomous driving AI" is an artificial intelligence technology used to control the driving of self-driving cars.

[1528] A "prompt" is an instruction entered into a generative AI model to generate a specific scenario.

[1529] A "scenario" is a hypothetical situational setting that describes a specific driving situation or environment that an autonomous vehicle might encounter.

[1530] "Saving" is the act of recording the generated scenario in a database or file system.

[1531] "Test results" are data on the behavior and performance of autonomous driving AI obtained when running a simulation.

[1532] This invention is a system that generates scenarios for testing autonomous driving AI using a generative AI model and performs simulations using a digital twin. Specific embodiments of this system are described below.

[1533] System configuration

[1534] Hardware

[1535] Server: Use a server with high-performance computing capabilities. Specifically, a server equipped with a GPU is preferable.

[1536] Device: The user uses a device such as a computer or tablet to enter the prompt.

[1537] software

[1538] Generative AI model: An artificial intelligence technology for generating virtual vehicles, people, and environments from input data such as natural language and video. Specifically, it uses machine learning libraries such as TensorFlow and PyTorch.

[1539] Digital twin: Software for recreating real-world physical systems and environments in virtual space. Game engines such as Unity and Unreal Engine can be used.

[1540] Database: A database for saving the generated scenarios. Specifically, a database management system such as MySQL or MongoDB is used.

[1541] System Operation

[1542] Entering a prompt statement

[1543] The user inputs a prompt sentence to the generative AI model using the terminal. For example, the user inputs the prompt sentence "Please generate a right turn scenario at an intersection."

[1544] Scenario Generation Execution

[1545] The server passes the received prompt sentence to the generative AI model, which then generates a detailed scenario based on the prompt sentence. For example, the generated scenario may include the shape of the intersection, the status of traffic lights, and the movements of other vehicles and pedestrians.

[1546] Saving a Scenario

[1547] The server saves the generated scenarios in JSON format, which are later used to test the autonomous driving AI.

[1548] Test run of the scenario

[1549] The server runs tests on the self-driving AI using the saved scenarios, and the test results are logged and later analyzed.

[1550] Specific examples

[1551] As a concrete example, the following scenario is generated:

[1552] A scenario in which an autonomous vehicle is turning right at an intersection and an oncoming vehicle is coming straight ahead.

[1553] Scenario where a pedestrian suddenly jumps out onto the crosswalk

[1554] Scenarios where other vehicles suddenly change lanes

[1555] Prompt Sentence Examples

[1556] "Generate a right turn scenario at an intersection"

[1557] "Generate a scenario where a pedestrian suddenly jumps out."

[1558] "Generate a scenario where another vehicle suddenly changes lanes."

[1559] This system allows the self-driving AI to be tested to handle a variety of scenarios, allowing it to be improved and adjusted based on the results.

[1560] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1561] Step 1:

[1562] The user inputs a prompt sentence to the generative AI model using a terminal. For example, the user inputs a prompt sentence such as "Please generate a right turn scenario at an intersection."

[1563] Input: prompt statement

[1564] Output: The prompt is sent to the server.

[1565] Specific operation: The user uses the terminal keyboard to enter a prompt sentence and clicks the send button.

[1566] Step 2:

[1567] The server passes the received prompt sentence to the generative AI model, which then generates a detailed scenario based on the prompt sentence.

[1568] Input: prompt statement

[1569] Output: Generated scenario

[1570] Specific operation: The server inputs the prompt sentence into the generative AI model, which then generates a scenario including details such as the shape of the intersection, the status of traffic lights, and the movement of other vehicles and pedestrians.

[1571] Step 3:

[1572] The server saves the generated scenarios in JSON format, which are later used to test the autonomous driving AI.

[1573] Input: Generated scenario

[1574] Output: JSON format scenario file

[1575] Specific operation: The server converts the generated scenario into JSON format and saves it in a database or file system.

[1576] Step 4:

[1577] The server runs tests on the self-driving AI using the saved scenarios, and the test results are logged and later analyzed.

[1578] Input: JSON format scenario file

[1579] Output: Test result log file

[1580] Specific operation: The server loads the saved scenario, executes the scenario for the autonomous driving AI, and records the test results in a log file.

[1581] (Application example 2)

[1582] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1583] As autonomous driving technology advances, there is an increasing need to test the performance of autonomous driving AI extensively and thoroughly. However, testing in real-world road environments is time-consuming and costly, and ensuring safety is also a challenge. Furthermore, there is a lack of a means to efficiently visualize generated scenarios and propose improvements to autonomous driving AI based on the results. To solve these challenges, a more efficient and comprehensive test system is needed.

[1584] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1585] In this invention, the server includes means for generating virtual cars, people, and environments from natural language, videos, etc. using a generative AI and inputting them into a digital twin, means for implementing highly realistic and diverse simulations using the digital twin, means for extensively and thoroughly testing the autonomous driving AI using the simulations, means for simulating the generated scenarios on a smartphone and visualizing the results, and means for proposing improvements to the autonomous driving AI based on the results of the scenarios, thereby enabling efficient and comprehensive testing of the performance of the autonomous driving AI and rapid identification of areas for improvement.

[1586] "Generative AI" is an artificial intelligence technology that generates virtual cars, people, and environments from natural language, videos, etc.

[1587] A "digital twin" is a digital model that recreates a real-world physical object or environment in a virtual space.

[1588] "Simulation" is a method of using digital twins to virtually recreate highly realistic and diverse situations and conduct experiments and tests.

[1589] "Autonomous driving AI" is an artificial intelligence technology that enables cars to drive autonomously.

[1590] A "smartphone" is a mobile information terminal that has the functions of a computer in addition to the functions of a mobile phone.

[1591] "Visualization" is a technique for visually displaying data and simulation results to make them easier to understand.

[1592] "Areas for improvement" are corrections and refinements that are identified based on the simulation results to improve the performance of the autonomous driving AI.

[1593] The system for implementing this invention includes elements of generative AI, digital twin, simulation, autonomous driving AI, smartphone, visualization, and improvement. Specific embodiments are described below.

[1594] The server uses generative AI to generate virtual cars, people, and environments from natural language and video. The generated virtual cars, people, and environments are then input into a digital twin, a digital model that recreates physical objects and environments in the real world in a virtual space.

[1595] The server then uses the digital twin to perform a variety of highly realistic simulations, a method of experimenting and testing in virtually recreated situations, used to extensively and thoroughly test the performance of autonomous driving AI.

[1596] The generated scenario is simulated on a smartphone, and the results are visualized. A smartphone is a mobile information terminal that has the functionality of a computer in addition to a mobile phone. Visualization is a technique for visually displaying data and simulation results to make them easier to understand.

[1597] Users can identify and propose improvements to the autonomous driving AI based on the simulation results. Improvements are corrections and improvements that can be made to improve the performance of the autonomous driving AI, as identified based on the simulation results.

[1598] For example, you can generate a scenario by inputting the following prompt into the generation AI:

[1599] "Generate a scenario in which an autonomous vehicle makes a right turn at an intersection."

[1600] "Generate a lane change scenario for an autonomous vehicle on a highway."

[1601] "Generate a scenario in which an autonomous vehicle drives through a residential area at night."

[1602] "Generate a scenario in which an autonomous vehicle drives through urban areas in the rain."

[1603] This allows users to test autonomous driving AI against a variety of scenarios and evaluate its performance efficiently and comprehensively.

[1604] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1605] Step 1:

[1606] The server uses generative AI to generate virtual cars, people, and environments from natural language, videos, etc.

[1607] Input: natural language and video data

[1608] Data processing: Generative AI models analyze input data and generate virtual cars, people, and environments.

[1609] Output: Digital data of virtual vehicles, humans, and environments

[1610] Step 2:

[1611] The server then inputs the generated virtual vehicles, people, and environments into the digital twin.

[1612] Input: Digital data of virtual vehicles, humans, and environments

[1613] Data processing: Integrating virtual vehicles, humans and environments into the digital twin.

[1614] Output: Virtual environment integrated into a digital twin

[1615] Step 3:

[1616] The server uses the digital twin to perform highly realistic and diverse simulations.

[1617] Input: Virtual environment integrated into the digital twin

[1618] Data calculation: The simulation engine calculates the behavior in the virtual environment and executes the scenario.

[1619] Output: Simulation result data

[1620] Step 4:

[1621] The server simulates the generated scenario on a smartphone and visualizes the results.

[1622] Input: Simulation result data

[1623] Data processing: Generate graphs and charts to visually display the simulation results.

[1624] Output: Visualized data displayed on a smartphone

[1625] Step 5:

[1626] Users will identify and propose improvements to the autonomous driving AI based on the simulation results.

[1627] Input: Visualized data displayed on a smartphone

[1628] Data Computation: Users analyze simulation results and identify areas for improvement.

[1629] Output: Proposal data for improvements to the autonomous driving AI

[1630] This allows users to test autonomous driving AI against a variety of scenarios and evaluate its performance efficiently and comprehensively.

[1631] Example 3

[1632] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1633] In the development of autonomous driving AI, testing in real-world environments is time-consuming and costly, making it difficult to conduct efficient testing. Furthermore, it is difficult to repeatedly test reproducible scenarios in real-world environments, making it difficult to adequately prepare for various scenarios.

[1634] The identification processing by the identification processing unit 290 of the data processing device 12 in Example 3 is realized by the following means. In this invention, the server includes: means for generating virtual objects, people, and environments from natural language, videos, etc. using a generative AI and inputting them into a digital twin; means for performing highly realistic and diverse simulations using the digital twin; means for extensively and thoroughly testing autonomous driving AI using the simulations; means for a user to set a test scenario using a terminal; means for the server to generate a simulation environment using a generative AI model; means for the server to execute a simulation on the digital twin platform; means for the server to analyze the test results; and means for the user to modify the scenario and retest. This significantly reduces the time and cost involved in testing and enables repeated testing of reproducible scenarios.

[1635] "Generative AI" is an artificial intelligence technology that generates virtual objects, people, and environments from input data such as natural language, images, and videos.

[1636] "Digital twin" is a technology that recreates real-world objects and environments in a virtual space and performs simulations and analyses.

[1637] A "simulation environment" is a virtual space for testing and analysis that includes virtual objects, people, and environments generated by generative AI.

[1638] "Autonomous driving AI" is an artificial intelligence technology for automating automobile driving.

[1639] A "terminal" is a device such as a computer or smartphone that is operated by a user.

[1640] A "server" is a high-performance computer system that runs generative AI models, manages simulations, and analyzes data.

[1641] A "test scenario" is a scenario that includes specific conditions or situations set up to evaluate the performance of autonomous driving AI.

[1642] A "generative AI model" is a specific implementation of generative AI, an algorithm or program for generating virtual objects, people, and environments from input data such as natural language, images, and videos.

[1643] A "prompt" is an instruction entered into a generative AI model to cause it to generate a specific output.

[1644] A "digital twin platform" is software and systems that realize digital twin technology.

[1645] MODE FOR CARRYING OUT THE INVENTION

[1646] This invention is a system that significantly reduces the time and cost involved in testing autonomous driving AI through simulations using generative AI and digital twins. This system involves a series of processes: a user sets up a test scenario using a terminal, a server generates a simulation environment using a generative AI model, runs the simulation on a digital twin platform, and analyzes the test results.

[1647] Hardware and software used

[1648] Hardware: High-performance servers (e.g., high-performance computer systems)

[1649] Software: Generative AI models (e.g., advanced generative AI algorithms), digital twin platforms (e.g., simulation software)

[1650] Data processing and calculation flow

[1651] 1. The user sets up a specific test scenario using the device. For example, the user selects "changing lanes on a highway in the rain" as the scenario. Detailed parameters such as traffic volume and weather conditions are entered through the device interface.

[1652] 2. Based on the scenario settings received from the user, the server uses a generative AI model to generate a detailed simulation environment. The generative AI model then builds a realistic simulation environment based on the input conditions.

[1653] 3. The server imports the generated simulation environment into the digital twin platform and runs autonomous driving AI tests. The simulation is performed in real time, and vehicle behavior and changes in the environment are recorded in detail.

[1654] 4. The server analyzes the simulation results and evaluates the performance and problems of the autonomous driving AI. The analysis results are provided to the user in the form of graphs and reports.

[1655] 5. The user modifies the scenario as needed based on the analysis results. After modification, the user runs the simulation again and checks the results. By repeating this process, the performance of the autonomous driving AI is optimized.

[1656] Specific examples

[1657] For example, if a user sets "changing lanes on a highway in the rain" as a test scenario, they would input the following prompt sentence into the generative AI model:

[1658] Example prompt sentence:

[1659] "Generate a lane change scenario on a highway in the rain. Specifically, simulate a lane change in heavy traffic and record the vehicle's behavior in detail."

[1660] When this prompt is entered, the generative AI model creates a detailed simulation environment and runs autonomous driving AI tests on the digital twin platform. The test results are analyzed by the server and fed back to the user.

[1661] The above is an embodiment of the present invention. This system significantly reduces the time and cost involved in testing, and enables repeatable testing of scenarios. The flow of the identification process in the third embodiment will be described with reference to FIG. 15.

[1662] Step 1:

[1663] The user sets up a test scenario using the device. Through the device interface, the user inputs detailed parameters such as specific traffic and weather conditions. For example, the user selects "changing lanes on a highway in rainy weather" as a scenario. The input data is then sent to the server.

[1664] Input: Detailed parameters of the test scenario (e.g., rain, highway, lane change)

[1665] Output: Scenario configuration data sent to the server

[1666] Specific behavior:

[1667] The user operates the GUI on the terminal to open the scenario setting screen.

[1668] The user inputs conditions such as "rainy weather," "highway," and "lane change."

[1669] Step 2:

[1670] Based on the scenario settings received from the user, the server uses a generative AI model to generate a detailed simulation environment. The generative AI model builds a realistic simulation environment based on the input conditions. A prompt statement is input into the generative AI model to generate the simulation environment.

[1671] Input: Scenario setting data, prompt text

[1672] Output: Generated simulation environment data

[1673] Specific behavior:

[1674] The server receives input data from the user.

[1675] The server inputs prompt statements into the generated AI model and generates a simulation environment.

[1676] Step 3:

[1677] The server then imports the generated simulation environment into the digital twin platform and runs autonomous driving AI tests. The simulation is performed in real time, and vehicle behavior and changes in the environment are recorded in detail.

[1678] Input: Generated simulation environment data

[1679] Output: Simulation execution result data

[1680] Specific behavior:

[1681] The server imports the generated simulation environment into the digital twin platform.

[1682] The server runs the autonomous driving AI in a simulated environment and collects data.

[1683] Step 4:

[1684] The server analyzes the simulation results and evaluates the performance and problems of the autonomous driving AI. The analysis results are provided to the user in the form of graphs and reports.

[1685] Input: Simulation execution result data

[1686] Output: Analysis result data (graphs, reports)

[1687] Specific behavior:

[1688] The server runs algorithms that analyze the simulation data.

[1689] The server visualizes the analysis results and generates a report.

[1690] Step 5:

[1691] The user modifies the scenario as necessary based on the analysis results. After modification, the user runs the simulation again and checks the results. By repeating this process, the performance of the autonomous driving AI is optimized.

[1692] Input: Analysis result data, modified scenario setting data

[1693] Output: New simulation run results data

[1694] Specific behavior:

[1695] The user uses the terminal to check the analysis results.

[1696] The user corrects the scenario and runs the test again.

[1697] (Application example 3)

[1698] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1699] In the development of autonomous driving AI, testing in real-world environments is time-consuming, costly, and difficult to ensure safety. Furthermore, extensive and thorough testing is required to adequately address a wide range of scenarios, which is also difficult to achieve in real-world environments. To address these challenges, a means is needed to efficiently and safely test autonomous driving AI.

[1700] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1701] In this invention, the server includes means for generating virtual cars, people, and environments from natural language, video, etc. using a generative AI and inputting them into a digital twin, means for implementing highly realistic and diverse simulations using the digital twin, means for extensively and thoroughly testing autonomous driving AI using the simulations, means for running simulation tests on a smartphone, and means for evaluating the simulation results and calculating safety and efficiency scores. This makes it possible to efficiently and safely test autonomous driving AI and quickly take measures for a variety of scenarios.

[1702] "Generative AI" is an artificial intelligence technology that generates virtual cars, people, and environments from data such as natural language and video.

[1703] "Digital twin" is a technology that recreates real-world physical environments and objects in virtual space.

[1704] "Simulation" is a method of recreating real-world environments and scenarios in a virtual space for testing and evaluation.

[1705] "Autonomous driving AI" is an artificial intelligence technology that enables autonomous driving of vehicles.

[1706] A "smartphone" is a mobile device that has the functions of a computer in addition to the functions of a mobile phone.

[1707] A "simulation test" is a test conducted using a simulation in a virtual space.

[1708] "Safety" is the ability of a system or technology to operate without accidents or failures.

[1709] "Efficiency" is the ability of a system or technology to achieve maximum results with minimum resources.

[1710] The "score" is a numerical representation of the evaluation result.

[1711] The system for implementing this invention utilizes generative AI and digital twin technology to perform simulation tests of autonomous driving AI. Specific embodiments are described below.

[1712] First, the server uses generative AI to generate virtual cars, people, and environments from data such as natural language and video. This generated data is then input into a digital twin, which recreates real-world physical environments and objects in a virtual space.

[1713] The digital twin is then subjected to a variety of highly realistic simulations, allowing the autonomous driving AI to be tested extensively and thoroughly. The simulations run on a smartphone, allowing users to set up various scenarios and repeatedly test them.

[1714] The simulation results are evaluated by the server and safety and efficiency scores are calculated, allowing users to evaluate the performance of their autonomous driving AI and make any necessary improvements.

[1715] Hardware and software used:

[1716] Hardware: Smartphone (iOS or Android)

[1717] Software: Python, TensorFlow, Digital Twin Library

[1718] Data processing and calculation:

[1719] The server uses generative AI to generate virtual vehicles, people, and environments from natural language and video. The generated data is input into a digital twin, where a simulation is conducted in the virtual space. The simulation results are evaluated by the server, and a safety and efficiency score is calculated.

[1720] Examples:

[1721] For example, to test an urban driving scenario, the following prompt sentences are input to the generative AI model:

[1722] Example prompt sentence:

[1723] "Simulate urban driving scenarios. The scenarios include traffic lights, pedestrians, and other vehicles. The AI ​​model needs to drive safely and efficiently."

[1724] A user opens the AutoDrive Sim app on their smartphone and selects an urban driving scenario. The app loads the scenario and starts the simulation. When the simulation is finished, a safety and efficiency score is displayed, allowing the user to evaluate the AI ​​model's performance.

[1725] In this way, it becomes possible to efficiently and safely test autonomous driving AI and quickly develop countermeasures for a variety of scenarios.

[1726] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1727] Step 1:

[1728] The server uses generative AI to generate virtual cars, people, and environments from natural language and video.

[1729] Input: Natural language and video data

[1730] Data processing: Generative AI models are used to generate virtual vehicles, people, and environments from input data.

[1731] Output: Virtual vehicle, human, and environment data

[1732] Step 2:

[1733] The server inputs data about the generated virtual cars, people, and environment into the digital twin.

[1734] Input: Virtual vehicle, human, and environment data

[1735] Data processing: Input data into the digital twin and recreate it in virtual space.

[1736] Output: Virtual environment in the digital twin

[1737] Step 3:

[1738] The server uses the digital twin to perform highly realistic and diverse simulations.

[1739] Input: Virtual environment in the digital twin

[1740] Data processing: Using a simulation engine to run scenarios in a virtual environment.

[1741] Output: Simulation result data

[1742] Step 4:

[1743] The terminal (smartphone) executes a simulation test.

[1744] Input: Simulation scenario data

[1745] Data processing: Run simulations on your smartphone and collect the results.

[1746] Output: Simulation result data

[1747] Step 5:

[1748] The server evaluates the simulation results and calculates a safety and efficiency score.

[1749] Input: Simulation result data

[1750] Data processing: Analyze the results data and calculate safety and efficiency scores.

[1751] Output: Safety score, efficiency score

[1752] Step 6:

[1753] Users can check the simulation results on their smartphones and evaluate the performance of the AI ​​model.

[1754] Input: Safety score, Efficiency score

[1755] Data processing: Display scores through a smartphone interface.

[1756] Output: User's evaluation result

[1757] The above is the flow of processing of the program of the system that realizes the application example.

[1758] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1759] "Example 1"

[1760] In one embodiment of the present invention, the generation AI uses an emotion engine that recognizes the user's emotions to generate virtual cars, people, and environments that correspond to the user's emotions. Specifically, the emotion engine analyzes the user's emotions from their facial expressions, tone of voice, and choice of words, and transmits the results to the generation AI. Based on the analysis results, the generation AI generates, for example, a virtual environment with bright weather if the user is happy, or a virtual environment with rain if the user is sad.

[1761] "Example 2"

[1762] The emotion engine also adjusts the simulation scenario according to the user's emotions. For example, if the user is angry, the emotion engine communicates this information to the digital twin, and the digital twin prioritizes simulating scenarios that are likely to anger the user, such as traffic congestion or rude driver behavior.

[1763] "Example 3"

[1764] Furthermore, the emotion engine adjusts the testing priorities of the autonomous driving AI based on the user's emotions. For example, if the user is feeling scared, the emotion engine will convey that information to the autonomous driving AI, and the autonomous driving AI will prioritize testing responses to accidents and dangerous situations. This enables more effective testing based on the user's emotions.

[1765] The processing flow of each embodiment will be described below.

[1766] "Example 1"

[1767] Step 1: The emotion engine analyzes the user's emotions based on their facial expressions, tone of voice, and choice of words.

[1768] Step 2: The emotion engine communicates the analysis results to the generative AI.

[1769] Step 3: Based on the analysis results, the AI ​​generates virtual cars, people, and environments according to the user's emotions. For example, if the user is happy, it generates a virtual environment with bright weather, and if the user is sad, it generates a virtual environment with rain.

[1770] "Example 2"

[1771] Step 1: The emotion engine analyzes the user's emotions.

[1772] Step 2: The emotion engine communicates the analysis results to the digital twin.

[1773] Step 3: The digital twin adjusts the simulation scenarios according to the user's emotions. For example, if the user is angry, it will prioritize scenarios that are likely to anger the user, such as traffic congestion or rude driver behavior.

[1774] "Example 3"

[1775] Step 1: The emotion engine analyzes the user's emotions.

[1776] Step 2: The emotion engine communicates the analysis results to the autonomous driving AI.

[1777] Step 3: The self-driving AI adjusts test priorities based on the user's emotions. For example, if the user is feeling scared, it will prioritize testing responses to accidents and dangerous situations.

[1778] Example 1

[1779] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1780] In the development of autonomous driving technology, testing in real road environments is time-consuming and costly, and it is difficult to cover all scenarios. Furthermore, it is not possible to provide a simulation environment that responds to the user's emotions, making it difficult to improve the user experience.

[1781] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1782] In this invention, the server includes: a means for generating virtual cars, people, and environments from natural language, video, etc. using a generative AI and inputting them into a digital twin; a means for conducting highly realistic and diverse simulations using the digital twin; a means for extensively and thoroughly testing autonomous driving AI using the simulation; a means for generating virtual cars, people, and environments based on a user's emotions using an emotion engine that recognizes the user's emotions; a means for the emotion engine to analyze the user's emotions from their facial expressions, tone of voice, choice of words, etc. and communicate the results to the generative AI; and a means for the generative AI to generate a virtual environment based on the analysis results. This reduces the time and cost required for testing in a real road environment and enables all scenarios to be covered. Furthermore, providing a simulation environment that responds to the user's emotions improves the user experience.

[1783] "Generative AI" is an artificial intelligence technology that analyzes information such as natural language and videos to generate virtual cars, people, and environments.

[1784] "Digital twin" is a technology that recreates physical objects and environments in a virtual space and performs simulations in real time.

[1785] "Simulation" is the process of replicating the behavior of autonomous vehicles and other elements in a virtual environment for testing and evaluation.

[1786] "Autonomous driving AI" is an artificial intelligence technology used to control and make decisions about autonomous vehicles.

[1787] An "emotion engine" is a technology that analyzes emotions from a user's facial expressions, tone of voice, choice of words, etc., and transmits the results to other systems.

[1788] "User emotions" refers to the psychological state that a user expresses through facial expressions, tone of voice, choice of words, etc.

[1789] A "virtual environment" is a simulated setting that includes elements such as cars, people, and the environment in a digital space generated by generative AI.

[1790] MODE FOR CARRYING OUT THE INVENTION

[1791] This invention is a system that uses a generative AI model to analyze information such as natural language and video to generate virtual cars, people, and environments. The generated virtual environment is input into a digital twin, where realistic and diverse simulations are performed. Furthermore, an emotion engine that recognizes the user's emotions can be used to generate a virtual environment that corresponds to the user's emotions.

[1792] Hardware and software used

[1793] The server uses Google Cloud's natural language processing API and OpenAI's GPT-4 as generative AI models. Game engines such as Unity and Unreal Engine are used to implement the digital twin. The emotion engine includes software for analyzing the user's facial expressions, tone of voice, and word choice.

[1794] Data processing and calculation

[1795] The server receives natural language and video input from the user and analyzes it using a generative AI model. Based on the analysis results, virtual cars, people, and environments are generated. The generated data is input into a digital twin, where a simulation is carried out in real time. The emotion engine analyzes the user's emotions and conveys the results to the generative AI. Based on the analysis results, the generative AI generates a virtual environment that corresponds to the user's emotions.

[1796] Specific examples

[1797] If a user inputs into the system, "I would like to simulate autonomous driving in an urban environment on a sunny day," the server will analyze this natural language and generate an urban environment on a sunny day. The generated environment is then input into the digital twin, and the simulation begins.

[1798] Furthermore, when using the emotion engine, emotions are analyzed from the user's facial expressions, tone of voice, choice of words, etc. For example, if a user says, "I'm in a good mood today," the emotion engine will interpret this as the user being happy, and the generation AI will generate a virtual environment with bright weather.

[1799] Prompt Sentence Examples

[1800] "I want to simulate autonomous driving in an urban environment on a sunny day."

[1801] "I want to simulate autonomous driving in a suburban environment on a rainy day."

[1802] "I'm feeling good today, so I want to see a bright weather simulation."

[1803] In this way, by having the server, terminal, and user work together to execute system processing, it is possible to reduce the time and cost required for testing in a real road environment and cover all scenarios.In addition, by providing a simulation environment that responds to the user's emotions, it is possible to improve the user experience.

[1804] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1805] Step 1:

[1806] The user provides input

[1807] The user inputs text into the system interface, such as "I would like to simulate autonomous driving in an urban environment on a sunny day," or speaks a similar message using voice input. The input data can be text data or voice data.

[1808] Step 2:

[1809] The server receives the input

[1810] The server receives text and voice input from the user. In the case of voice input, it first converts it into text using a speech recognition API. The input data is saved as text data.

[1811] Step 3:

[1812] The server parses the input

[1813] The server then sends the received text data to Google Cloud's natural language processing API and obtains the analysis results. The analysis uses natural language processing technology to understand the user's request. The analysis results include information about the simulation environment the user desires.

[1814] Step 4:

[1815] The server creates a virtual environment

[1816] The server uses Unity to generate a sunny urban environment based on the analysis results. Specifically, it creates a 3D model of the city and sets the weather conditions. The generated virtual environment data is then input into the digital twin.

[1817] Step 5:

[1818] The server populates the digital twin with a virtual environment

[1819] The server sends the generated urban environment data to the digital twin platform, which prepares the simulation in real time.Digital twin is a platform for simulating virtual environments in real time.

[1820] Step 6:

[1821] The server runs the simulation

[1822] The server then runs a simulation of the autonomous vehicle on the digital twin, simulating, for example, how the vehicle behaves when passing through an intersection and interacts with other vehicles and pedestrians. Simulation results are generated in real time.

[1823] Step 7:

[1824] The terminal displays the simulation results.

[1825] The device displays the simulation results sent from the server in real time, allowing the user to see how the autonomous vehicle navigates through an urban environment on the device screen. The displayed data includes the progress and results of the simulation.

[1826] In this way, by having the server, terminal, and user work together to execute system processing, it is possible to reduce the time and cost required for testing in a real road environment and cover all scenarios.In addition, by providing a simulation environment that responds to the user's emotions, it is possible to improve the user experience.

[1827] (Application example 1)

[1828] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1829] In the development of autonomous vehicles, testing in real road environments is not only time-consuming and costly, but can also be dangerous. Furthermore, it is difficult to reproduce various scenarios and weather conditions, limiting the scope of testing. Furthermore, it is not possible to provide a real-time simulation environment that responds to user emotions, so there is a need to improve the user experience.

[1830] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1831] In this invention, the server includes: means for generating virtual cars, people, and environments from natural language, video, etc. using a generative AI and inputting them into a digital twin; means for implementing highly realistic and diverse simulations using the digital twin; means for extensively and thoroughly testing autonomous driving AI using the simulations; means for analyzing user emotions using an emotion engine and generating a virtual environment based on the analysis results; and means for displaying the simulation results in real time using a smartphone or head-mounted display and allowing the user to operate an autonomous driving vehicle in the virtual environment. This makes it possible to reproduce various scenarios and weather conditions and provide a real-time simulation environment that responds to the user's emotions.

[1832] "Generative AI" is an artificial intelligence that analyzes information such as natural language and videos, and generates virtual cars, people, and environments.

[1833] "Digital twin" is a technology that recreates and simulates physical objects and environments in a virtual space.

[1834] The "emotion engine" is a system that analyzes emotions from the user's facial expressions, tone of voice, choice of words, etc.

[1835] The "virtual environment" is a simulated environment within the digital twin generated by generative AI.

[1836] "Autonomous driving AI" is artificial intelligence used to control the operation of autonomous vehicles.

[1837] A "smartphone" is a portable information terminal that, in addition to the functions of a mobile phone, can also connect to the Internet and run applications.

[1838] A "head-mounted display" is a device worn on the user's head that displays images in the user's field of vision.

[1839] "Simulation" is the process of recreating and testing the behavior of autonomous vehicles in a virtual environment.

[1840] "Real-time" refers to processing and display occurring immediately without delay.

[1841] "User" refers to a person who uses the system.

[1842] The system for implementing this invention uses generative AI, a digital twin, an emotion engine, a smartphone, and a head-mounted display (HMD). The specific configuration and operation of the system are described below.

[1843] System configuration

[1844] 1. Generative AI: Analyzes information such as natural language and video to generate virtual cars, people, and environments.

[1845] 2. Digital Twin: Reproducing and simulating real-world physical objects and environments in a virtual space.

[1846] 3. Emotion engine: Analyzes emotions from the user's facial expressions, tone of voice, choice of words, etc.

[1847] 4. Smartphone: A mobile information terminal that has the functionality of a mobile phone and can also connect to the Internet and run applications.

[1848] 5. Head-mounted display (HMD): A device worn on the user's head that displays images in the user's field of vision.

[1849] System Operation

[1850] 1. Data entry and analysis:

[1851] The server receives natural language and video input from the user.

[1852] Generative AI analyzes this data and generates virtual cars, people, and environments.

[1853] 2. Emotion analysis:

[1854] The server uses an emotion engine to analyze the user's facial expressions and tone of voice.

[1855] The emotion engine communicates the analysis results to the generative AI.

[1856] 3. Create a virtual environment:

[1857] The generative AI generates a virtual environment based on the analysis results from the emotion engine.

[1858] For example, if the user is happy, a bright virtual environment is generated, and if the user is sad, a rainy virtual environment is generated.

[1859] 4. Run the simulation:

[1860] The server inputs the generated virtual environment into the digital twin and performs a simulation.

[1861] The digital twin simulates the behavior of an autonomous vehicle in a generated virtual environment in real time.

[1862] 5. Viewing and manipulating results:

[1863] Using a smartphone or HMD, users can check the simulation results in real time.

[1864] Users can operate autonomous vehicles in a virtual environment and test them in various scenarios.

[1865] Specific examples

[1866] When a user simulates an autonomous vehicle using their smartphone, a camera captures their facial expressions and an emotion engine detects "happiness." The generative AI generates a sunny urban environment and runs the simulation in the digital twin. The user can view the simulation results in real time on their smartphone screen and control the vehicle's behavior.

[1867] Prompt Sentence Examples

[1868] "If the user is happy, generate an urban environment with sunny weather and run a simulation of an autonomous vehicle."

[1869] In this way, the embodiment of the invention can reproduce various scenarios and weather conditions and provide a real-time simulation environment that responds to the user's emotions.

[1870] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1871] Step 1:

[1872] The server receives natural language and video input from the user. The input data is information about the scenario and environment the user wants to use in the simulation. The server then passes this data to the generation AI.

[1873] Step 2:

[1874] The generative AI analyzes the natural language and video it receives to generate virtual cars, people, and environments. Specifically, it uses natural language processing technology to analyze text data and video analysis technology to analyze video data. The generated virtual cars, people, and environments are then input into the digital twin.

[1875] Step 3:

[1876] The server uses an emotion engine to analyze the user's facial expressions and tone of voice. The input data is the user's facial image and voice data. The emotion engine analyzes this data and identifies the user's emotional state. The analysis results are transmitted to the generation AI.

[1877] Step 4:

[1878] The generative AI generates a virtual environment based on the analysis results from the emotion engine. For example, if the user is happy, it generates a virtual environment with bright weather, and if they are sad, it generates a virtual environment with rain. The generated virtual environment is then input into the digital twin.

[1879] Step 5: ...

Claims

[Claim 1] a means for the user to input a prompt sentence; means for transmitting the input data of the prompt sentence to a server by the terminal; means for the server to analyze the prompt data using a generative AI model; a means for generating virtual objects including virtual vehicles, humans, and environments based on the analysis results; A means for inputting the virtual object generated by the server into a digital twin; means for recognizing the user's emotions by analyzing the user's facial expressions, tone of voice, or choice of words; A means for the digital twin to preferentially simulate a scenario corresponding to the emotion from among various scenarios that reproduce the real world using the virtual car, human, and environment; means for the server to transmit a simulation result to the terminal; means for displaying the simulation results to the user in the terminal; A system including:

Images