VILS-based test system and method for autonomous vehicle considering interaction between virtual environment and actual environment

Abstract

The present invention relates to a technology related to a VILS-based test for an autonomous vehicle capable of performing a VIL test in an actual road environment by fusing actual surrounding vehicle information and surrounding vehicle information generated in a virtual environment. This VILS-based test system for an autonomous vehicle according to an embodiment may comprise: a sensor-fused precise positioning unit for estimating the current location of the vehicle on the basis of information collected from a GPS signal reception device, an in-vehicle sensor, and a LiDAR sensor; and an integrated processing unit for integrating real-world data collected from the sensors of the autonomous vehicle into a virtual environment of an autonomous driving simulator by applying an intelligent driver model (IDM) that approximates a driver's behavior and integrates responses.

Description

A VILS-based testing system and method for autonomous vehicles considering the interaction between virtual and real environments.

[0001] The present invention relates to a technology related to a VILS-based test of an autonomous vehicle that enables VIL testing in an actual road environment by fusing actual surrounding vehicle information and surrounding vehicle information generated in a virtual environment.

[0002] Autonomous driving technology requires real-world vehicle testing to address a variety of scenarios that could arise on public roads. However, setting up a test environment is time-consuming, and the potential for injury to occupants during crash tests poses limitations to real-world vehicle verification. VILS (Virtual Integral Testing) combines a real-world test vehicle with a virtual external environment, enabling testing of a wide range of scenarios with actual vehicles.

[0003] However, since it does not take into account the actual surrounding environment, it can only be implemented in limited places such as a 6-axis simulator or a driving test site.

[0004] Specifically, software verification is a crucial step in the development of modern autonomous vehicles. According to the RAND Corporation, commercialization of fully autonomous vehicles requires at least 100 vehicles and approximately 440 million kilometers of driving tests. However, the recent COVID-19 pandemic has significantly limited real-world driving tests by autonomous driving companies, resulting in a significant reduction in driving distances.

[0005] To effectively respond to the diverse situations that can arise during autonomous driving, the best solution is to develop a variety of scenarios that could occur on actual roads and test them on actual vehicles. However, actual vehicle testing requires considerable time to set up the experimental environment, and crash tests pose a risk of injury to occupants due to the impact they inflict.

[0006] There are various methods for safely testing by simulating complex real-world driving environments. In particular, validation methods utilizing vehicle simulation software and Hardware In-the-Loop Simulation (HILS) equipment, which can create virtual environments to replicate actual vehicle testing, are widely used. While these methods offer the advantage of replacing real-world vehicle testing, they struggle to perfectly reflect the dynamic characteristics of an actual vehicle. To address this issue, Vehicle In-the-Loop Simulation (VILS) testing, which integrates real-world test vehicles with virtual external environments, is an alternative.

[0007] VILS is a method that effectively reproduces more diverse and complex driving scenarios by blending real and virtual elements. This approach offers significant advantages in expanding the scope and depth of testing while reducing the risks of real-world testing. However, VILS has several limitations. First, because it does not fully account for the real-world environment, it can only be implemented in specific, limited locations, such as a 6-axis simulator or a proving ground. This limits the scope of simulation and makes it difficult to perfectly reproduce a vehicle's responses to changes in the external environment. Second, when VILS is implemented on a proving ground, the freedom to set up and implement scenarios is restricted due to other vehicles already under test. This reduces testing flexibility and can hinder the collection of all necessary data.

[0008] The purpose of the present invention is to provide a new technology that overcomes the limitations of existing VILS and enables more effective simulation in a realistic environment.

[0009] The purpose of the present invention is to provide a technology that can more organically integrate virtual and real environmental elements within a VILS system and accurately simulate and analyze the responses of autonomous vehicles under various external environmental conditions.

[0010] The purpose of the present invention is to enable VIL testing in an actual road environment by fusing actual surrounding vehicle information with surrounding vehicle information generated in a virtual environment.

[0011] The present invention aims to reduce the possibility of collision between virtual and real environments when the two environments are mixed.

[0012] A VILS-based test system for an autonomous vehicle according to an embodiment may include a sensor fusion precision positioning unit that estimates the current location of the vehicle based on information collected from a GPS signal receiving device, an internal vehicle sensor, and a lidar sensor, and an integrated processing unit that integrates real-world data collected from sensors of the autonomous vehicle into a virtual environment of the autonomous vehicle simulator by applying an Intelligent Driver Model (IDM) that approximates the driver's actions and integrates responses.

[0013] The sensor fusion precision positioning unit according to one embodiment may include a pose estimation unit that estimates pose information of a vehicle by matching a precision map based on lidar sensor data, a twist estimation unit that estimates twist information of a vehicle by estimating at least one of a speed, an angular speed, an acceleration, or an angular acceleration of the vehicle by utilizing information from an internal sensor of the vehicle and an inertial measurement unit (IMU), and a pose-twist fusion filter unit that fuses the estimated pose information and the twist information to output the most probable position, speed, acceleration, and covariance.

[0014] According to one embodiment, the pose estimation unit can estimate the position of the vehicle using a Normal Distributions Transform (NDT) algorithm and estimate the pose information based on the estimated position.

[0015] The integrated processing unit according to one embodiment may include a 3D object detection unit that detects real-world data in a 3D space using a deep learning-based 3D object detection algorithm, and a test scenario control unit that modifies a test scenario in real time based on interaction with an actual vehicle so that information generated in a virtual environment reflects and adapts to conditions of the actual environment.

[0016] An operation method of a VILS-based test system for an autonomous vehicle according to an embodiment may include a step of estimating a current location of a vehicle based on information collected from a GPS signal receiving device, a vehicle internal sensor, and a lidar sensor, and a step of applying an Intelligent Driver Model (IDM) that approximates a driver's behavior and integrates responses to integrate real-world data collected from sensors of the autonomous vehicle into a virtual environment of the autonomous vehicle simulator.

[0017] The step of estimating the current position of the vehicle based on information collected from the GPS signal receiving device, the vehicle internal sensor, and the lidar sensor according to one embodiment may include the step of estimating pose information of the vehicle by matching the lidar sensor data with a precision map, the step of estimating twist information of the vehicle by estimating at least one of the speed, angular speed, acceleration, or angular acceleration of the vehicle by utilizing the vehicle internal sensor and inertial measurement unit (IMU) information, and the step of merging the estimated pose information and the twist information to output the most probable position, speed, acceleration, and covariance.

[0018] The step of estimating pose information of a vehicle by matching the lidar sensor data with a precision map according to an embodiment may include a step of estimating the position of the vehicle using a Normal Distributions Transform (NDT) algorithm, and estimating the pose information based on the estimated position.

[0019] The step of integrating real-world data collected from sensors of the autonomous vehicle into a virtual environment of the autonomous driving simulator by applying an Intelligent Driver Model (IDM) that approximates the driver's actions and integrates responses according to an embodiment of the present invention may include a step of detecting real-world data in a three-dimensional space using a deep learning-based 3D object detection algorithm, and a step of modifying a test scenario in real time based on interaction with an actual vehicle so that information generated in the virtual environment reflects and adapts to conditions of the actual environment.

[0020] According to one example, it can provide a new technology that overcomes the limitations of existing VILS and enables more effective simulations in realistic environments.

[0021] According to one embodiment, it is possible to provide technology that can more organically integrate virtual and real environmental elements within a VILS system and accurately simulate and analyze the responses of autonomous vehicles under various external environmental conditions.

[0022] According to one embodiment, VIL testing is possible in an actual road environment by fusing information about actual surrounding vehicles with information about surrounding vehicles generated in a virtual environment.

[0023] In one embodiment, when a virtual environment and a real environment are mixed, the possibility of collisions occurring between the two environments can be reduced.

[0024] FIG. 1 is a drawing illustrating a VILS-based test system for an autonomous vehicle according to an embodiment.

[0025] FIG. 2 is a drawing that more specifically explains the operation of the entire system to which the VILS-based test system of an autonomous vehicle according to one embodiment is applied.

[0026] Figure 3 is a drawing that more specifically explains a sensor fusion precision positioning unit according to one embodiment.

[0027] Figure 4 is a drawing that more specifically explains an integrated processing unit according to one embodiment.

[0028] Figure 5 is a drawing showing the results of 3D object detection.

[0029] Figure 6 is a flowchart illustrating an operation method of a VILS-based test system for an autonomous vehicle according to an embodiment.

[0030] Specific structural or functional descriptions of embodiments according to the concept of the present invention disclosed in this specification are merely illustrative for the purpose of explaining embodiments according to the concept of the present invention, and embodiments according to the concept of the present invention may be implemented in various forms and are not limited to the embodiments described in this specification.

[0031] Embodiments according to the concept of the present invention may have various modifications and take various forms, and thus, embodiments are illustrated in the drawings and described in detail in this specification. However, this is not intended to limit embodiments according to the concept of the present invention to specific disclosed forms, but rather includes modifications, equivalents, or alternatives that fall within the spirit and technical scope of the present invention.

[0032] While terms such as "first" or "second" may be used to describe various components, these components should not be limited by these terms. These terms are intended solely to distinguish one component from another. For example, a first component may be referred to as a "second component," and similarly, a second component may also be referred to as a "first component," without departing from the scope of the invention.

[0033] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components in between. Conversely, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between. Expressions that describe relationships between components, such as "between," "immediately between," or "directly adjacent to," should be interpreted similarly.

[0034] The terminology used herein is for the purpose of describing specific embodiments only and is not intended to limit the present invention. The singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, it should be understood that the terms "comprises" or "has" are intended to specify the presence of a described feature, number, step, operation, component, part, or combination thereof, but do not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0035] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0036]

[0037] Hereinafter, embodiments will be described in detail with reference to the attached drawings. However, the scope of the patent application is not limited or restricted by these embodiments. The same reference numerals in each drawing represent the same components.

[0038] FIG. 1 is a drawing illustrating a VILS-based test system (100) for an autonomous vehicle according to one embodiment.

[0039] A VILS-based test system (100) for an autonomous vehicle according to an embodiment overcomes the limitations of existing VILS and enables more effective simulation in a real environment.

[0040] To this end, a VILS-based test system (100) for an autonomous vehicle according to an embodiment can more organically integrate virtual and real environmental elements within the VILS system and accurately simulate and analyze the response of an autonomous vehicle under various external environmental conditions.

[0041] Specifically, a VILS-based test system (100) for an autonomous vehicle according to one embodiment may include a sensor fusion precision positioning unit (110) and an integrated processing unit (120).

[0042] A sensor fusion precision positioning unit (110) according to one embodiment can estimate the current location of a vehicle based on information collected from a GPS signal receiving device, a vehicle internal sensor, and a lidar sensor.

[0043] Accurately estimating the location of an autonomous vehicle is crucial for safe and efficient operation. To achieve this, a sensor fusion precision positioning unit (110) combining various sensors and technologies is required. According to one embodiment, the sensor fusion precision positioning unit (110) integrates information collected from a GPS signal receiver, in-vehicle sensors, and a LiDAR sensor to estimate the vehicle's current location with high precision.

[0044] The GPS (Global Positioning System) signal receiver plays a fundamental role in estimating a vehicle's location. This device receives signals transmitted from multiple satellites and calculates the vehicle's geographic coordinates. While GPS can accurately determine location anywhere in the world, its signals can be weakened or blocked in environments such as tall buildings or tunnels in urban areas. To compensate for these situations, the sensor fusion precision positioning unit (110) utilizes data from other sensors.

[0045] In-vehicle sensors provide a variety of data regarding the vehicle's status and operation. For example, the inertial measurement unit (IMU) measures the vehicle's acceleration and rotational speed using accelerometers and gyroscopes. This information is useful for tracking the vehicle's movements even when the GPS signal is weak or interrupted. Furthermore, wheel speed sensors and steering angle sensors are also included, allowing the vehicle to accurately determine its direction and speed.

[0046] LiDAR sensors use laser pulses to create a 3D map of the surrounding environment. This provides high-resolution distance measurement data, enabling precise recognition of obstacles and road surfaces around the vehicle. Data from LiDAR complements GPS and IMU data, and is particularly essential for precise positioning and path planning.

[0047] The sensor fusion precision positioning unit (110) integrates data collected from the various sensors mentioned above to estimate the vehicle's current location. To achieve this, a sensor data fusion algorithm is used. This algorithm calculates an optimal location estimate by considering the strengths and weaknesses of each sensor. For example, combining absolute position information from GPS with relative movement information from the IMU enables more accurate location determination. Furthermore, the reliability of the location estimate is enhanced by adding environmental awareness data from the lidar sensor.

[0048] An integrated processing unit (120) according to one embodiment can integrate real-world data collected from sensors of the autonomous vehicle into a virtual environment of the autonomous driving simulator by applying an Intelligent Driver Model (IDM) that approximates the driver's actions and integrates responses.

[0049] For the efficient and safe operation of autonomous vehicles, integrating data from the real world and virtual environments to make accurate decisions is essential. To achieve this, the integrated processing unit (120) applies an Intelligent Driver Model (IDM), which approximates driver behavior and integrates responses. This model allows for the integration of real-world data collected from the autonomous vehicle's sensors into the virtual environment of the autonomous driving simulator.

[0050]

[0051] *Specifically, the integrated processing unit (120) comprehensively processes data collected from various sensors of the autonomous vehicle, thereby aligning simulations in a virtual environment with real-world situations. This can be used to test and verify the autonomous vehicle's decision-making algorithm.

[0052] The Intelligent Driver Model (IDM) is a method for modeling vehicle motion by mathematically approximating driver behavior. This model requires information on the inter-vehicle distance, relative velocity, target velocity, and acceleration.

[0053] The distance between vehicles indicates the distance maintained from the vehicle in front, the relative speed indicates the speed difference from the vehicle in front, the target speed indicates the target speed according to road conditions, and the acceleration indicates the change from the current speed to the target speed.

[0054] IDM enables autonomous vehicles to react like real drivers, resulting in more natural and safer driving.

[0055] Autonomous vehicles collect data about their surroundings through various sensors (cameras, lidar, radar, GPS, etc.). This data includes road conditions, obstacles, and the location and speed of other vehicles. The integrated processing unit (120) receives this data in real time and analyzes it using sophisticated algorithms to transform it into meaningful information.

[0056] The integrated processing unit (120) can process real-world data and then integrate it into the virtual environment of the autonomous driving simulator. The virtual environment simulates the actual driving environment, allowing for testing of autonomous vehicle driving algorithms. The integrated processing unit (120) can align the vehicle's position, speed, path, etc. in the virtual environment with real-world data.

[0057] FIG. 2 is a drawing (200) that more specifically explains the operation of the entire system to which the VILS-based test system of an autonomous vehicle according to one embodiment is applied.

[0058] Conventional VILS is primarily used in controlled environments, such as six-axis simulators or proving grounds, where no other vehicles are present. These environments, due to differences in actual road conditions, limit the scope and effectiveness of simulations. However, if real-world data acquired from actual vehicles can be organically integrated with the virtual environment, the primary limitation of conventional VILS—that testing can only be performed in controlled environments—can be overcome.

[0059] The entire system, which is equipped with a VILS-based test system for autonomous vehicles according to an embodiment, can be effectively verified in a typical road environment by integrating information from the virtual environment and the real world.

[0060] When virtual and real environments are combined, conflicts between the two environments are likely to arise. Therefore, information generated in a virtual environment can be designed to take into account interactions with elements of the real environment.

[0061] The overall system, to which the VILS-based test system of an autonomous vehicle according to an embodiment is applied, provides a new VILS system architecture as shown in Fig. 2 to effectively manage such interactions.

[0062] This entire system is configured to utilize essential precision positioning technology used in autonomous driving to synchronize the virtual environment with the real world, and 3D object detection technology to distinguish objects in the surrounding environment in the real world.

[0063] The Visual-Inertial-Lidar-Simulation (VILS)-based test system for autonomous vehicles is a complex system that integrates various components to test vehicle performance and safety. It consists of a sensing module, perception module, localization module, simulator, IDM module, ADAS software, and actuator.

[0064] The sensing module collects data from various sensors in an autonomous vehicle. These include cameras, LiDAR, radar, and GPS. This module gathers information about the surrounding environment in real time and transmits it to the next module.

[0065] The Perception module processes data collected from the Sensing module to recognize the vehicle's surroundings. For example, it performs object detection, lane recognition, and traffic light status recognition. This module provides crucial environmental information needed while the vehicle is driving, thereby enabling safe driving.

[0066] The localization module can correspond to a sensor fusion precision positioning unit that estimates the current location of the vehicle based on information collected from a GPS signal receiving device, an internal vehicle sensor, and a lidar sensor.

[0067] The localization module accurately estimates the vehicle's current location based on information collected from GPS signal receivers, in-vehicle sensors (IMU, wheel speed sensors, etc.), and lidar sensors. This module fuses various sensor data to track the vehicle's location in real time, and location information serves as core data for autonomous driving systems.

[0068] The IDM module can correspond to an integrated processing unit that integrates real-world data collected from sensors of the autonomous vehicle into the virtual environment of the autonomous driving simulator by applying an Intelligent Driver Model (IDM) that approximates the driver's actions and integrates responses.

[0069] The Intelligent Driver Model (IDM) module applies a vehicle-following model that approximates driver behavior and integrates responses. This module integrates real-world data collected from the autonomous vehicle's sensors into a virtual simulator environment. The IDM module naturally models the vehicle's driving patterns, ensuring that driving tests in the virtual environment are consistent with reality.

[0070] A simulator is a tool for testing autonomous vehicles in a virtual environment. It can evaluate vehicle performance based on real-world data and driving scenarios. The simulator receives data processed by the IDM module and reflects it in a virtual environment, reproducing various driving situations to verify the stability and reliability of autonomous driving systems.

[0071] ADAS S / W (Advanced Driver Assistance Systems Software) is a software system that controls the driving of autonomous vehicles. Based on data provided by the Perception and Localization modules, this software plans the driving path and controls the vehicle's speed and direction. ADAS S / W provides various driving assistance features to ensure safe and efficient driving.

[0072] Actuators translate commands from ADAS software into actual vehicle movement. This includes the vehicle's engine, brakes, and steering. Actuators control the vehicle according to its driving path, ensuring that the vehicle moves as planned by the autonomous driving system.

[0073] The Sensing module collects real-time data from sensors such as cameras, lidar, radar, and GPS, and the Perception module processes this data to perform tasks such as object detection and lane recognition. Furthermore, the Localization module fuses various sensor data to accurately estimate the vehicle's current location, and the IDM module integrates real-world data into a virtual simulator and approximates driver behavior to model driving patterns.

[0074] Additionally, the simulator can test the driving of a vehicle in a virtual environment, reproduce various driving scenarios to verify the stability of the system, and ADAS software can plan a driving path and control the speed and direction of the vehicle based on recognized environmental information and location data.

[0075] The actuator can actually move the vehicle and perform driving according to the commands of the ADAS S / W.

[0076] In this way, the VILS-based test system for autonomous vehicles can test and optimize the performance of autonomous driving systems by integrating real and virtual environments through collaboration among each component.

[0077] Figure 3 is a drawing that more specifically explains a sensor fusion precision positioning unit (110) according to one embodiment.

[0078] To synchronize autonomous driving simulators and actual autonomous vehicles in VILS, accurately estimating the vehicle's current location is essential. Existing systems rely on expensive DGPS equipment to estimate vehicle location, resulting in significant costs for building VILS. The present invention utilizes sensor fusion precision positioning technology, which integrates GPS, in-vehicle sensors, and lidar sensors in real time, as a cost-effective alternative.

[0079] The sensor fusion precision positioning unit (110) may include a pose estimation unit (310), a twist estimation unit (320), and a pose-twist fusion filter unit (330).

[0080] The pose estimation unit (310) plays a key role in determining the precise location and orientation of an autonomous vehicle. It estimates the vehicle's pose information by matching LiDAR sensor data with a precision map. Pose estimation is a crucial process that helps autonomous vehicles accurately recognize their location and follow the correct path.

[0081] The pose estimation unit (310) processes data collected from the lidar sensor to estimate the vehicle's 3D position and posture. Here, pose information includes both the vehicle's position and its orientation (roll, pitch, and yaw). This information enables the autonomous driving system to accurately perceive its environment, plan a safe path, and drive.

[0082] Lidar sensors use laser pulses to generate a 3D point cloud of the surrounding environment. This data has extremely high resolution and can accurately measure the distance and shape of surrounding objects. Lidar sensor data plays a crucial role in pose estimation for autonomous vehicles.

[0083] A precision map is high-precision 3D map data that includes information on road shapes, buildings, obstacles, and more. The pose estimation unit (310) compares real-time point cloud data collected from the lidar sensor with the precision map to estimate the vehicle's current position and pose. This process is called mapping or matching.

[0084] The twist estimation unit (320) can estimate twist information about the vehicle by estimating at least one of the vehicle's speed, angular velocity, acceleration, or angular acceleration using information from an internal vehicle sensor and an inertial measurement unit (IMU).

[0085] The twist estimation unit (320) accurately determines the current speed and direction of the vehicle to ensure that it follows a given path.

[0086] It utilizes acceleration and angular acceleration information to prevent safety issues caused by rapid acceleration or rotation, monitors the vehicle's dynamic status in real time to maintain driving stability, and combines data from other sensors to enable more precise environmental awareness.

[0087] In addition, the twist estimation unit (320) can collect acceleration and angular velocity data in real time from the vehicle's internal sensors and IMU, filter the collected data, and remove noise to extract accurate values.

[0088] In addition, the twist estimation unit (320) calculates the speed, angular velocity, acceleration, and angular acceleration of the vehicle using the filtered data, and integrates the estimated twist information with other modules of the autonomous driving system to comprehensively understand the dynamic state of the vehicle.

[0089] The pose-twist fusion filter unit (330) can fuse the estimated pose information and the twist information to output the most probable position, velocity, acceleration, and covariance.

[0090] The pose-twist fusion filter unit (330) estimates the dynamic state of the vehicle in real time to support the autonomous driving system in making accurate driving decisions.

[0091] The pose-twist fusion filter unit (330) integrates estimated pose information to calculate the vehicle's precise position and posture. Pose information can typically be estimated based on external environmental sensors, such as lidar sensors.

[0092] The pose-twist fusion filter unit (330) can calculate the dynamic state of the vehicle by integrating estimated twist information (speed, angular velocity, acceleration, etc.). The twist information is mainly estimated based on the vehicle's internal sensors and IMU.

[0093] The pose-twist fusion filter unit (330) combines pose information and twist information to output the most likely position, velocity, acceleration, and covariance. This aims to represent the current state of the vehicle as accurately as possible.

[0094] The pose-twist fusion filter unit (330) processes data using various fusion filter algorithms. Representative examples include the Kalman Filter and the Extended Kalman Filter (EKF). These algorithms estimate the vehicle's status based on incoming data in real time and minimize prediction errors, thereby increasing accuracy.

[0095] The Kalman filter is used in linear systems and assumes a linear relationship between pose information and twist information, while the Extended Kalman Filter can also be used in nonlinear systems and effectively fuses pose information and twist information using nonlinear functions.

[0096] For example, the pose estimation unit (310) can estimate the position of the vehicle using the Normal Distributions Transform (NDT) algorithm and estimate the pose information based on the estimated position.

[0097] The Normal Distributions Transform (NDT) algorithm can map point cloud data in 3D space by transforming it into a normal distribution. It can be primarily used to process 3D point cloud data collected from LiDAR sensors.

[0098] The Normal Distribution Transform (NDT) algorithm enables precise position estimation using high-resolution lidar data. It is particularly suitable for real-time applications due to its fast computational speed, robust processing capabilities against noise, and adaptability to environmental changes. Furthermore, the pose estimation unit (310) can receive 3D point cloud data collected from the lidar sensor as input.

[0099] The pose estimation unit (310) maps the input point cloud data into a normal distribution using an NDT algorithm. This process creates a map that reflects the structure of the surrounding environment and the locations of obstacles.

[0100] The pose estimation unit (310) estimates the current location of the vehicle based on the NDT mapping results, and can accurately determine the location of the vehicle by finding a matching point between the mapped data and the existing map using the NDT algorithm.

[0101] The pose estimation unit (310) estimates the vehicle's pose information (direction, attitude) through additional calculations based on the estimated position. This process provides precise pose information by considering the vehicle's movement path and rotation angle.

[0102] Figure 4 is a drawing that more specifically explains an integrated processing unit according to one embodiment.

[0103] An integrated processing unit (120) according to one embodiment may include a 3D object detection unit (121) and a test scenario control unit (122).

[0104] The 3D object detection unit (121) can detect real-world data in a 3D space using a 3D object detection algorithm based on deep learning.

[0105] In most cases, neural network architectures are based on convolutional neural networks (CNNs), which include methods for processing 3D point cloud data rather than 2D images.

[0106] The 3D object detection unit (121) uses 3D point cloud data collected from a LiDAR sensor or other depth information as input. This data accurately reflects the shape and distance of objects and obstacles in the real world.

[0107] The network analyzes input data to identify the location, size, and shape of objects in a given space. This is typically represented by the object's bounding box, and it can even classify the object's category.

[0108] The 3D object detection unit (121) receives 3D data collected from a lidar sensor or other depth sensor in the surrounding environment as input.

[0109] The test scenario control unit (122) can modify the test scenario in real time based on interaction with the actual vehicle to ensure that information generated in the virtual environment reflects and adapts to the conditions of the actual environment.

[0110] Integrating real-world data collected from autonomous vehicle sensors into a virtual environment presupposes the ability to precisely detect objects such as vehicles and pedestrians in three-dimensional space. This is because determining the precise location and size of objects in three dimensions is crucial. Therefore, it is advantageous to use a lidar sensor, which can detect objects further away than a camera and provides more precise three-dimensional distance information.

[0111] In the present invention, a deep learning-based 3D object detection algorithm can be applied to classify objects from a lidar sensor and identify and locate them.

[0112] When integrating real-world data collected from autonomous vehicles into a virtual environment, the potential for conflict between the two environments arises. The virtual environment must be designed to dynamically interact with real-world elements, ensuring that information generated in the virtual environment accurately reflects and adapts to real-world conditions. To address this, test scenarios can be modified in real time based on interactions with the real vehicle.

[0113] Intelligent Driver Model (IDM) can be applied to facilitate synchronization between virtual and real objects.

[0114] IDM is a vehicle-following model that approximates the behavior and responses of a skilled driver. In the present invention, it can be used to model the behavior and interactions of a virtual vehicle. VILS simulations can more accurately reproduce vehicle behavior patterns under real-world traffic conditions, thereby enhancing the predictive power of simulations.

[0115] Figure 5 is a drawing showing the results of 3D object detection.

[0116] In experiments related to the present invention, a VILS test environment was built by integrating a vehicle equipped with a lidar sensor, GPS, and IMU sensor with an autonomous driving simulator, and testing was conducted on the DGIST campus. A VILS test scenario that mixed virtual and real environments was created, in which the vehicle quickly drove behind a virtual lane while changing lanes to evade a stopped vehicle ahead.

[0117] In this scenario, the autonomous driving software recognized the presence of a real vehicle and two virtual vehicles, waited for the virtual vehicles to pass, and then performed a lane change. This test allows for the simulation of complex, realistic interactions under controlled conditions, enhancing the safety of testing and providing valuable behavioral data on autonomous systems.

[0118] Currently, the development and verification of autonomous vehicles requires safely and effectively reproducing complex real-world driving environments. The present invention addresses this challenge through the organic integration of virtual and real environments, providing improved VILS technology that blends virtual and real environments.

[0119] In particular, data is integrated in real time by utilizing 3D object detection technology and IDM, a representative longitudinal technology of traffic models, to synchronize between autonomous driving simulators and actual vehicles.

[0120] The present invention enhances the flexibility of virtual simulation and expands its scope of application, enabling it to effectively simulate and respond to complex driving scenarios encountered on public roads. Furthermore, this technology minimizes the risks of real-world vehicle testing while simultaneously expanding the scope and depth of testing.

[0121] Figure 6 is a flowchart illustrating an operation method of a VILS-based test system for an autonomous vehicle according to an embodiment.

[0122] According to an embodiment, a method of operating a VILS-based test system for an autonomous vehicle can estimate the current location of a vehicle based on information collected from a GPS signal receiving device, a vehicle internal sensor, and a lidar sensor (step 601).

[0123] Using the Normal Distributions Transform (NDT) algorithm, the position of the vehicle can be estimated, and the pose information can be estimated based on the estimated position.

[0124] According to an embodiment, a method of operating a VILS-based test system for an autonomous vehicle can estimate at least one of a vehicle's speed, angular velocity, acceleration, or angular acceleration by utilizing information from a vehicle's internal sensor and an inertial measurement unit (IMU) (step 602).

[0125] An operation method of a VILS-based test system for an autonomous vehicle according to an embodiment can output the most probable position, velocity, acceleration, and covariance by fusing the estimated pose information and the twist information (step 603).

[0126] The operation method of the VILS-based test system for an autonomous vehicle according to an embodiment can apply an Intelligent Driver Model (IDM) that approximates the driver's actions and integrates the responses (step 604).

[0127] The operating method of a VILS-based test system for an autonomous vehicle according to an embodiment can detect real-world data in a three-dimensional space using a deep learning-based 3D object detection algorithm (step 605).

[0128] The method of operating a VILS-based test system for an autonomous vehicle according to an embodiment can modify a test scenario in real time based on interactions with an actual vehicle so that information generated in a virtual environment reflects and adapts to conditions of an actual environment (step 606).

[0129] Ultimately, the present invention can provide a new technology that overcomes the limitations of existing VILS and enables more effective simulation in a realistic environment.

[0130] In addition, it can provide technology that can more organically integrate virtual and real environment elements within the VILS system, accurately simulate and analyze the responses of autonomous vehicles under various external environmental conditions, and enable VIL testing in real road environments by fusing information about real surrounding vehicles with information about surrounding vehicles generated in a virtual environment.

[0131] In addition, it can reduce the possibility of conflict between virtual and real environments when they are mixed.

[0132]

[0133] The devices described above may be implemented as hardware components, software components, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose computers or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding to them. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used alone; however, one of ordinary skill in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, a processing unit may include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0134] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may, independently or collectively, command the processing device. The software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave, for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more computer-readable recording media.

[0135] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the embodiment or may be those known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of the program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

[0136] Although the embodiments described above have been described with limited drawings, those skilled in the art will recognize that various modifications and variations can be made based on the above description. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0137] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

Claims

1. A sensor fusion precision positioning unit that estimates the current location of the vehicle based on information collected from a GPS signal receiving device, an internal vehicle sensor, and a lidar sensor; and An integrated processing unit that integrates real-world data collected from the sensors of the autonomous vehicle into the virtual environment of the autonomous driving simulator by applying an Intelligent Driver Model (IDM) that approximates the driver's actions and integrates responses. A VILS-based test system for an autonomous vehicle, characterized by including:

2. In paragraph 1, The above sensor fusion precision positioning unit, A pose estimation unit that estimates the pose information of a vehicle by matching it with a precision map based on lidar sensor data; A twist estimation unit that estimates twist information about a vehicle by estimating at least one of the vehicle's speed, angular velocity, acceleration, or angular acceleration using information from an internal sensor and an inertial measurement unit (IMU) of the vehicle; and A pose-twist fusion filter unit that fuses the estimated pose information and the twist information to output the most likely position, velocity, acceleration, and covariance. A VILS-based test system for an autonomous vehicle, characterized by including:

3. In paragraph 2, The above pose estimation unit, A VILS-based test system for an autonomous vehicle, characterized in that the position of the vehicle is estimated using a Normal Distributions Transform (NDT) algorithm and the pose information is estimated based on the estimated position.

4. In paragraph 1, The above integrated processing unit, A 3D object detection unit that detects real-world data in a 3D space using a 3D object detection algorithm based on deep learning; and A test scenario control unit that modifies test scenarios in real time based on interactions with real vehicles to ensure that information generated in a virtual environment reflects and adapts to conditions in the real environment. A VILS-based test system for an autonomous vehicle, characterized by including:

5. A step of estimating the current location of the vehicle based on information collected from a GPS signal receiving device, an internal vehicle sensor, and a lidar sensor; and A step of integrating real-world data collected from the sensors of the autonomous vehicle into the virtual environment of the autonomous driving simulator by applying an Intelligent Driver Model (IDM) that approximates the driver's actions and integrates responses. A method of operating a VILS-based test system for an autonomous vehicle, characterized in that it includes:

6. In paragraph 5, The step of estimating the current location of the vehicle based on information collected from the above GPS signal receiving device, vehicle internal sensor, and lidar sensor is as follows: A step of estimating the pose information of a vehicle by matching it with a precision map based on lidar sensor data; A step of estimating twist information about a vehicle by estimating at least one of the vehicle's speed, angular velocity, acceleration, or angular acceleration using information from a vehicle's internal sensor and an inertial measurement unit (IMU); and A step of fusing the estimated pose information and the twist information to output the most likely position, velocity, acceleration, and covariance. A method of operating a VILS-based test system for an autonomous vehicle, characterized in that it includes:

7. In paragraph 6, The step of estimating the pose information of the vehicle by matching it with a precision map based on the above lidar sensor data is as follows: A step of estimating the position of a vehicle using the Normal Distributions Transform (NDT) algorithm and estimating the pose information based on the estimated position. A method of operating a VILS-based test system for an autonomous vehicle, characterized in that it includes:

8. In paragraph 5, The step of integrating real-world data collected from the sensors of the autonomous vehicle into the virtual environment of the autonomous driving simulator by applying an Intelligent Driver Model (IDM) that approximates the driver's actions and integrates the responses is as follows. A step of detecting real-world data in a three-dimensional space using a deep learning-based 3D object detection algorithm; and A step to modify test scenarios in real time based on interactions with real vehicles to ensure that information generated in a virtual environment reflects and adapts to conditions in the real environment. A method of operating a VILS-based test system for an autonomous vehicle, characterized in that it includes:

Images