Vils-based testing system and method for autonomous vehicles considering interaction between virtual and real environments

Abstract

The present invention relates to a VILS-based testing technology for autonomous vehicles that enables VIL testing in a real road environment by fusing actual surrounding vehicle information and surrounding vehicle information generated in a virtual environment. A VILS-based test system for an autonomous vehicle according to one embodiment may include a sensor fusion precision positioning unit that estimates the current position of the vehicle based on information collected from a GPS signal receiver, an in-vehicle sensor, and a LiDAR sensor, and an integrated processing unit that integrates real-world data collected from the sensors of the autonomous vehicle into the virtual environment of the autonomous driving simulator by applying an Intelligent Driver Model (IDM) that approximates the driver's behavior and integrates the response.

Description

Technology Field

[0001] The present invention relates to a VILS-based testing technology for autonomous vehicles that enables VIL testing in a real road environment by fusing actual surrounding vehicle information and surrounding vehicle information generated in a virtual environment. Background Technology

[0002] Real-vehicle-based testing is essential for autonomous driving technology to respond to various scenarios that may occur on public roads. However, there are limitations to real-vehicle verification because setting up the test environment takes a significant amount of time, and there is a risk of injury as impacts are applied to occupants during crash tests. VILS is a test environment where a real-world test vehicle is fused with a virtual external environment, providing a method to test various scenarios using actual vehicles.

[0003] However, since it does not take into account the actual surrounding environment, it can only be implemented in limited locations such as 6-axis simulators or test tracks.

[0004] Specifically, software verification is a critical process in the development of modern autonomous vehicles. According to the RAND Corporation, commercialization of fully autonomous driving requires at least 100 vehicles to conduct approximately 440 million kilometers of driving tests. However, the recent COVID-19 pandemic has significantly restricted real-vehicle testing by autonomous driving companies, resulting in a substantial decrease in driving distance compared to the past.

[0005] To effectively respond to the various situations that may occur during autonomous driving, the best solution is to develop diverse scenarios that could take place on actual roads and test them using real vehicles. However, real-vehicle testing requires a significant amount of time to set up the experimental environment, and there is a risk of injury because impact is applied to the occupants during crash tests.

[0006] There are various methods available to safely test by simulating complex real-world driving environments. In particular, verification methods utilizing vehicle simulation software and Hardware In-the-Loop Simulation (HILS) equipment, which can simulate actual automotive test environments by constructing a virtual environment, are widely used. While these methods have the advantage of replacing actual vehicle testing, it is difficult to perfectly reflect the dynamic characteristics of a real vehicle. To address this problem, Vehicle In-the-Loop Simulation (VILS) testing, which fuses a real-world test vehicle with a virtual external environment, is an alternative.

[0007] VILS is a method that combines real and virtual elements to effectively reproduce more diverse and complex driving scenarios. This approach offers significant advantages in expanding the scope and depth of testing while reducing the risks associated with actual vehicle testing. However, there are several limitations to the application of VILS. First, because VILS does not fully account for the real environment, it can only be implemented in specific and restricted locations, such as 6-axis simulators or proving grounds. This limits the scope of the simulation and makes it difficult to perfectly reproduce the vehicle's response to changes in the external environment. Second, when implementing VILS at a proving ground, the presence of other vehicles already undergoing testing restricts the free setting and execution of scenarios. This reduces test flexibility and can hinder the collection of all necessary data. Prior art literature

[0008] Korean Published Patent No. 10-2021-0159582 "Bidirectional LDM-HIL Simulation Device and Method Reflecting Real and Virtual Environments" Korean Published Patent No. 10-2021-0149861 "Autonomous Driving Verification System and Method in Which Real Information and Virtual Information Are Selectively Mixed" Korean Published Patent No. 10-2020-0004510 "Method and Device for Generating Domain-Adapted Traffic Scenarios for a Virtual Driving Environment to Learn, Test, and Verify an Autonomous Driving Vehicle" The problem to be solved

[0009] The present invention aims to provide a new technology that overcomes the limitations of existing VILS and enables more effective simulation in realistic environments.

[0010] The present invention aims to provide a technology that more organically integrates virtual and real environmental elements within a VILS system and can accurately simulate and analyze the response of an autonomous vehicle under various external environmental conditions.

[0011] The present invention aims to enable VIL testing in a real road environment by fusing actual surrounding vehicle information and surrounding vehicle information generated in a virtual environment.

[0012] The present invention aims to reduce the possibility of conflict between two environments when a virtual environment and a real environment are mixed. means of solving the problem

[0013] A VILS-based test system for an autonomous vehicle according to one embodiment may include a sensor fusion precision positioning unit that estimates the current position of the vehicle based on information collected from a GPS signal receiver, an in-vehicle sensor, and a LiDAR sensor, and an integrated processing unit that integrates real-world data collected from the sensors of the autonomous vehicle into the virtual environment of the autonomous driving simulator by applying an Intelligent Driver Model (IDM) that approximates the driver's behavior and integrates the response.

[0014] The sensor fusion precision positioning unit according to one embodiment may include a pose estimation unit that estimates pose information of a vehicle by matching it with a precision map based on LiDAR sensor data, a twist estimation unit that estimates twist information for a vehicle by estimating at least one of the vehicle's speed, angular velocity, acceleration, or angular acceleration using internal vehicle sensors and inertial measurement unit (IMU) information, and a pose-twist fusion filter unit that fuses the estimated pose information and the twist information to output the most probable position, speed, acceleration, and covariance.

[0015] The pose estimation unit according to one embodiment can estimate the position of a vehicle using a Normal Distributions Transform (NDT) algorithm and estimate the pose information based on the estimated position.

[0016] The integrated processing unit according to one embodiment may include a 3D object detection unit that detects real-world data in a 3D space using a deep learning-based 3D object detection algorithm, and a test scenario control unit that modifies a test scenario in real time based on interaction with a real vehicle so that information generated in a virtual environment reflects and adapts to the conditions of the actual environment.

[0017] A method of operation of a VILS-based test system for an autonomous vehicle according to one embodiment may include the step of estimating the current position of the vehicle based on information collected from a GPS signal receiver, an in-vehicle sensor, and a LiDAR sensor, and the step of integrating real-world data collected from the sensors of the autonomous vehicle into the virtual environment of the autonomous driving simulator by applying an Intelligent Driver Model (IDM) that approximates the driver's behavior and integrates the response.

[0018] The step of estimating the current position of a vehicle based on information collected from the GPS signal receiving device, the vehicle interior sensor, and the LiDAR sensor according to one embodiment may include: a step of estimating pose information of the vehicle by matching it with a precision map based on LiDAR sensor data; a step of estimating twist information for the vehicle by estimating at least one of the vehicle's speed, angular velocity, acceleration, or angular acceleration using the vehicle interior sensor and inertial measurement unit (IMU) information; and a step of outputting the most probable position, speed, acceleration, and covariance by fusing the estimated pose information and the twist information.

[0019] The step of estimating pose information of a vehicle by matching it with a precision map based on the lidar sensor data according to one embodiment may include the step of estimating the position of the vehicle using a Normal Distributions Transform (NDT) algorithm and estimating the pose information based on the estimated position.

[0020] The step of integrating real-world data collected from sensors of the autonomous vehicle into the virtual environment of the autonomous driving simulator by applying an Intelligent Driver Model (IDM) that approximates the behavior of the driver and integrates the response according to one embodiment may include the step of detecting real-world data in a three-dimensional space using a deep learning-based 3D object detection algorithm, and the step of modifying a test scenario in real time based on interaction with the actual vehicle so that the information generated in the virtual environment reflects and adapts to the conditions of the actual environment. Effects of the invention

[0021] According to one embodiment, a new technology can be provided that overcomes the limitations of existing VILS and enables more effective simulation in a realistic environment.

[0022] According to one embodiment, a technology can be provided that more organically integrates virtual and real environment elements within a VILS system and accurately simulates and analyzes the response of an autonomous vehicle under various external environmental conditions.

[0023] According to one embodiment, VIL testing in a real road environment is possible by fusing actual surrounding vehicle information and surrounding vehicle information generated in a virtual environment.

[0024] According to one embodiment, when a virtual environment and a real environment are mixed, the possibility of a conflict occurring between the two environments can be reduced. Brief explanation of the drawing

[0025] FIG. 1 is a diagram illustrating a VILS-based test system for an autonomous vehicle according to one embodiment. FIG. 2 is a diagram that more specifically explains the operation of the entire system to which a VILS-based test system of an autonomous vehicle according to one embodiment is applied. FIG. 3 is a drawing that more specifically explains a sensor fusion precision positioning unit according to one embodiment. FIG. 4 is a drawing that explains the integrated processing unit according to one embodiment in more detail. Figure 5 is a diagram illustrating the results of three-dimensional object detection. FIG. 6 is a flowchart illustrating the operation method of a VILS-based test system for an autonomous vehicle according to one embodiment. Specific details for implementing the invention

[0026] Specific structural or functional descriptions of embodiments according to the concept of the present invention disclosed herein are provided merely for the purpose of explaining embodiments according to the concept of the present invention, and embodiments according to the concept of the present invention may be implemented in various forms and are not limited to the embodiments described herein.

[0027] Embodiments according to the concept of the present invention may be subject to various modifications and may take various forms; therefore, embodiments are illustrated in the drawings and described in detail in this specification. However, this is not intended to limit the embodiments according to the concept of the present invention to specific disclosed forms, and includes modifications, equivalents, or substitutions that fall within the spirit and scope of the present invention.

[0028] Terms such as "first" or "second" may be used to describe various components, but said components should not be limited by said terms. For the sole purpose of distinguishing one component from another, for example, without departing from the scope of rights according to the concept of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component.

[0029] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. Conversely, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between. Expressions describing the relationships between components, such as "between," "exactly between," or "directly adjacent to," should be interpreted in the same way.

[0030] The terms used herein are used merely to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as “comprising” or “having” are intended to specify the existence of the described features, numbers, steps, actions, components, parts, or combinations thereof, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0031] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which the present invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this specification.

[0033] Hereinafter, embodiments will be described in detail with reference to the attached drawings. However, the scope of the patent application is not limited or restricted by these embodiments. Identical reference numerals in each drawing indicate identical components.

[0034] FIG. 1 is a drawing illustrating a VILS-based test system (100) of an autonomous vehicle according to one embodiment.

[0035] A VILS-based test system (100) for an autonomous vehicle according to one embodiment overcomes the limitations of existing VILS and enables more effective simulation in a real environment.

[0036] To this end, the VILS-based test system (100) of an autonomous vehicle according to one embodiment more organically integrates virtual and real environment elements within the VILS system and can accurately simulate and analyze the response of the autonomous vehicle under various external environmental conditions.

[0037] Specifically, a VILS-based test system (100) of an autonomous vehicle according to one embodiment may include a sensor fusion precision positioning unit (110) and an integrated processing unit (120).

[0038] A sensor fusion precision positioning unit (110) according to one embodiment can estimate the current position of a vehicle based on information collected from a GPS signal receiving device, a vehicle internal sensor, and a lidar sensor.

[0039] Accurate position estimation of an autonomous vehicle is very important for safe and efficient operation. To this end, a sensor fusion precision positioning unit (110) combining various sensors and technologies is required. The sensor fusion precision positioning unit (110) according to one embodiment can estimate the current position of the vehicle very precisely by integrating information collected from a GPS signal receiver, an internal vehicle sensor, and a LiDAR sensor.

[0040] The GPS (Global Positioning System) signal receiver plays the most fundamental role in estimating the vehicle's location. This device receives signals transmitted from various satellites and calculates the vehicle's geographical coordinates. While GPS can determine location with consistent accuracy anywhere in the world, signals may weaken or be blocked in environments such as tall buildings in urban areas or tunnels. To compensate for such cases, the sensor fusion precision positioning unit (110) utilizes data from other sensors together.

[0041] In-vehicle sensors provide various data regarding the vehicle's status and operation. For example, an Inertial Measurement Unit (IMU) measures the vehicle's acceleration and rotational speed through accelerometers and gyroscopes. This information is useful for tracking the vehicle's movement even when GPS signals are weak or interrupted. Additionally, sensors such as wheel speed sensors and steering angle sensors are included, enabling accurate determination of the vehicle's driving direction and speed.

[0042] LiDAR sensors use laser pulses to generate a 3D map of the surrounding environment. This provides high-resolution distance measurement data, enabling the accurate recognition of obstacles or road shapes around the vehicle. Data obtained from LiDAR sensors complements GPS and IMU data and is particularly necessary for precise position estimation and path planning.

[0043] The sensor fusion precision positioning unit (110) estimates the current position of the vehicle by integrating data collected from the various sensors mentioned above. To this end, a sensor data fusion algorithm is used. This algorithm calculates an optimal position estimation value by considering the strengths and weaknesses of each sensor. For example, a more accurate position can be determined by combining the absolute position information of GPS with the relative movement information of the IMU. In addition, the reliability of the position estimation is increased by adding environmental perception data from the LiDAR sensor.

[0044] An integrated processing unit (120) according to one embodiment can apply an Intelligent Driver Model (IDM) that approximates the driver's behavior and integrates the response, thereby integrating real-world data collected from the sensors of the autonomous vehicle into the virtual environment of the autonomous driving simulator.

[0045] For the efficient and safe operation of an autonomous vehicle, it is essential to make accurate judgments by integrating data from the real world and a virtual environment. To this end, the integrated processing unit (120) applies an Intelligent Driver Model (IDM) that approximates the driver's behavior and integrates the response. Through this model, real-world data collected from the sensors of the autonomous vehicle can be integrated into the virtual environment of the autonomous driving simulator.

[0046] Specifically, the integrated processing unit (120) comprehensively processes data collected from various sensors of the autonomous vehicle and plays a role in matching the simulation in the virtual environment with the situation in the real world. This can test and verify the decision-making algorithm of the autonomous vehicle.

[0047] The Intelligent Driver Model (IDM) is a method that models vehicle movement by mathematically approximating driver behavior. This model requires information regarding distance between vehicles, relative speed, target speed, and acceleration.

[0048] The distance between vehicles represents maintaining distance from the vehicle ahead, the relative speed represents the speed difference from the vehicle ahead, the target speed represents the target speed according to road conditions, and the acceleration represents the change from the current speed to the target speed.

[0049] Using IDM allows autonomous vehicles to react like real drivers, enabling more natural and safer driving.

[0050] Autonomous vehicles collect data of the surrounding environment through various sensors (camera, lidar, radar, GPS, etc.). This data includes road conditions, obstacles, and the location and speed of other vehicles. An integrated processing unit (120) receives this data in real time and can convert it into meaningful information by analyzing it through a sophisticated algorithm.

[0051] The integrated processing unit (120) can process real-world data and then integrate it into the virtual environment of the autonomous driving simulator. The virtual environment is a space where the driving algorithm of the autonomous vehicle can be tested by simulating the actual driving environment, and the integrated processing unit (120) can match the vehicle's position, speed, path, etc., with real-world data in the virtual environment.

[0052] FIG. 2 is a drawing (200) that more specifically explains the operation of the entire system to which a VILS-based test system of an autonomous vehicle according to one embodiment is applied.

[0053] Conventional VILS is primarily utilized in controlled environments, which refers to environments without other vehicles in the vicinity, such as 6-axis simulators or proving grounds. These environments limit the scope and effectiveness of simulations due to differences from actual road conditions. However, if real-world data acquired from actual vehicles can be organically integrated with a virtual environment, the disadvantage of conventional VILS—that testing is only possible in controlled environments—can be overcome.

[0054] The entire system to which the VILS-based test system for an autonomous vehicle according to one embodiment is applied enables effective verification in a general road environment by fusing information from the virtual environment and the real world.

[0055] When virtual and real environments are mixed, there is a high probability of conflict occurring between the two. Therefore, information generated in the virtual environment can be designed to consider interactions with elements of the real environment.

[0056] The entire system to which the VILS-based test system of an autonomous vehicle according to one embodiment is applied provides a new VILS system architecture as shown in FIG. 2 to effectively manage these interactions.

[0057] This entire system is configured to utilize the essential precision positioning technology used in autonomous driving to synchronize the virtual environment and the real world, and 3D object detection technology to distinguish surrounding environment objects in the real world.

[0058] The VILS (Visual-Inertial-Lidar-Simulation)-based test system for autonomous vehicles is a complex system that integrates various components to test the vehicle's driving performance and safety. This system consists of a sensing module, a perception module, a localization module, a simulator, an IDM module, ADAS software, and an actuator.

[0059] The sensing module is responsible for collecting data from various sensors in an autonomous vehicle. These include cameras, LiDAR, radar, and GPS. This module collects information about the surrounding environment in real time and transmits it to the next stage module.

[0060] The Perception module processes data collected from the Sensing module to recognize the situation around the vehicle. For example, it performs tasks such as object detection, lane recognition, and traffic light status assessment. This module provides critical environmental information necessary for the vehicle while driving, thereby enabling safe driving.

[0061] The localization module can correspond to a sensor fusion precision positioning unit that estimates the vehicle's current location based on information collected from a GPS signal receiver, in-vehicle sensors, and LiDAR sensors.

[0062] The Localization module accurately estimates the vehicle's current location based on information collected from GPS signal receivers, in-vehicle sensors (IMU, wheel speed sensors, etc.), and LiDAR sensors. This module fuses various sensor data to track the vehicle's position in real time, and the location information is used as core data for the autonomous driving system.

[0063] The IDM module can correspond to an integrated processing unit that integrates real-world data collected from the sensors of the autonomous vehicle into the virtual environment of the autonomous driving simulator by applying an Intelligent Driver Model (IDM) that approximates the driver's behavior and integrates the response.

[0064] The Intelligent Driver Model (IDM) module applies a vehicle-following model that approximates driver behavior and integrates responses. This module plays the role of integrating real-world data collected from the autonomous vehicle's sensors into a virtual simulator environment. By naturally modeling the vehicle's driving patterns, the IDM module ensures that driving tests in the virtual environment match reality.

[0065] The Simulator is a tool for testing the driving of autonomous vehicles in a virtual environment, capable of evaluating vehicle performance based on real-world data and driving scenarios. The Simulator receives data processed by the IDM module and reflects it in the virtual environment, reproducing various driving situations to verify the stability and reliability of the autonomous driving system.

[0066] ADAS S / W (Advanced Driver Assistance Systems Software) is a software system that controls the driving of autonomous vehicles. Based on data provided by the Perception and Localization modules, this software plans the driving path and controls the vehicle's speed and direction. ADAS S / W provides various driving assistance functions to enable the vehicle to drive safely and efficiently.

[0067] Actuators play the role of converting commands from ADAS software into actual vehicle movements. This includes the vehicle's engine, brakes, and steering. Actuators control the vehicle according to the driving path, enabling it to move as planned by the autonomous driving system.

[0068] The Sensing module collects real-time data from sensors such as cameras, LiDAR, radar, and GPS, while the Perception module processes the collected data to perform tasks such as object detection and lane recognition. Additionally, the Localization module fuses various sensor data to accurately estimate the vehicle's current location, and the IDM module integrates real-world data into a virtual simulator and approximates driver behavior to model driving patterns.

[0069] In addition, the simulator tests the vehicle's driving in a virtual environment and verifies the system's stability by reproducing various driving scenarios, and the ADAS software can plan a driving path and control the vehicle's speed and direction based on recognized environmental information and location data.

[0070] The actuator can actually move the vehicle and perform driving according to the commands of the ADAS software.

[0071] In this way, the VILS-based test system for autonomous vehicles can test and optimize the performance of the autonomous driving system by integrating the real and virtual environments through the collaboration of each component.

[0072] FIG. 3 is a drawing that more specifically explains a sensor fusion precision positioning unit (110) according to one embodiment.

[0073] In VILS, accurately estimating the vehicle's current position is essential to synchronize the autonomous driving simulator with the actual autonomous vehicle. Existing systems require expensive DGPS equipment for vehicle position estimation, resulting in high costs for establishing VILS. As a cost-effective alternative, this invention utilizes sensor fusion precision positioning technology that fuses GPS, in-vehicle sensors, and LiDAR sensors in real time.

[0074] The sensor fusion precision positioning unit (110) may include a pose estimation unit (310), a twist estimation unit (320), and a pose-twist fusion filter unit (330).

[0075] The pose estimation unit (310) plays a key role in determining the precise position and orientation of the autonomous vehicle. It estimates the vehicle's pose information by matching it with a precision map based on LiDAR sensor data. Pose estimation is an important process that helps the autonomous vehicle accurately recognize its position and follow the correct path.

[0076] The pose estimation unit (310) processes data collected from the LiDAR sensor to estimate the three-dimensional position and attitude of the vehicle. Here, pose information is a concept that includes the vehicle's position along with its direction (roll, pitch, yaw). This information enables the autonomous driving system to accurately perceive the environment, plan a safe path, and drive.

[0077] Lidar sensors use laser pulses to generate a 3D point cloud of the surrounding environment. This data has very high resolution and can accurately measure the distance and shape of surrounding objects. Lidar sensor data plays a crucial role in pose estimation for autonomous vehicles.

[0078] The precision map is high-precision 3D map data that includes information such as the shape of the road, buildings, and obstacles. The pose estimation unit (310) estimates the current position and attitude of the vehicle by comparing real-time point cloud data collected from the LiDAR sensor with the precision map. This process is called mapping or matching.

[0079] The twist estimation unit (320) can estimate twist information for the vehicle by utilizing information from the vehicle's internal sensors and inertial measurement unit (IMU) to estimate at least one of the vehicle's speed, angular velocity, acceleration, or angular acceleration.

[0080] The twist estimation unit (320) accurately determines the current speed and direction of the vehicle and allows it to follow a given path.

[0081] By utilizing acceleration and angular acceleration information, it prevents safety issues caused by sudden acceleration or turning, maintains driving stability by monitoring the vehicle's dynamic state in real time, and enables more precise environmental perception by combining with data from other sensors.

[0082] Additionally, the twist estimation unit (320) can collect acceleration and angular velocity data in real time from the vehicle's internal sensors and IMU, filter the collected data, and remove noise to extract accurate values.

[0083] In addition, the twist estimation unit (320) can calculate the vehicle's speed, angular velocity, acceleration, and angular acceleration using filtered data, and integrate the estimated twist information with other modules of the autonomous driving system to comprehensively determine the vehicle's dynamic state.

[0084] The pose-twist fusion filter section (330) can fuse the estimated pose information and the twist information to output the most likely position, velocity, acceleration, and covariance.

[0085] The pose-twist fusion filter section (330) estimates the dynamic state of the vehicle in real time to support the autonomous driving system in making accurate driving decisions.

[0086] The pose-twist fusion filter section (330) integrates the estimated pose information to calculate the accurate position and attitude of the vehicle. The pose information can usually be estimated based on external environment sensors such as a lidar sensor.

[0087] The pose-twist fusion filter section (330) can calculate the dynamic state of the vehicle by integrating estimated twist information (velocity, angular velocity, acceleration, etc.). The twist information is mainly estimated based on internal vehicle sensors and IMUs.

[0088] The pose-twist fusion filter section (330) combines pose information and twist information to output the most probable position, velocity, acceleration, and covariance. This aims to represent the current state of the vehicle as accurately as possible.

[0089] The pose-twist fusion filter unit (330) processes data using various fusion filter algorithms. Representative examples include the Kalman Filter and the Extended Kalman Filter (EKF). These algorithms estimate the state of the vehicle based on real-time incoming data and increase accuracy by minimizing prediction errors.

[0090] The Kalman filter is used in linear systems and assumes a linear relationship between pose and twist information, while the extended Kalman filter can be used in nonlinear systems and effectively fuses pose and twist information using nonlinear functions.

[0091] For example, the pose estimation unit (310) can estimate the position of the vehicle using the Normal Distributions Transform (NDT) algorithm and estimate the pose information based on the estimated position.

[0092] The Normal Distributions Transform (NDT) algorithm can map given point cloud data in 3D space by transforming it into transformed normal distributions. This can be primarily used to process 3D point cloud data collected from LiDAR sensors.

[0093] The Normal Distributions Transform (NDT) algorithm enables precise position estimation using high-resolution LiDAR data. In particular, it is suitable for real-time applications due to its fast computation speed, has robust processing capabilities against noise, and can respond to environmental changes. Additionally, the pose estimation unit (310) can receive 3D point cloud data collected from the LiDAR sensor as input.

[0094] The pose estimation unit (310) maps the input point cloud data into a normal distribution using an NDT algorithm. In this process, a map is generated that reflects the structure of the surrounding environment and the location of obstacles.

[0095] The pose estimation unit (310) estimates the current position of the vehicle based on the NDT mapping results, and can accurately determine the position of the vehicle by finding a point of agreement between the mapped data and the existing map using an NDT algorithm.

[0096] The pose estimation unit (310) estimates the pose information (direction, attitude) of the vehicle based on the estimated position through additional calculations. In this process, precise pose information is provided by considering the vehicle's movement path and rotation angle.

[0097] FIG. 4 is a drawing that explains the integrated processing unit according to one embodiment in more detail.

[0098] An integrated processing unit (120) according to one embodiment may include a three-dimensional object detection unit (121) and a test scenario control unit (122).

[0099] The 3D object detection unit (121) can detect real-world data in 3D space using a deep learning-based 3D object detection algorithm.

[0100] In most cases, neural network architectures are based on Convolutional Neural Networks (CNNs). This includes methods capable of processing 3D point cloud data rather than 2D images.

[0101] The 3D object detection unit (121) uses 3D point cloud data or other depth information collected from a LiDAR sensor as input. This data accurately reflects the shape and distance of objects and obstacles in the real world.

[0102] Networks analyze input data to identify the location, size, and shape of objects within a given space. This is typically represented by the object's bounding box, and furthermore, it can classify the object's category.

[0103] The 3D object detection unit (121) receives 3D data collected from a LiDAR sensor or other depth sensor in the surrounding environment as input.

[0104] The test scenario control unit (122) can modify the test scenario in real time based on interaction with the actual vehicle so that information generated in the virtual environment reflects and adapts to the conditions of the actual environment.

[0105] Integrating real-world data collected from sensors of autonomous vehicles into a virtual environment presupposes the ability to precisely detect objects such as vehicles or pedestrians in three-dimensional space. This is because determining the precise location and size of objects in three dimensions is crucial. Therefore, it is advantageous to use LiDAR sensors, which can detect objects further away than cameras and provide more precise three-dimensional distance information.

[0106] In the present invention, a deep learning-based 3D object detection algorithm can be applied to classify objects from a LiDAR sensor and identify their locations.

[0107] When integrating real-world data collected from autonomous vehicles into a virtual environment, there is a possibility of conflict between the two environments. The virtual environment must be designed to dynamically interact with elements of the real environment, ensuring that information generated within the virtual environment accurately reflects and adapts to the conditions of the real environment. To address this, test scenarios can be modified in real-time based on interaction with the actual vehicle.

[0108] An Intelligent Driver Model (IDM) can be applied to facilitate synchronization between virtual and real objects.

[0109] IDM is a vehicle-following model that approximates the behavior of a skilled driver and integrates their reactions; in this invention, it can be used to model the behavior and interactions of a virtual vehicle. VILS simulation can more accurately reproduce vehicle behavior patterns under actual traffic conditions, thereby enhancing the predictive capabilities of the simulation.

[0110] Figure 5 is a diagram illustrating the results of three-dimensional object detection.

[0111] In the experiment related to the present invention, a VILS test environment was established by integrating a vehicle equipped with LiDAR, GPS, and IMU sensors with an autonomous driving simulator, and tests were conducted within the DGIST campus. As a VILS test scenario combining virtual and real environments, a scenario was generated in which a vehicle drives rapidly behind a virtual lane while performing evasive driving by changing lanes in a situation where a real vehicle is stopped ahead.

[0112] In this scenario, the autonomous driving software recognized the presence of a real vehicle and two virtual vehicles, and it was confirmed to wait for the virtual vehicles to pass before performing a lane change. Through this test, complex and realistic interactions can be simulated under controlled conditions, enhancing test safety and enabling the valuable collection of behavioral data from autonomous systems.

[0113] Currently, in the development and verification of autonomous vehicles, it is necessary to safely and effectively reproduce complex real-world driving environments. This invention aims to solve this problem through the organic integration of virtual and real environments, and provides an improved VILS technology that combines virtual and real environments in this process.

[0114] In particular, to synchronize between the autonomous driving simulator and the actual vehicle, data is integrated in real time by utilizing 3D object detection technology and IDM, a representative longitudinal technology of traffic models.

[0115] The present invention enhances the flexibility of virtual simulations and expands their scope of application, enabling the effective mimicry and response to complex driving scenarios encountered on public roads. Furthermore, this technology minimizes the risks associated with actual vehicle testing while simultaneously expanding the scope and depth of testing.

[0116] FIG. 6 is a flowchart illustrating the operation method of a VILS-based test system for an autonomous vehicle according to one embodiment.

[0117] A method of operation of a VILS-based test system for an autonomous vehicle according to one embodiment can estimate the current position of the vehicle based on information collected from a GPS signal receiver, an in-vehicle sensor, and a LiDAR sensor (step 601).

[0118] By using the Normal Distributions Transform (NDT) algorithm, the position of the vehicle can be estimated, and the pose information can be estimated based on the estimated position.

[0119] A method of operation of a VILS-based test system for an autonomous vehicle according to one embodiment can estimate at least one of the vehicle's speed, angular velocity, acceleration, or angular acceleration by utilizing information from internal vehicle sensors and an inertial measurement unit (IMU) (step 602).

[0120] A method of operation of a VILS-based test system for an autonomous vehicle according to one embodiment can fuse the estimated pose information and the twist information to output the most probable position, velocity, acceleration, and covariance (step 603).

[0121] The method of operation of a VILS-based test system for an autonomous vehicle according to one embodiment may apply an Intelligent Driver Model (IDM) that approximates the driver's behavior and integrates the response (step 604).

[0122] The operation method of a VILS-based test system for an autonomous vehicle according to one embodiment can detect real-world data in a three-dimensional space using a deep learning-based 3D object detection algorithm (step 605).

[0123] A method of operation of a VILS-based test system for an autonomous vehicle according to one embodiment can modify a test scenario in real time based on interaction with a real vehicle so that information generated in a virtual environment reflects and adapts to the conditions of the actual environment (step 606).

[0124] Ultimately, by utilizing the present invention, it is possible to overcome the limitations of existing VILS and provide a new technology that enables more effective simulation in realistic environments.

[0125] Furthermore, it can provide technology that organically integrates virtual and real-world environmental elements within the VILS system and accurately simulates and analyzes the response of autonomous vehicles under various external environmental conditions, and enables VILS testing in real road environments by fusing real-world surrounding vehicle information with surrounding vehicle information generated in the virtual environment.

[0126] Furthermore, when virtual and real environments are mixed, the possibility of conflicts between the two environments can be reduced.

[0128] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0129] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0130] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0131] Although the embodiments have been described above with reference to the limited drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0132] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

Claim 1 A VILS-based test system for an autonomous vehicle, comprising: a sensor fusion precision positioning unit that estimates the current position of a vehicle based on information collected from a GPS signal receiver, an in-vehicle sensor, and a LiDAR sensor; and an integrated processing unit that integrates real-world data collected from the sensors of an autonomous vehicle into a virtual environment of an autonomous driving simulator by applying an Intelligent Driver Model (IDM) that approximates the driver's behavior and integrates the response, wherein the integrated processing unit processes real-world object information collected from the sensors of the autonomous vehicle, inputs the object information into the Intelligent Driver Model to model the behavior of a virtual vehicle, reflects the modeled behavior information of the virtual vehicle in the autonomous driving simulator, and modifies a test scenario of the virtual environment in real time based on the interaction between the actual vehicle and the virtual vehicle. Claim 2 A VILS-based test system for an autonomous vehicle according to claim 1, wherein the sensor fusion precision positioning unit comprises: a pose estimation unit that estimates pose information of a vehicle by matching it with a precision map based on LiDAR sensor data; a twist estimation unit that estimates twist information for a vehicle by estimating at least one of the vehicle's speed, angular velocity, acceleration, or angular acceleration using internal vehicle sensors and inertial measurement unit (IMU) information; and a pose-twist fusion filter unit that fuses the estimated pose information and the twist information to output the most probable position, speed, acceleration, and covariance. Claim 3 A VILS-based test system for an autonomous vehicle according to claim 2, wherein the pose estimation unit estimates the position of the vehicle using a Normal Distributions Transform (NDT) algorithm and estimates the pose information based on the estimated position. Claim 4 A VILS-based test system for an autonomous vehicle according to claim 1, wherein the integrated processing unit comprises: a 3D object detection unit that detects real-world data in a 3D space using a deep learning-based 3D object detection algorithm; and a test scenario control unit that modifies a test scenario in real time based on interaction with an actual vehicle so that information generated in a virtual environment reflects and adapts to the conditions of the actual environment. Claim 5 A method of operation of a VILS-based test system for an autonomous vehicle, characterized in that it includes: a step of estimating the current position of a vehicle based on information collected from a GPS signal receiver, an in-vehicle sensor, and a LiDAR sensor; and a step of integrating real-world data collected from the sensors of an autonomous vehicle into a virtual environment of an autonomous driving simulator by applying an Intelligent Driver Model (IDM) that approximates the driver's behavior and integrates the response, wherein the step of integrating real-world data collected from the sensors of an autonomous vehicle into a virtual environment of an autonomous driving simulator by applying the Intelligent Driver Model (IDM) that approximates the driver's behavior and integrates the response comprises: processing real-world object information collected from the sensors of the autonomous vehicle, inputting the object information into the Intelligent Driver Model to model the behavior of a virtual vehicle, reflecting the modeled behavior information of the virtual vehicle into the autonomous driving simulator, and modifying a test scenario of the virtual environment in real time based on the interaction between the actual vehicle and the virtual vehicle. Claim 6 A method of operation of a VILS-based test system for an autonomous vehicle, characterized in that, in claim 5, the step of estimating the current position of a vehicle based on information collected from the GPS signal receiver, the vehicle interior sensor, and the LiDAR sensor comprises: a step of estimating pose information of the vehicle by matching it with a precision map based on the LiDAR sensor data; a step of estimating twist information for the vehicle by utilizing the vehicle interior sensor and inertial measurement unit (IMU) information to estimate at least one of the vehicle's speed, angular velocity, acceleration, or angular acceleration; and a step of fusion of the estimated pose information and the twist information to output the most probable position, speed, acceleration, and covariance. Claim 7 A method of operation of a VILS-based test system for an autonomous vehicle, characterized in that, in claim 6, the step of estimating pose information of a vehicle by matching it with a precision map based on the lidar sensor data includes the step of estimating the position of the vehicle using a Normal Distributions Transform (NDT) algorithm and estimating the pose information based on the estimated position. Claim 8 delete

Images