Remote driving vehicle in-the-loop test system based on augmented reality technology

Abstract

The invention provides a remote driving vehicle in-loop test system based on an augmented reality technology. The remote driving vehicle in-loop test system comprises an augmented reality module, a hardware in-loop simulation module, a self-adaptive scene generation module, a data analysis module, a remote storage and cloud computing module and an automatic report generation module. According to the system, virtual information is superposed in a real driving environment through augmented reality equipment, so that a driver can perform remote driving test in a mixed reality environment, and real-time interaction of vehicle control instructions is realized in combination with hardware-in-the-loop simulation. The adaptive scene generation module can dynamically adjust a test environment according to sensor data, and simulate different traffic conditions and emergencies. The system can be widely applied to remote driving, automatic driving testing and intelligent traffic system research and development, a real, efficient and safe testing environment is provided, and the reliability and adaptability of a remote driving system are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of automatic control and simulation testing, and particularly to a remote driving vehicle-in-the-loop test system based on augmented reality technology. Background Art

[0002] Remote driving technology has developed rapidly in recent years and is widely used in fields such as unmanned delivery, autonomous driving taxis, and special operation vehicles. However, in the development and verification process of remote driving systems, testing is a key link. Existing testing methods mainly include: 1. Real environment testing: directly test the performance and safety of remote driving vehicles on actual roads. However, this method is costly, has high safety risks, and is limited by the testing environment, making it difficult to simulate all possible driving scenarios, such as extreme weather and sudden obstacles.

[0003] 2. Pure software simulation testing: such as virtual simulation testing based on software such as CarSim, MATLAB / Simulink, and Unity 3D. Although it can provide a certain testing environment, it lacks interaction with real vehicle controllers, and the driver's operation feedback is also relatively limited, resulting in a large deviation between the test results and the actual situation.

[0004] 3. Hardware-in-the-loop (VIL) simulation testing: by connecting the remote driving controller (ECU, TCU, etc.) to the simulation system to achieve real-time interaction of vehicle control signals. However, existing VIL testing technologies usually only focus on the response of the control system, making it difficult to provide an immersive testing experience for drivers and unable to dynamically adjust complex driving environments.

[0005] Therefore, the existing technology still needs to be improved. Summary of the Invention

[0006] In view of the above deficiencies of the existing technology, the present invention proposes a remote driving vehicle-in-the-loop test system based on augmented reality, which combines AR, VIL simulation, AI analysis, and cloud computing, can not only provide an immersive testing experience, but also simulate complex dynamic scenarios, and uses AI to analyze driving behaviors to improve the safety and stability of remote driving systems.

[0007] The technical solution of the present invention is as follows: A remote driving vehicle-in-the-loop test system based on augmented reality technology, the system includes: An augmented reality module, configured to superimpose virtual environment information in the driver's field of view and adjust the display content in combination with the real-time scene; A hardware-in-the-loop simulation module, configured to access the controller and sensors of the remote driving vehicle, generate real-time dynamic data, and perform data interaction with the augmented reality module; An adaptive scenario generation module, which is used to dynamically adjust the test scenario according to the road environment, weather conditions and vehicle status, and trigger preset test events; A data analysis module, which is used to analyze driving behaviors based on machine learning algorithms and optimize the control strategy of the remote driving system; A remote storage and cloud computing module, which is used to store test data and perform multi-device collaborative simulation through cloud computing.

[0008] In one embodiment, the augmented reality module includes an AR helmet or a transparent display screen, and combines driver's line-of-sight tracking and environmental perception algorithms to realize a holographic augmented reality test environment.

[0009] In one embodiment, the hardware-in-the-loop simulation module includes a vehicle dynamics model, a sensor simulation interface and a vehicle control unit, which are used to simulate real driving behaviors and perform real-time data feedback.

[0010] In one embodiment, the adaptive scenario generation module can adjust the test environment in real time based on camera and lidar sensor data to simulate complex driving scenarios.

[0011] In one embodiment, the data analysis module performs deep learning training based on driver operation data to optimize the decision-making ability of the remote driving system.

[0012] In one embodiment, the remote storage and cloud computing module supports cross-regional collaborative testing and provides real-time data playback and in-depth analysis functions.

[0013] In one embodiment, the system supports real-time environment reconstruction, collects real road data through cameras, radars and other sensors, and reproduces the actual test scenario in the augmented reality environment.

[0014] In one embodiment, the test event triggering mechanism includes intelligent triggering based on environmental perception.

[0015] In one embodiment, the system integrates an automatic report generation module, which automatically generates a performance evaluation report based on driver behavior and vehicle response data and provides visual analysis.

[0016] In one embodiment, the system is applicable to the safety evaluation of driverless vehicles, assisted driving systems and remote driving systems, and supports compatibility testing of different vehicle models and sensor configurations.

[0017] In summary, the proposed in-the-loop test system for remotely driven vehicles based on augmented reality technology can superimpose virtual information in a real driving environment, enabling drivers to conduct remote driving tests in a mixed reality environment. Combined with hardware-in-the-loop simulation, the virtual test environment can dynamically respond to real vehicle control commands. At the same time, AI is introduced to analyze the driver's operation behavior, optimize the control strategy of the remote driving system, and achieve cross-regional collaborative testing through cloud computing technology.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. Enhance the authenticity of testing and overcome the limitations of pure software simulation testing; Although existing pure software simulations (such as CarSim, MATLAB / Simulink) can simulate traffic environments, they lack the real feelings of drivers and cannot provide an immersive experience; The present invention adopts augmented reality technology (AR) to superimpose virtual traffic elements in the real environment, enabling test drivers to see and perceive information such as virtual vehicles, pedestrians, and traffic signals, making the test closer to actual driving conditions; 2. Dynamic test environment, capable of simulating various complex scenarios; Existing hardware-in-the-loop (VIL) test environments are usually fixed scenarios and are difficult to simulate emergencies or complex road conditions.

[0019] The present invention can dynamically adjust the test environment based on sensor data through an adaptive scenario generation module, for example, simulating rainy days, fog, night driving, sudden obstacles, etc., which is applicable to different remote driving applications; 3. Stronger interactivity and driver behavior feedback; Traditional hardware-in-the-loop (VIL) testing mainly focuses on signal interaction of remote driving controllers and lacks real-time feedback on driver operation behavior; The present invention combines augmented reality + driver gaze tracking + gesture recognition technologies, which can record the driver's operation habits, reaction time, and driving behavior, optimize the remote driving control strategy, and improve the intelligent level of the system; 4. AI-driven data analysis to optimize test efficiency; Traditional testing methods rely on manual analysis of driving data, which is time-consuming and inefficient and difficult to quickly optimize the remote driving system; The present invention integrates an AI-driven data analysis module, which can automatically analyze driver operation data (such as throttle, brake, steering angle, etc.), evaluate the system response ability, automatically optimize the control strategy, and enhance the safety and stability of the remote driving system; 5. Support remote collaborative testing and improve R & D efficiency; Traditional testing systems usually rely on local devices and are difficult to conduct remote collaborative testing; Through the remote storage and cloud computing module, the present invention realizes the cloud storage and multi-device synchronization of test data, supports remote team collaborative testing, and improves testing efficiency; 6. Automatically generate test reports and reduce manual intervention; Existing test methods require engineers to manually organize data and generate test reports, which is time-consuming and error-prone; The automatic report generation module of the present invention can automatically generate performance evaluation reports and driving behavior analysis reports of the remote driving system based on test data, improve data processing efficiency, and reduce manual analysis costs; 7. Reduce test costs and improve safety; Real road testing is costly and there is a risk of traffic accidents; pure software simulation testing cannot truly simulate the physical responses of the remote driving controller; The present invention combines augmented reality + VIL simulation to achieve high-fidelity remote driving testing without real road testing, which not only reduces test costs but also avoids the safety risks of real-world testing. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The present invention will be further described below in conjunction with the drawings and embodiments. In the drawings: Figure 1 is a block diagram of a remote driving vehicle-in-the-loop test system based on augmented reality technology of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0021] To make the objectives, technical solutions and effects of the present invention clearer and more definite, the present invention will be further described in detail below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. The embodiments of the present invention will be introduced below in conjunction with the drawings.

[0022] The remote driving vehicle-in-the-loop test system based on augmented reality technology provided in this embodiment, please refer to Figure 1 , the system includes the following modules: The augmented reality module 1, specifically, includes AR display devices (such as AR helmets, transparent OLED screens), interaction devices (such as touch panels, gesture recognition sensors), etc., to realize the integration of the virtual traffic environment and the real driving environment, for superimposing virtual environment information in the driver's field of view and adjusting the display content in combination with the real-time scene.

[0023] The hardware-in-the-loop simulation module 2 includes a remote driving controller, a virtual vehicle model, and a sensor simulation system, which can perform high-precision simulation and real-time feedback on remote driving operations, for accessing the controller and sensors of the remote driving vehicle, generating real-time dynamic data, and performing data interaction with the augmented reality module; The adaptive scenario generation module 3 dynamically adjusts the test scenario according to the road environment, weather conditions and vehicle status, and triggers preset test events, such as weather, road conditions, sudden traffic events, etc.; The data analysis module 4 analyzes driving behaviors based on machine learning algorithms, optimizes the control strategy of the remote driving system, analyzes the stability of the remote driving system, and provides optimization suggestions; The remote storage and cloud computing module 5 is used to store test data and perform multi-device collaborative simulation through cloud computing, supporting multi-team data sharing.

[0024] This system adopts a distributed computing architecture, has strong scalability, and can adapt to remote driving test requirements of different scales.

[0025] In a further embodiment, the augmented reality module of the present invention superimposes virtual traffic elements in the driver's field of view through an AR helmet, a transparent OLED display or a projection device, enabling the driver to conduct remote driving tests in the case of the fusion of the real environment + virtual information.

[0026] The specific implementation steps are as follows: A1. Real-time collect test environment data through an environment perception system (camera, lidar); A2. Generate virtual traffic elements, such as vehicles, pedestrians, traffic lights, etc. through an augmented reality engine (such as Unity, Unreal Engine); A3. Project virtual elements onto an AR helmet or a transparent display through optical overlay technology, enabling the driver to see virtual test content in the real environment; A4. Combine technologies such as voice interaction and gesture recognition, enabling the driver to control the virtual test scenario through voice or gesture, such as adjusting traffic density, triggering emergencies, etc.

[0027] In a further embodiment, the hardware-in-the-loop (VIL) simulation module of the present invention is used to simulate the dynamic characteristics of a real vehicle, the response of a remote driving controller (ECU / TCU), and support real-time feedback of remote driving operations.

[0028] The specific implementation steps are as follows: B1. Remote driving controller simulation: Connect the ECU / TCU of the remote driving system to the simulation system through a hardware-in-the-loop (HIL) simulation platform to achieve the input / output of remote driving signals.

[0029] Transmit control instructions (steering, braking, accelerating, etc.) in real time through the CAN bus or Ethernet.

[0030] B2. Sensor simulation: Adopt virtual sensor models (such as cameras, radars, ultrasonic sensors, etc.) to simulate the sensor signal input in a real environment and improve the authenticity of the test.

[0031] B3. Virtual vehicle dynamics model: Adopt vehicle dynamics simulation software (such as CarSim, Simulink) to simulate the dynamic responses of the vehicle under different working conditions.

[0032] Through this module, the remote driving system can be tested safely, efficiently and at low cost without relying on real test vehicles.

[0033] In a further embodiment, the adaptive scenario generation module provided by the present invention can dynamically adjust the test scenario according to sensor data and test requirements to achieve diverse remote driving tests.

[0034] The specific implementation steps are as follows: C1. Scenario initialization: Generate a basic test scenario through preset test parameters (such as weather, road type, traffic density).

[0035] C2. Environmental dynamic adjustment: Combine environmental perception data to adjust the virtual scenario in real time. For example, simulate the change from sunny to foggy, from day to night, etc. to test the adaptability of the remote driving system.

[0036] C3. Trigger of emergencies: Through AI algorithms, simulate emergencies such as traffic accidents, pedestrians crossing the road, and sudden braking of the vehicle in front, and observe the driver's operation reactions.

[0037] This module can be widely applied to scenarios such as remote driving safety tests and assessment of emergency response capabilities.

[0038] In a further embodiment, the present invention uses AI algorithms to deeply analyze remote driving test data, optimize driving strategies, and improve system stability.

[0039] The specific implementation steps are as follows: D1. Data collection: Record the operation data of the remote driver (throttle, brake, steering wheel angle).

[0040] Record the response data of the system (vehicle speed, acceleration, path deviation, etc.).

[0041] D2. Data analysis and pattern recognition: Adopt machine learning algorithms (such as decision trees, deep learning networks) to identify the operation patterns of the driver and analyze whether there are potential safety hazards.

[0042] Calculate the driver's reaction time and emergency operation frequency to judge the driver's adaptability to the remote driving system.

[0043] D3. Optimize the driving strategy: Combine the AI analysis results to adjust the control parameters (such as braking force, throttle response time) of the remote driving system to optimize the driving experience.

[0044] Through this module, the intelligent level of the remote driving system can be improved, and driving safety can be enhanced.

[0045] In a further embodiment, the system further includes: Implementation of the automatic report generation module; Specifically, the automatic report generation module of the present invention can automatically analyze the test data, generate a structured remote driving test report, reduce manual intervention, and improve the test efficiency.

[0046] The implementation steps are as follows: E1. Data collection: Collect all test data, including operation records, system responses, scenario changes, etc.

[0047] E2. Report generation: Adopt data visualization techniques (such as Matplotlib, Power BI) to generate driving behavior evaluation charts.

[0048] Adopt natural language processing (NLP) techniques to automatically write test reports, including key driving data, system stability evaluation, improvement suggestions, etc.

[0049] E3. Report output: Generate an automatic test report in PDF format, which is convenient for the remote driving system development team to consult and optimize the system parameters.

[0050] In summary, the present invention proposes a remote driving vehicle-in-the-loop test system based on augmented reality technology. Through multiple technological innovations such as augmented reality (AR) + vehicle-in-the-loop simulation (VIL) + AI data analysis + automatic test report generation, an efficient, realistic, and flexible remote driving test platform is constructed; this system breaks through the limitations of traditional remote driving test methods, provides a more reliable test environment, and ensures the safety, stability, and intelligent level of the remote driving system.

[0051] Compared with traditional remote driving test methods (such as real road tests, pure software simulation tests, traditional hardware-in-the-loop tests), the present invention has significant advantages in multiple aspects: The present invention can be widely applied to the following fields: 1. Remote driving system testing and optimization: It can be used for the testing and optimization of systems such as unmanned delivery vehicles, autonomous driving taxis, and remote driving mining trucks.

[0052] 2. Autopilot algorithm verification: It can provide a high-fidelity and high-security simulation test environment for Advanced Driver Assistance Systems (ADAS).

[0053] 3. Intelligent transportation system research: It supports the testing and evaluation of emerging technologies such as intelligent traffic signal control, vehicle-road cooperation, and vehicle-to-everything (V2X).

[0054] 4. Driver training and evaluation: Through the AR simulation environment, it can be used for driver training and driving behavior evaluation to improve driving safety.

[0055] The present invention combines a number of cutting-edge technologies such as augmented reality (AR), vehicle-in-the-loop simulation (VIL), AI data analysis, and automated test report generation, breaking through the limitations of traditional testing methods and realizing a remote driving test solution with high fidelity, high security, low cost, and high-efficiency automation. This system can effectively improve the R & D efficiency of remote driving and autonomous driving technologies and provide solid technical support for the development of future intelligent transportation.

[0056] It should be understood that the application of the present invention is not limited to the above examples. For those of ordinary skill in the art, improvements or transformations can be made according to the above description, and all such improvements and transformations should fall within the protection scope of the appended claims of the present invention.

Claims

1. A remote driving vehicle-in-the-loop test system based on augmented reality technology, characterized in that, The system includes: An augmented reality module for overlaying virtual environment information in the driver's field of view and adjusting the display content in combination with the real-time scene; A hardware-in-the-loop simulation module for accessing the controller and sensors of a remotely driven vehicle, generating real-time dynamic data, and performing data interaction with the augmented reality module; An adaptive scenario generation module for dynamically adjusting the test scenario according to the road environment, weather conditions, and vehicle state, and triggering preset test events; A data analysis module for analyzing driving behavior based on machine learning algorithms and optimizing the control strategy of the remotely driven system; A remote storage and cloud computing module for storing test data and performing multi-device collaborative simulation through cloud computing.

2. The in-the-loop test system for remotely driven vehicles based on augmented reality technology according to claim 1, characterized in that, The augmented reality module includes an AR helmet or a transparent display screen, and combines driver gaze tracking and environment perception algorithms to implement a holographic augmented reality test environment.

3. The in-the-loop test system for remotely driven vehicles based on augmented reality technology according to claim 1, wherein The hardware-in-the-loop simulation module includes a vehicle dynamics model, a sensor simulation interface, and a vehicle control unit for simulating real driving behavior and providing real-time data feedback.

4. The in-the-loop test system for remotely driven vehicles based on augmented reality technology according to claim 1, characterized in that, The adaptive scenario generation module can adjust the test environment in real time based on camera and lidar sensor data to simulate complex driving scenarios.

5. The in-the-loop test system for remotely driven vehicles based on augmented reality technology according to claim 1, wherein The data analysis module performs deep learning training based on driver operation data to optimize the decision-making ability of the remotely driven system.

6. The in-the-loop test system for remotely driven vehicles based on augmented reality technology according to claim 1, characterized in that, The remote storage and cloud computing module supports cross-regional collaborative testing and provides real-time data playback and in-depth analysis functions.

7. The in-the-loop test system for remotely driven vehicles based on augmented reality technology according to claim 1, wherein The system supports real-time environment reconstruction, collects real road data through cameras, radars, and other sensors, and reproduces the actual test scenario in the augmented reality environment.

8. The in-the-loop test system for remotely driven vehicles based on augmented reality technology according to claim 1, characterized in that, The test event triggering mechanism includes intelligent triggering based on environment perception.

9. The in-the-loop test system for remotely driven vehicles based on augmented reality technology according to claim 1, characterized in that, The system integrates an automatic report generation module that automatically generates a performance evaluation report based on driver behavior and vehicle response data and provides visual analysis.

10. The in-the-loop test system for remotely driven vehicles based on augmented reality technology according to claim 1, characterized in that, The system is applicable to the safety assessment of driverless vehicles, assisted driving systems, and remotely driven systems, and supports compatibility testing of different vehicle models and sensor configurations.

Images