High-immersion man-machine interaction virtual driving vehicle test system

By integrating virtual reality and multi-sensor data fusion in the virtual driving vehicle test system, combining force feedback and vibration feedback equipment to simulate real driving scenarios, the problem of missing human-computer interaction system in the virtual driving vehicle test is solved and the vehicle reliability is improved and user experience promote the commercial application of technology.

CN119065345APending Publication Date: 2024-12-03ZHEJIANG KECHI INTELLIGENT TECH CO LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411095822.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-12
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The existing virtual driving vehicle testing system lacks sufficient attention to human-computer interaction systems, affects vehicle reliability and passenger experience, and is difficult to provide effective support in emergencies.

Method used

A highly immersive human-computer interactive virtual driving vehicle testing system is designed, including virtual reality module, database input module, human-computer interaction module and control system module. Through high-resolution display screen, multi-sensor data fusion, force feedback and Vibration feedback equipment, etc., simulate real driving scenarios, evaluate interaction design and user experience.

Benefits of technology

It improves the reliability and safety of virtual driving vehicles, improves user experience, enhances market acceptance, and promotes the commercialization and popularization of technology.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119065345A_ABST
    Figure CN119065345A_ABST
Patent Text Reader

Abstract

The invention relates to the related field of virtual driving, in particular to a high-immersion man-machine interaction virtual driving vehicle test system, which can help to find and correct design defects possibly causing safety problems by simulating various interaction scenes including emergency and non-standard operations. The comprehensive test ensures that the behavior of the virtual driving vehicle in the actual real world is safer and more reliable; through highly real interactive simulation, a designer can better understand user demands and preferences; the virtual reality and augmented reality technology is utilized, the testing and iteration process can be carried out under the condition that no physical prototype exists, the development time and cost are greatly reduced, and meanwhile high-standard performance and reliability are kept; by simulating various potential use situations, a designer can identify and cope with problems in the design in an early stage, and the prospective test method enables a final product to more flexibly adapt to future technology and market changes.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field related to virtual driving, and in particular to a highly immersive human-computer interactive virtual driving vehicle testing system. Background Art

[0002] In the current field of virtual driving vehicle testing, the focus is mainly on evaluating the vehicle's hardware performance and software functions, such as sensor accuracy, data processing speed, and the efficiency of automated control systems. Despite this, the human-machine interaction (HMI) system between the vehicle and the driver or passenger often does not receive sufficient attention. This system is a key factor in improving vehicle reliability and passenger experience.

[0003] The human-machine interaction system covers all technologies that enable users to interact with the vehicle, including touch interfaces, voice command systems, and safety warning mechanisms. An efficient human-machine interaction system can not only enhance passengers' sense of security, but also enhance users' trust in autonomous driving technology and improve the riding experience. For example, an intuitive user interface can help users clearly understand the vehicle status and expected behavior, while an efficient voice interaction system can provide instant help when needed.

[0004] Therefore, it is crucial to develop a highly immersive human-computer interaction test system. This test system needs to be able to simulate various scenarios in real life, including routine operations and user reactions in emergency situations. Using virtual reality (VR) or augmented reality (AR) technology, a test environment that is both controlled and realistic can be created, allowing testers to fully evaluate whether the interaction design is humane and whether it can effectively provide support at critical moments.

[0005] In addition, an advanced human-computer interaction test system should also include a comprehensive evaluation of the user experience (UX). By analyzing the user's behavior, physiological reactions, and feedback in a simulated environment, the user's satisfaction and the ease of use of the interactive interface can be evaluated. This approach not only helps to identify and solve potential interaction problems, but also improves the overall reliability and market acceptance of virtual driving vehicles. The development and improvement of this test platform is of great significance to promoting the commercialization and popularization of virtual driving vehicle technology. Summary of the invention

[0006] The purpose of the present invention is to provide a highly immersive human-computer interactive virtual driving vehicle testing system to solve the problems raised in the above background technology.

[0007] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: a highly immersive human-computer interactive virtual driving vehicle testing system, comprising: a virtual reality module, a database input module, a human-computer interaction module, a control system module and a data recording and analysis module, wherein the virtual reality module is used to create a virtual driving environment, including several driving scenes such as urban roads, high-speed elevated highways and rural roads, and the virtual reality module uses a high-resolution display screen to achieve an immersive visual experience;

[0008] The database input module is used to combine the data in the database into the vehicle surrounding environment information and generate a detailed environment model;

[0009] Human-computer interaction module: including driver input devices, specifically steering wheel, pedals, touch screen; feedback devices, specifically force feedback system, vibration system, used to achieve interaction between the driver and the system;

[0010] Control system module: used to process the data of the virtual reality module and the multi-sensor fusion module, and to control the behavior of the vehicle in real time according to the input of the human-computer interaction module;

[0011] It is used to record the control system module and human-computer interaction module data during the test process, and analyze the data to optimize the human-computer interaction design and improve the performance of the virtual driving vehicle.

[0012] Preferably, the virtual reality module specifically includes the following contents:

[0013] a. The virtual reality module includes the following components: high-resolution display device: a high-definition giant screen display that provides realistic visual effects; computing device: a high-performance computer or server for generating and rendering virtual environments; audio output device: a high-fidelity sound system or headphones that provide three-dimensional spatial sound effects to enhance immersion;

[0014] b. The virtual reality module creates a virtual driving environment through the following steps: Scene modeling: Use 3D modeling software to create detailed driving scenes, including urban roads, high-speed elevated highways and rural roads. The scenes should include traffic signs, pedestrians, other vehicles and natural environment elements; Texture and lighting: Use high-resolution textures and advanced lighting technology to render the scene to enhance the visual authenticity and details; Physical simulation: Integrated physics engine, specifically using NVIDIA PhysX to simulate real physical phenomena such as collisions, friction and weather changes, to improve the interactivity and reliability of the environment;

[0015] c. The virtual driving environment includes a variety of driving scenarios to meet different testing needs, specifically: Urban roads: complex traffic conditions, including intersections, traffic lights, and pedestrian crossings; Expressways: lane changes, overtaking, and emergency braking at high speeds; Rural roads: narrow roads, sharp turns, and rugged road conditions.

[0016] Preferably, the database input module includes the following contents:

[0017] a. Data collection in the database consists of: LiDAR simulation data: used to generate high-precision 3D point cloud data; Radar simulation data: used to simulate the measurement of speed and distance data under harsh conditions; Camera simulation data: used to provide high-resolution visual information and identify color, texture and shape details, where visual information is collected by RGB camera data, depth camera data, and infrared camera data;

[0018] b. Data collection: LiDAR simulation data: Use high-frequency scanning to collect point cloud data to form a three-dimensional environment model. The specific scanning frequency is between 10Hz and 20Hz; Radar simulation data: Collect reflected signals to calculate the position, distance and relative speed of objects; Camera simulation data: Obtain depth information through stereo vision or structured light technology, capture two-dimensional images and video data for object recognition, area monitoring and auxiliary navigation;

[0019] c. Data processing:

[0020] Data preprocessing: LiDAR simulation data: filter out noise points and abnormal data to improve the quality of point cloud; Radar simulation data: signal processing and target detection, specifically through FFT to separate different objects; Camera simulation data: image preprocessing, specifically through denoising, white balance and image enhancement;

[0021] Synchronization and calibration: Time synchronization: ensure that the laser radar simulation data, radar simulation data and camera simulation data are on the same time basis; Space calibration: perform spatial calibration on the laser radar simulation position, radar simulation position and camera simulation position based on the vehicle reference system to ensure the accuracy of data fusion; Calibration tool: use calibration plate and laser calibrator tool for precise calibration;

[0022] d. Data information fusion:

[0023] Fusion of database data: Use geometric transformation to project data from different sensors into the same coordinate system. Specifically, the point cloud data of the lidar is registered with the image data of the camera, and a high-precision 3D model is constructed through multi-frame fusion. The SLAM algorithm is used for real-time mapping and positioning.

[0024] Data structure management: Point cloud data processing: Use the octree data structure to quickly process lidar point cloud data to achieve object detection and scene understanding; Image processing: Use convolutional neural networks to extract camera image features and identify targets; Radar simulation data processing: Process radar frequency domain and time domain data to detect and track moving targets;

[0025] e. Environmental model generation: A two-dimensional grid map formed based on camera simulation data and radar simulation data is used to form a two-dimensional environmental model for path planning on the plane; a three-dimensional model is generated based on the point cloud data of the lidar and the depth information of the camera, and an oxygen grid map and a digital elevation map are created to improve the accuracy of environmental perception.

[0026] Preferably, the specific steps of registering the point cloud data of the laser radar with the image of the camera in the fusion of the database data are as follows:

[0027] Step S1, camera intrinsic parameter calibration: Use the calibration board to take multiple pictures, calibrate through the OpenCV tool, and obtain the camera intrinsic parameter matrix K and distortion coefficient \mathbf{d}. The specific formula is as follows:

[0028]

[0029] Where (f x , f y ) is the focal length, (c x , c y ) are the principal point coordinates;

[0030] Step S2, laser radar and camera extrinsic calibration: by registering the calibration plate, simultaneously obtain the point cloud and image data, perform manual or automatic registration, and calculate the extrinsic transformation matrix \mathbf{R} and translation vector \mathbf{t} between the two. The specific calculation formula is as follows:

[0031]

[0032] Step S3, point cloud data processing: remove outliers and noise points in the point cloud by statistical filtering and radius filtering methods; use voxel filtering method to reduce the point cloud data density and improve processing efficiency;

[0033] Step S4: Project the processed LiDAR point cloud data onto the image plane of the camera, where the point cloud coordinates (\mathbf{P}{LiDAR}{pmatrix}X {LiDAR} &y {LiDAR} &z {LiDAR} &1{pmatrix} T ) represents a point in the laser radar coordinate system, specifically:

[0034] Use the external parameter matrix \mathbf{T} to convert the laser radar point cloud coordinates into the camera coordinate system \mathbf{P}{Cam}:

[0035] [\mathbf{P}{Cam}=\mathbf{T}·\mathbf{P}{Cam}]

[0036] Projected onto the image plane, use the camera intrinsic parameter matrix K: [\mathbf{p}{img}=\mathbf{K}·\mathbf{P}{Cam}], where (\mathbf{P}{Cam}{pmatrix}X {Cam} &Y {Cam} &Z {Cam} {pmatrix} T );

[0037] Finally, the homogeneous coordinates are converted to image pixel coordinates (u, v) and normalized: [u = \frac{X {Cam}}{Z {Cam}}, \quad v = \frac{Y {Cam}}{Z {Cam}}];

[0038] Step S5, draw the projection point in the image: according to the pixel coordinates (u, v) obtained in the previous step, draw the corresponding point cloud point on the image, color-code the depth or intensity in the point cloud, superimpose it on the image to enhance the display effect, then obtain the RGB value of the point from the image, and assign the corresponding point color of the point cloud to achieve color display of the point cloud.

[0039] Preferably, the human-computer interaction module specifically includes the following contents:

[0040] Driver input device: Steering wheel: used to control the steering of the vehicle, wherein the steering wheel is provided with an angle sensor, an electronic power steering system and a force feedback device. Specifically, the steering wheel rotation angle is obtained through the angle sensor; the electronic power steering system provides auxiliary steering force to reduce the driver's operation force; the force feedback motor transmits road information, such as collision and friction, to the driver, making the driving feel more realistic; Pedals: including accelerator pedals and brake pedals, used to control the acceleration and deceleration of the vehicle, wherein the pedals are provided with position sensors and wire control systems, the position sensors are used to detect the pedal position, and the wire control system is used to replace traditional mechanical control to improve the corresponding speed and accuracy; Touch screen: used for interactive display and user input;

[0041] Feedback device: Force feedback system: applies reverse torque to the steering wheel through a motor or actuator to simulate road feedback and handling stability, increases reaction force on the pedals, and provides a realistic pedaling feel; Vibration feedback system: provides vibration feedback through a vibration motor or actuator under the seat, and the steering wheel has a built-in vibration device to remind the driver to pay attention; Visual feedback: projects important navigation and speed information onto the windshield, so that the driver does not need to look down at the dashboard, and provides vehicle status and warning information through an LCD or OLED display; Auditory feedback: provides voice navigation, warnings and operation prompts through the on-board speakers, and emits warning sounds through a buzzer or speaker.

[0042] Preferably, the control system module specifically includes the following contents:

[0043] Data processing and integration: Virtual reality module data processing: convert the data of the virtual reality module into a format that can be recognized and processed by the vehicle control system; database input module data processing: synchronize the laser radar simulation data, radar simulation data and camera simulation data from the database to ensure time and space consistency, and use the Kalman filter algorithm to fuse the above synchronized data to improve data quality and accuracy;

[0044] Real-time control logic: Human-computer interaction data analysis: analyze the driver's operating intention expressed through the driver input device; decision-making and execution: based on the data collected by human-computer interaction data analysis, multi-sensor fusion module data and preset driving strategies, the computer algorithm determines the optimal driving plan for the vehicle;

[0045] Human-machine collaboration: Driver intention feedback: Through the dashboard and HUD, the recognition results of the real-time control logic and the operation intention are fed back to the driver in real time to ensure that the driver understands the vehicle status and the operations to be performed; Manual and automatic driving switching: Smooth switching between automatic driving and manual driving to ensure driving safety and comfort.

[0046] Preferably, the specific steps of the computer algorithm determining the optimal driving plan of the vehicle based on the data collected by human-computer interaction data analysis and the preset driving strategy are as follows:

[0047] Step A: First, obtain data from human-computer interaction data analysis, including vehicle position information, camera simulation data, radar simulation data, and lidar simulation data, as well as vehicle speed, acceleration, and steering wheel angle data;

[0048] Step B: Model the vehicle's surrounding environment using the data collected in step A, specifically using an extended Kalman filter for modeling:

[0049] [\hat{x}{t}=f(\hat{x}{t-1},u {t-1} )+w{t-1} ]

[0050] [z t =h(\hat{x} t )+v t ]

[0051] Among them, \hat{x}{t} is the current state estimate, u {t-1} is the control input at the previous moment, f is the state transfer function, w {t-1} is the process noise, z t is the current observation, h is the observation function, v t is the observation noise;

[0052] Step C, based on the environment model and the multi-sensor fusion module data of the vehicle, the optimal path from the current position of the vehicle to the target position is calculated by the RRT path planning algorithm;

[0053] Step D: After obtaining the best path, use the PID algorithm to generate specific control instructions to obtain the optimal driving plan. The specific algorithm formula is as follows:

[0054] [u(t) = K p e(t)+K i ∫e(t)dt+K d \frac{de(t){dt}}]

[0055] Where u(t) is the control input, e(t) is the speed error, K p , K i , K d is the gain of the PID controller.

[0056] Preferably, the present invention provides an electronic device, comprising: a processor and a memory, wherein the memory stores a computer program that can be called by the processor;

[0057] The processor calls the computer program stored in the memory to execute a highly immersive human-computer interactive virtual driving vehicle testing system.

[0058] Preferably, a computer program product stored on a computer-readable medium includes a computer-readable program, which, when executed on an electronic device, provides a user input interface to implement a highly immersive human-computer interactive virtual driving vehicle testing system.

[0059] Compared with the prior art, the present invention has the following beneficial effects:

[0060] 1. Improve vehicle reliability: By comprehensively testing the human-machine interaction system, potential interaction problems can be discovered and resolved, thereby improving the overall reliability and safety of virtual driving vehicles.

[0061] 2. Improve user experience: The optimized human-computer interaction system will make it easier for passengers to understand vehicle status and operation, enhance users' trust in technology, and thus improve the riding experience.

[0062] 3. Enhance market acceptance: An efficient and user-friendly human-computer interaction system will improve the market acceptance of virtual driving vehicles and promote their commercialization and popularization.

[0063] 4. Improve safety: Timely and effective interactive systems can provide necessary support and information in emergency situations, thereby enhancing passengers' sense of security and reducing the possibility of accidents.

[0064] 5. Promote technological innovation: Through comprehensive evaluation of user behavior and feedback, valuable feedback can be provided for interaction design, promoting the continuous innovation and development of virtual driving vehicle technology.

[0065] In general, the development of a human-computer interaction test system will help improve the overall performance and user experience of virtual driving vehicles, promote the development of this field and facilitate its commercial application. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is a schematic diagram of the system structure of the present invention;

[0067] Figure 2 A schematic diagram of a method for registering point cloud data of a laser radar with an image of a camera according to the present invention;

[0068] Figure 3 A schematic flow chart of a method for determining an optimal driving plan for a vehicle using a computer algorithm of the present invention;

[0069] Figure 4 A schematic diagram of the structure of an electronic device;

[0070] Figure 5 Schematic diagram of the system logic of the present invention. DETAILED DESCRIPTION

[0071] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0072] Example 1

[0073] See also Figure 1-3 and Figure 5, the present invention provides a technical solution: a highly immersive human-computer interactive virtual driving vehicle testing system, comprising: a virtual reality module, a database input module, a human-computer interaction module, a control system module and a data recording and analysis module, wherein the above modules are connected by wire or wireless means to realize data transmission between modules;

[0074] Virtual reality module: used to create a virtual driving environment, including urban roads, high-speed elevated highways, rural roads and several other driving scenarios. The virtual reality module uses a high-resolution display to achieve an immersive visual experience.

[0075] The virtual reality module specifically includes the following:

[0076] a. The virtual reality module includes the following components: high-resolution display device: a high-definition giant screen display that provides realistic visual effects; computing device: a high-performance computer or server for generating and rendering virtual environments; audio output device: a high-fidelity sound system or headphones that provide three-dimensional spatial sound effects to enhance immersion;

[0077] b. The virtual reality module creates a virtual driving environment through the following steps: Scene modeling: Use 3D modeling software to create detailed driving scenes, including urban roads, high-speed elevated highways and rural roads. The scenes should include traffic signs, pedestrians, other vehicles and natural environment elements; Texture and lighting: Use high-resolution textures and advanced lighting technology to render the scene to enhance the visual authenticity and details; Physical simulation: Integrated physics engine, specifically using NVIDIA PhysX to simulate real physical phenomena such as collisions, friction and weather changes, to improve the interactivity and reliability of the environment;

[0078] c. The virtual driving environment includes a variety of driving scenarios to meet different testing needs, specifically: Urban roads: complex traffic conditions, including intersections, traffic lights, and pedestrian crossings; Expressways: lane changes, overtaking, and emergency braking at high speeds; Rural roads: narrow roads, sharp turns, and rugged road conditions.

[0079] It should be noted that the virtual reality module is a key component that is used to create a variety of virtual driving environments to provide a realistic background for testing human-computer interaction systems, such as the following scenarios:

[0080] Urban road scene: This scene simulates busy urban traffic, including traffic lights, pedestrian crossings, bicycle lanes, bus stops, various commercial and residential buildings, etc. For example, on a city street during rush hour, vehicles need to stop and start frequently to deal with the sudden appearance of pedestrians, lane changes and congestion of other vehicles. The test system will evaluate how the human-computer interaction system helps users make decisions and operate under these complex traffic conditions;

[0081] High-speed elevated highway scenario: This scenario includes elements such as multi-lane high-speed elevated highways, traffic signs, exit ramps, bridges and tunnels, and is designed to simulate high-speed driving conditions. For example, when driving on a high-speed elevated highway, the vehicle needs to deal with high-speed lane changes, overtaking, road construction areas and severe weather conditions (such as heavy rain, smog). The test will evaluate the user's response to vehicle information and warnings at high speeds, and how the system helps the user maintain safe driving;

[0082] Rural road scenario: This scenario simulates the road environment in rural areas, including narrow two-lane roads, sharp turns, ramps, rural houses and farmland landscapes. For example, when driving on rural roads, you may encounter animals that suddenly appear, slow agricultural vehicles and uneven roads. The virtual reality module will test the vehicle's operational stability under these conditions and the performance of the human-computer interaction system in emergency situations;

[0083] Special event simulation: It can generate some emergencies, such as traffic accidents ahead, road closures, emergency vehicle traffic, etc. For example, a traffic accident suddenly occurs on a high-speed elevated highway, the lane ahead is closed, and the vehicle needs to respond quickly. The test will evaluate how the human-computer interaction system can provide information to users in a timely manner and assist in making correct driving decisions.

[0084] Multi-sensor fusion module: collects and processes information about the vehicle's surroundings through a variety of sensors and generates a detailed environmental model;

[0085] The database input module includes the following:

[0086] a. Data collection in the database consists of: LiDAR simulation data: used to generate high-precision 3D point cloud data; Radar simulation data: used to simulate the measurement of speed and distance data under harsh conditions; Camera simulation data: used to provide high-resolution visual information and identify color, texture and shape details, where visual information is collected by RGB camera data, depth camera data, and infrared camera data;

[0087] b. Data collection: LiDAR simulation data: Use high-frequency scanning to collect point cloud data to form a three-dimensional environment model. The specific scanning frequency is between 10Hz and 20Hz; Radar simulation data: Collect reflected signals to calculate the position, distance and relative speed of objects; Camera simulation data: Obtain depth information through stereo vision or structured light technology, capture two-dimensional images and video data for object recognition, area monitoring and auxiliary navigation;

[0088] c. Data processing:

[0089] Data preprocessing: LiDAR simulation data: filter out noise points and abnormal data to improve the quality of point cloud; Radar simulation data: signal processing and target detection, specifically through FFT to separate different objects; Camera simulation data: image preprocessing, specifically through denoising, white balance and image enhancement;

[0090] Synchronization and calibration: Time synchronization: ensure that the laser radar simulation data, radar simulation data and camera simulation data are on the same time basis; Space calibration: perform spatial calibration on the laser radar simulation position, radar simulation position and camera simulation position based on the vehicle reference system to ensure the accuracy of data fusion; Calibration tool: use calibration plate and laser calibrator tool for precise calibration;

[0091] d. Data information fusion:

[0092] Fusion of database data: Use geometric transformation to project data from different sensors into the same coordinate system. Specifically, the point cloud data of the lidar is registered with the image data of the camera, and a high-precision 3D model is constructed through multi-frame fusion. The SLAM algorithm is used for real-time mapping and positioning.

[0093] Data structure management: Point cloud data processing: Use the octree data structure to quickly process lidar point cloud data to achieve object detection and scene understanding; Image processing: Use convolutional neural networks to extract camera image features and identify targets; Radar simulation data processing: Process radar frequency domain and time domain data to detect and track moving targets;

[0094] e. Environmental model generation: A two-dimensional grid map formed based on camera simulation data and radar simulation data is used to form a two-dimensional environmental model for path planning on the plane; a three-dimensional model is generated based on the point cloud data of the lidar and the depth information of the camera, and an oxygen grid map and a digital elevation map are created to improve the accuracy of environmental perception.

[0095] The specific steps for registering the laser radar point cloud data with the camera image in the fusion of database data described above are as follows:

[0096] Step S1, camera intrinsic parameter calibration: Use the calibration board to take multiple pictures, calibrate through the OpenCV tool, and obtain the camera intrinsic parameter matrix K and distortion coefficient \mathbf{d}. The specific formula is as follows:

[0097]

[0098] Where (f x , f y ) is the focal length, (c x , c y ) are the principal point coordinates;

[0099] Step S2, laser radar and camera extrinsic calibration: by registering the calibration plate, simultaneously obtain the point cloud and image data, perform manual or automatic registration, and calculate the extrinsic transformation matrix \mathbf{R} and translation vector \mathbf{t} between the two. The specific calculation formula is as follows:

[0100]

[0101]

[0102] Step S3, point cloud data processing: remove outliers and noise points in the point cloud by statistical filtering and radius filtering methods; use voxel filtering method to reduce the point cloud data density and improve processing efficiency;

[0103] Step S4: Project the processed LiDAR point cloud data onto the image plane of the camera, where the point cloud coordinates (\mathbf{P}{LiDAR}{pmatrix}X {LiDAR} &y {LiDAR} &z {LiDAR} &1{pmatrix} T ) represents a point in the laser radar coordinate system, specifically:

[0104] Use the external parameter matrix \mathbf{T} to convert the laser radar point cloud coordinates into the camera coordinate system \mathbf{P}{Cam}:

[0105] [\mathbf{P}{Cam}=\mathbf{T}·\mathbf{P}{Cam}]

[0106] Projected onto the image plane, use the camera intrinsic parameter matrix K: [\mathbf{p}{img}=\mathbf{K}·\mathbf{P}{Cam}], where (\mathbf{P}{Cam}{pmatrix}X {Cam} &Y {Cam} &Z {Cam} {pmatrix} T );

[0107] Finally, the homogeneous coordinates are converted to image pixel coordinates (u, v) and normalized: [u = \frac{X {Cam}}{Z {Cam}}, \quad v = \frac{Y {Cam}}{Z {Cam}}];

[0108] Step S5, draw the projection point in the image: according to the pixel coordinates (u, v) obtained in the previous step, draw the corresponding point cloud point on the image, color-code the depth or intensity in the point cloud, superimpose it on the image to enhance the display effect, then obtain the RGB value of the point from the image, and assign the corresponding point color of the point cloud to achieve color display of the point cloud.

[0109] It should be noted that the lidar simulation data, radar simulation data and camera simulation data in the database input module are real environment data collected by lidar, radar and camera. The above collected data are stored in the database. When the system needs to combine the data into the vehicle's surrounding environment information and generate a detailed environmental model, it only needs to read the database data. Among them, the lidar simulation data and radar simulation data can better display the environmental depth of field effect on the high-definition giant screen display, making the simulated driving environment more realistic.

[0110] Human-computer interaction module: including driver input devices, specifically steering wheel, pedals, touch screen; feedback devices, specifically force feedback system, vibration system, used to achieve interaction between the driver and the system;

[0111] The human-computer interaction module specifically includes the following contents:

[0112] Driver input device: Steering wheel: used to control the steering of the vehicle, wherein the steering wheel is provided with an angle sensor, an electronic power steering system and a force feedback device. Specifically, the steering wheel rotation angle is obtained through the angle sensor; the electronic power steering system provides auxiliary steering force to reduce the driver's operation force; the force feedback motor transmits road information, such as collision and friction, to the driver, making the driving feel more realistic; Pedals: including accelerator pedals and brake pedals, used to control the acceleration and deceleration of the vehicle, wherein the pedals are provided with position sensors and wire control systems, the position sensors are used to detect the pedal position, and the wire control system is used to replace traditional mechanical control to improve the corresponding speed and accuracy; Touch screen: used for interactive display and user input;

[0113] Feedback device: Force feedback system: applies reverse torque to the steering wheel through a motor or actuator to simulate road feedback and handling stability, increases reaction force on the pedals, and provides a realistic pedaling feel; Vibration feedback system: provides vibration feedback through a vibration motor or actuator under the seat, and the steering wheel has a built-in vibration device to remind the driver to pay attention; Visual feedback: projects important navigation and speed information onto the windshield, so that the driver does not need to look down at the dashboard, and provides vehicle status and warning information through an LCD or OLED display; Auditory feedback: provides voice navigation, warnings and operation prompts through the on-board speakers, and emits warning sounds through a buzzer or speaker.

[0114] Control system module: used to process the data of the virtual reality module and the multi-sensor fusion module, and to control the behavior of the vehicle in real time according to the input of the human-computer interaction module;

[0115] The control system module specifically includes the following:

[0116] Data processing and integration: Virtual reality module data processing: convert the data of the virtual reality module into a format that can be recognized and processed by the vehicle control system; database input module data processing: synchronize the laser radar simulation data, radar simulation data and camera simulation data from the database to ensure time and space consistency, and use the Kalman filter algorithm to fuse the above synchronized data to improve data quality and accuracy;

[0117] Real-time control logic: Human-computer interaction data analysis: analyze the driver's operating intention expressed through the driver input device; decision-making and execution: based on the data collected by human-computer interaction data analysis, multi-sensor fusion module data and preset driving strategies, the computer algorithm determines the optimal driving plan for the vehicle;

[0118] Human-machine collaboration: Driver intention feedback: Through the dashboard and HUD, the recognition results of the real-time control logic and the operation intention are fed back to the driver in real time to ensure that the driver understands the vehicle status and the operations to be performed; Manual and automatic driving switching: Smooth switching between automatic driving and manual driving to ensure driving safety and comfort.

[0119] It is used to record the control system module and human-computer interaction module data during the test process, and analyze the data to optimize the human-computer interaction design and improve the performance of the virtual driving vehicle.

[0120] Based on the data collected by human-computer interaction data analysis and the preset driving strategy, the specific steps for the computer algorithm to determine the optimal driving plan for the vehicle are as follows:

[0121] Step A: First, obtain data from human-computer interaction data analysis, including vehicle position information, camera simulation data, radar simulation data, and lidar simulation data, as well as vehicle speed, acceleration, and steering wheel angle data;

[0122] Step B: Model the vehicle's surrounding environment using the data collected in step A, specifically using an extended Kalman filter for modeling:

[0123] [\hat{x}{t}=f(\hat{x}{t-1},u {t-1} )+w {t-1} ]

[0124] [z t =h(\hat{x} t )+vt ]

[0125] Among them, \hat{x}{t} is the current state estimate, u {t-1} is the control input at the previous moment, f is the state transfer function, w {t-1} is the process noise, z t is the current observation, h is the observation function, v t is the observation noise;

[0126] Step C, based on the environment model and the multi-sensor fusion module data of the vehicle, the optimal path from the current position of the vehicle to the target position is calculated by the RRT path planning algorithm;

[0127] Step D: After obtaining the best path, use the PID algorithm to generate specific control instructions to obtain the optimal driving plan. The specific algorithm formula is as follows:

[0128] [u(t) = K p e(t)+K i ∫e(t)dt+K d \frac{de(t){dt}}]

[0129] Where u(t) is the control input, e(t) is the speed error, K p , K i , K d is the gain of the PID controller.

[0130] Example 2

[0131] An electronic device according to an exemplary embodiment includes: a processor and a memory, wherein the memory stores a computer program that can be called by the processor;

[0132] The processor executes the above-mentioned highly immersive human-computer interactive virtual driving vehicle testing system by calling the computer program stored in the memory.

[0133] Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. The electronic device may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) and one or more memories, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor to implement a highly immersive human-computer interactive virtual driving vehicle testing system provided in the above-mentioned various method embodiments.

[0134] The electronic device may also include other components for realizing the functions of the device, for example, the electronic device may also have components such as a wired or wireless network interface and an input / output interface for input and output. The embodiments of the present application will not be described in detail here.

[0135] This embodiment also provides a computer program product stored on a computer-readable medium, including a computer-readable program, which, when executed on an electronic device, provides a user input interface to implement the highly immersive human-computer interactive virtual driving vehicle testing system.

[0136] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0137] It should be understood that determining B based on A does not mean determining B only based on A. B can also be determined based on A and / or other information.

[0138] By simulating various interactive scenarios, including emergency and non-standard operations, the test system of the present invention can help discover and correct design defects that may cause safety problems. This comprehensive test ensures that the behavior of the virtual driving vehicle test results in the real world is safer and more reliable; through highly realistic interactive simulation, designers can better understand user needs and preferences, so as to design an interactive interface that better meets user expectations. This not only improves user satisfaction, but also helps to improve user acceptance and trust in virtual driving technology; using virtual reality and augmented reality technology, the testing and iteration process can be carried out without physical prototypes, greatly reducing development time and cost. This enables products to enter the market faster while maintaining high standards of performance and reliability; with the rapid development of virtual driving technology, relevant regulations and standards are also constantly evolving. The highly immersive test system can ensure that the vehicle meets the latest safety standards and regulatory requirements in design and function, helping manufacturers to pass the certification process more smoothly; by simulating various potential usage scenarios, designers can identify and respond to problems in the design at an early stage. This forward-looking testing method makes the final product more flexible to adapt to future technology and market changes.

[0139] In short, the development and application of this highly immersive human-computer interaction test system will not only significantly improve the performance and user experience of virtual driving vehicles, but will also drive the entire industry towards higher technical standards and commercial success.

[0140] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A highly immersive human-computer interactive virtual driving vehicle testing system, include: A virtual reality module, a database input module, a human-computer interaction module, a control system module and a data recording and analysis module, wherein the virtual reality module is used to create a virtual driving environment, including several driving scenes such as urban roads, high-speed elevated highways and rural roads, and the virtual reality module uses a high-resolution display screen to achieve an immersive visual experience; The database input module is used to combine the data in the database into the vehicle surrounding environment information and generate a detailed environment model; Human-computer interaction module: including driver input devices, specifically steering wheel, pedals, touch screen; feedback devices, specifically force feedback system, vibration system, used to achieve interaction between the driver and the system; Control system module: used to process the data of the virtual reality module and the multi-sensor fusion module, and to control the behavior of the vehicle in real time according to the input of the human-computer interaction module; Data recording and analysis module: used to record the control system module and human-computer interaction module data during the test process, and analyze the data to optimize the human-computer interaction design and improve the performance of the driving vehicle.

2. A highly immersive human-computer interactive virtual driving vehicle testing system according to claim 1, Features: The virtual reality module specifically includes the following contents: a. The virtual reality module includes the following components: high-resolution display device: a high-definition giant screen display that provides realistic visual effects; computing device: a high-performance computer or server for generating and rendering virtual environments; audio output device: a high-fidelity sound system or headphones that provide three-dimensional spatial sound effects to enhance immersion; b. The virtual reality module creates a virtual driving environment through the following steps: Scene modeling: Use 3D modeling software to create detailed driving scenes, including urban roads, high-speed elevated highways and rural roads. The scenes should include traffic signs, pedestrians, other vehicles and natural environment elements; Texture and lighting: Use high-resolution textures and advanced lighting technology to render the scene to enhance the visual authenticity and details; Physical simulation: Integrated physics engine, specifically using NVIDIA PhysX to simulate real physical phenomena such as collisions, friction and weather changes, to improve the interactivity and reliability of the environment; c. The virtual driving environment includes a variety of driving scenarios to meet different testing needs, specifically: Urban roads: complex traffic conditions, including intersections, traffic lights, and pedestrian crossings; Expressways: lane changes, overtaking, and emergency braking at high speeds; Rural roads: narrow roads, sharp turns, and rugged road conditions.

3. A highly immersive human-computer interactive virtual driving vehicle testing system according to claim 1, Features: The database input module includes the following contents: a. Data collection in the database consists of: LiDAR simulation data: used to generate high-precision 3D point cloud data; Radar simulation data: used to simulate the measurement of speed and distance data under harsh conditions; Camera simulation data: used to provide high-resolution visual information and identify color, texture and shape details, where visual information is collected by RGB camera data, depth camera data, and infrared camera data; b. Data collection: LiDAR simulation data: Use high-frequency scanning to collect point cloud data to form a three-dimensional environment model. The specific scanning frequency is between 10Hz and 20Hz; Radar simulation data: Collect reflected signals to calculate the position, distance and relative speed of objects; Camera simulation data: Obtain depth information through stereo vision or structured light technology, capture two-dimensional images and video data for object recognition, area monitoring and auxiliary navigation; c. Data processing: Data preprocessing: LiDAR simulation data: filter out noise points and abnormal data to improve the quality of point cloud; Radar simulation data: signal processing and target detection, specifically through FFT to separate different objects; Camera simulation data: image preprocessing, specifically through denoising, white balance and image enhancement; Synchronization and calibration: Time synchronization: ensure that the laser radar simulation data, radar simulation data and camera simulation data are on the same time basis; Space calibration: perform spatial calibration on the laser radar simulation position, radar simulation position and camera simulation position based on the vehicle reference system to ensure the accuracy of data fusion; Calibration tool: use calibration plate and laser calibrator tool for precise calibration; d. Data information fusion: Fusion of database data: Use geometric transformation to project data from different sensors into the same coordinate system. Specifically, the point cloud data of the lidar is registered with the image data of the camera, and a high-precision 3D model is constructed through multi-frame fusion. The SLAM algorithm is used for real-time mapping and positioning. Data structure management: Point cloud data processing: Use the octree data structure to quickly process lidar point cloud data to achieve object detection and scene understanding; Image processing: Use convolutional neural networks to extract camera image features and identify targets; Radar simulation data processing: Process radar frequency domain and time domain data to detect and track moving targets; e. Environmental model generation: A two-dimensional grid map formed based on camera simulation data and radar simulation data is used to form a two-dimensional environmental model for path planning on the plane; a three-dimensional model is generated based on the point cloud data of the lidar and the depth information of the camera, and an oxygen grid map and a digital elevation map are created to improve the accuracy of environmental perception.

4. A highly immersive human-computer interactive virtual driving vehicle testing system according to claim 3, Features: The specific steps of registering the laser radar point cloud data with the camera image in the fusion of the database data are as follows: Step S1, camera intrinsic parameter calibration: Use the calibration board to take multiple pictures, calibrate through the OpenCV tool, and obtain the camera intrinsic parameter matrix K and distortion coefficient \mathbf{d}. The specific formula is as follows: Where (f x ,f y ) is the focal length, (c x ,c y ) are the principal point coordinates; Step S2, laser radar and camera extrinsic calibration: by registering the calibration plate, simultaneously obtain the point cloud and image data, perform manual or automatic registration, and calculate the extrinsic transformation matrix \mathbf{R} and translation vector \mathbf{t} between the two. The specific calculation formula is as follows: Step S3, point cloud data processing: remove outliers and noise points in the point cloud by statistical filtering and radius filtering methods; use voxel filtering method to reduce the point cloud data density and improve processing efficiency; Step S4: Project the processed LiDAR point cloud data onto the image plane of the camera, where the point cloud coordinates (\mathbf{P}{LiDAR}{pmatrix}X {LiDAR} &y {LiDAR} &z {LiDAR} &1{pmatrix} T ) represents a point in the laser radar coordinate system, specifically: Use the external parameter matrix \mathbf{T} to convert the laser radar point cloud coordinates into the camera coordinate system \mathbf{p}{Cam}: [\mathbf{p}{Cam}=\mathbf{T}·\mathbf{p}{Cam}] Projected onto the image plane, use the camera intrinsic parameter matrix K: [\mathbf{p}{img}=\mathbf{K}·\mathbf{P}{Cam}], where (\mathbf{P}{Cam}{pmatrix}X {Cam} &Y {Cam} &Z {Cam} {pmatrix} T ); Finally, the homogeneous coordinates are converted to image pixel coordinates (u, v) and normalized: [u = \frac{X {Cam} }{Z {Cam} },\quadv=\frac{Y {Cam} }{Z {Cam} }]; Step S5, draw the projection point in the image: according to the pixel coordinates (u, v) obtained in the previous step, draw the corresponding point cloud point on the image, color-code the depth or intensity in the point cloud, superimpose it on the image to enhance the display effect, then obtain the RGB value of the point from the image, and assign the corresponding point color to the point cloud to achieve color display of the point cloud.

5. A highly immersive human-computer interactive virtual driving vehicle testing system according to claim 1, Features: The human-computer interaction module specifically includes the following contents: Driver input device: Steering wheel: used to control the steering of the vehicle, wherein the steering wheel is provided with an angle sensor, an electronic power steering system and a force feedback device. Specifically, the rotation angle of the steering wheel is obtained through the angle sensor; The electronic power steering system provides auxiliary steering force, reducing the driver's operating effort; The force feedback motor transmits road information, such as collision and friction, to the driver, making the driving feel more realistic; Pedals: including accelerator pedal and brake pedal, used to control the acceleration and deceleration of the vehicle. The pedals are provided with position sensors and wire control systems. The position sensors are used to detect the pedal positions, and the wire control systems are used to replace traditional mechanical control to improve the corresponding speed and accuracy. Touch screen: for interactive display and user input; Feedback device: Force feedback system: applies reverse torque to the steering wheel through a motor or actuator to simulate road feedback and handling stability, increases reaction force on the pedals, and provides a realistic pedaling feel; Vibration feedback system: provides vibration feedback through a vibration motor or actuator under the seat, and the steering wheel has a built-in vibration device to remind the driver to pay attention; Visual feedback: projects important navigation and speed information onto the windshield, so that the driver does not need to look down at the dashboard, and provides vehicle status and warning information through an LCD or OLED display; Auditory feedback: provides voice navigation, warnings and operation prompts through the on-board speakers, and emits warning sounds through a buzzer or speaker.

6. A highly immersive human-computer interactive virtual driving vehicle testing system according to claim 1, Features: The control system module specifically includes the following contents: Data processing and integration: Virtual reality module data processing: convert the data of the virtual reality module into a format that can be recognized and processed by the vehicle control system; database input module data processing: synchronize the laser radar simulation data, radar simulation data and camera simulation data from the database to ensure time and space consistency, and use the Kalman filter algorithm to fuse the above synchronized data to improve data quality and accuracy; Real-time control logic: Human-computer interaction data analysis: Analysis of the driver's operating intentions expressed through the driver input device; Decision-making and execution: Based on the data collected through human-computer interaction data analysis, multi-sensor fusion module data and preset driving strategies, the computer algorithm determines the optimal driving plan for the vehicle; Human-machine collaboration: Driver intention feedback: Through the dashboard and HUD, the recognition results of the real-time control logic and the operation intention are fed back to the driver in real time to ensure that the driver understands the vehicle status and the operations to be performed; Manual and automatic driving switching: Smoothly switch between automatic and manual driving to ensure driving safety and comfort.

7. A highly immersive human-computer interactive virtual driving vehicle testing system according to claim 6, Features: The specific steps of the computer algorithm determining the optimal driving plan of the vehicle based on the data collected by the human-computer interaction data analysis and the preset driving strategy are as follows: Step A: First, obtain data from human-computer interaction data analysis, including vehicle position information, camera simulation data, radar simulation data, and lidar simulation data, as well as vehicle speed, acceleration, and steering wheel angle data; Step B: Model the vehicle's surrounding environment using the data collected in step A, specifically using an extended Kalman filter for modeling: [\hat{x}{t}=f(\hat{x}{t-1},u {t-1} )+w {t-1} ] [z t =h(\hat{x} t )+v t ] Among them, \hat{x}{t} is the current state estimate, u {t-1} is the control input at the previous moment, f is the state transfer function, w {t-1} is the process noise, z t is the current observation, h is the observation function, v t is the observation noise; Step C, based on the environment model and the multi-sensor fusion module data of the vehicle, the optimal path from the current position of the vehicle to the target position is calculated by the RRT path planning algorithm; Step D: After obtaining the best path, use the PID algorithm to generate specific control instructions to obtain the optimal driving plan. The specific algorithm formula is as follows: [u(t)=K p e(t)+K i ∫e(t)dt+K d \frac{de(t){dt}}] Where u(t) is the control input, e(t) is the speed error, K p , K i , K d is the gain of the PID controller.

8. An electronic device, It is characterized in that include: A processor and a memory, wherein the memory stores a computer program that can be called by the processor; The processor executes a highly immersive human-computer interactive virtual driving vehicle testing system as described in any one of claims 1 to 7 by calling a computer program stored in the memory.

9. A computer program product stored on a computer readable medium, Features: It comprises a computer-readable program, which, when executed on an electronic device, provides a user input interface to implement a highly immersive human-computer interactive virtual driving vehicle testing system as described in any one of claims 1-7.

Citation Information

Cited By

  • Immersive test system and method for automatic emergency braking system

    CN119756886A

  • Automobile intelligent driving emergency braking test method, device, equipment, medium and product

    CN120577033A