A visual augmented reality based vehicle-in-the-loop test method

Abstract

The application is a kind of vehicle-in-the-loop test method based on visual augmented reality. First, the positioning and motion information acquisition system of the real vehicle is established, and the high-precision motion information of the vehicle can be obtained. Then, the image information and motion information of the vehicle-mounted camera are synchronously collected, and the purpose is to obtain the camera output and vehicle motion information under the same time stamp. Secondly, through the internal parameters and initial external parameter information of the camera, the pose of the camera at each moment is calculated by combining the synchronously collected motion information. The calculated pose is synchronously input into the simulation software to obtain the virtual camera output consistent with the real camera in time and space. The real camera image collected is taken as the background, and the traffic participants provided by the virtual simulation camera are taken as the foreground. Through the image fusion algorithm, the fusion image information of the augmented reality perspective is obtained. Finally, the fusion image is input to the perception layer, the perception result is calculated and input to the decision control layer, and the decision instruction is output to the execution mechanism of the vehicle, so that the vehicle motion information is changed and collected by the high-precision information acquisition system, thereby completing a cycle of information.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of automatic driving test, in particular to a vehicle-in-the-loop test method based on visual augmented reality. BACKGROUND

[0002] In recent years, the development of advanced driver assistance systems (ADAS) and autonomous driving systems (AD) has significantly improved the convenience and safety of drivers and passengers. Unlike the traditional operation mode in which people, vehicles, and environments are independent of each other, autonomous vehicles form a highly coupled human-vehicle-environment-driving task integrated system. Therefore, the traditional testing method (mainly focusing on vehicle dynamic control, passive safety, and component performance) cannot fully meet the complex needs of autonomous driving systems.

[0003] Traditional real vehicle testing has limitations such as high cost, long cycle, and difficulty in reproducing specific scenarios, and has been unable to meet the requirements of autonomous vehicle testing. To solve these problems, car companies and research institutions have gradually turned to vehicle-in-the-loop testing (VIL). VIL testing has the advantages of rich test scenarios, fast computing speed, high efficiency, low resource consumption, and strong repeatability, and can be embedded in various aspects of vehicle development, becoming an ideal alternative.

[0004] Traditional VIL testing maps the motion state of the autonomous vehicle under test in the physical world to the digital twin vehicle in the simulation test scene in real time through high-speed data communication. The virtual sensors on the digital twin vehicle perceive the information of virtual traffic participants, and the algorithm under test makes decisions based on the received virtual simulation sensor signals. However, the testing accuracy of this method depends heavily on the accuracy of sensor modeling in virtual simulation software, and it is difficult to comprehensively test the perception layer of intelligent vehicles. Environmental perception is the basis for intelligent vehicles to understand the environment and understand real scenarios. Traditional VIL technology that only targets the decision layer lacks the ability to reproduce scenarios, resulting in a disconnect between perception and decision-making, making it difficult to fully test the functional defects of the algorithm.

[0005] In view of the above limitations, there is an urgent need for a new type of VIL testing method that combines the application of real vehicle sensors, fully reproduces real test scenarios, and effectively evaluates the perception layer of intelligent vehicles. The present application proposes an innovative testing scheme that not only inherits the advantages of traditional VIL testing, but also specifically tests the perception layer of intelligent vehicles, thereby improving the comprehensiveness and accuracy of testing. This improved testing method can better simulate actual driving conditions and ensure the safe and reliable development of autonomous driving technology. Summary of the Invention

[0006] To address the aforementioned issues, this invention provides a vehicle-in-the-loop testing method based on visual augmented reality. First, a real vehicle positioning and motion information acquisition system is established to obtain high-precision vehicle motion information. Then, image and motion information from an onboard camera are simultaneously acquired to obtain camera output and vehicle motion information at the same timestamp. Next, the camera's pose at each moment is calculated using the camera's intrinsic parameters and initial extrinsic parameters, combined with the synchronously acquired motion information. The calculated pose is synchronously input into simulation software to obtain virtual camera output consistent with the real camera's temporal and spatial dimensions. Using the real-world camera image as the background and traffic participants provided by the virtual simulation camera as the foreground, an image fusion algorithm is used to obtain fused image information from an augmented reality perspective. Finally, the fused image is input to the perception layer, and after calculating the perception results, it is input to the decision control layer, which outputs decision commands to the vehicle's actuators. This alters the vehicle's motion information, which is then collected again by the high-precision information acquisition system, thus completing one round of information circulation.

[0007] The technical solution of this invention is described below in conjunction with the accompanying drawings:

[0008] A vehicle-in-the-loop testing method based on visual augmented reality includes the following steps:

[0009] Step 1: Build a positioning and motion information acquisition system using RT3002 and Qianxun Positioning RTK service. This system can record the vehicle's position and dynamic parameters in real time and accurately, ensuring the accuracy of the collected vehicle positioning and motion data.

[0010] Step 2: Use the time synchronization mechanism in ROS to synchronously acquire image information from the vehicle-mounted camera and the corresponding vehicle motion information to ensure consistency between the camera output and the vehicle motion state at the same timestamp; this step is crucial for subsequent image fusion.

[0011] Step 3: Using the camera's initial extrinsic parameters and the synchronously acquired motion information, calculate the camera's specific position and attitude (pose) at each moment; this process ensures that the camera in the virtual environment behaves exactly the same as the camera in the real world.

[0012] Step 4: Synchronously input the calculated camera pose information into the simulation software, design the camera parameters of the virtual camera to be consistent with those of the real camera, and generate a virtual camera output that is completely consistent with the real camera in time and space.

[0013] Step 5: Using real-world camera images as the background and traffic participants provided by a virtual simulation camera as the foreground, an augmented reality-perspective fused image is generated using advanced image fusion algorithms. This fused image not only preserves the details of the real world but also incorporates simulated dynamic elements, greatly enriching the complexity and realism of the test scenario.

[0014] Step Six: The fused image is input into the vehicle's perception layer for processing to test the perception layer's ability to perceive the fusion of real-world scenes and virtual dynamic elements, as well as its adaptability to complex environments. By preserving real-world background details and overlaying dynamic traffic participants from the virtual environment, the fused image not only fully reproduces the real driving scenario but also overcomes the problem of separation between perception and decision-making in traditional VIL testing. This allows the perception layer to accurately evaluate object recognition and tracking performance in dynamic environments. The perception results are then transmitted to the decision control layer, generating corresponding decision commands and sending them to the vehicle's actuators. The vehicle adjusts its motion state according to these commands, while new motion information is captured again by the high-precision information acquisition system, ensuring the closed-loop consistency between vehicle behavior and perception decision-making.

[0015] Furthermore, the specific method for step two is as follows:

[0016] 21) The high-precision motion information acquisition module data and camera data are received uniformly through the ROS system. The high-precision motion information acquisition module data is in CAN format, and the camera data acquisition format is sensor_msgs / Image. The high-precision motion information acquisition module includes the vehicle's latitude and longitude information and heading, pitch, and roll angle information.

[0017] 22) Use the TimeSynchronizer module in the ROS system to synchronize the data from the high-precision motion information acquisition module and the camera data to ensure that the different received signals are aligned at the same timestamp, thereby achieving precise time synchronization of the signals.

[0018] Furthermore, the specific method for step three is as follows:

[0019] 31) The intrinsic and initial extrinsic parameters of the real camera are calibrated. The intrinsic parameter calibration uses Zhang Youding calibration method to obtain the intrinsic parameter matrix of the camera. The extrinsic parameter calibration first measures the world coordinate point and the corresponding pixel coordinate, and then solves the rotation matrix R and translation vector t of the camera in the coordinate system with RT as the origin through PnP algorithm.

[0020] 32) Calculate the vehicle's translation vector using latitude and longitude information; calculate the vehicle's rotation matrix using heading, pitch, and roll angle information; and calculate the camera's pose in the world coordinate system by combining the calibrated intrinsic parameter matrix of the vehicle camera with the initial extrinsic parameter information.

[0021] Furthermore, the specific method for step four is as follows:

[0022] 41) Using the calibrated camera intrinsic parameter information, set the parameters of the virtual camera: principal point coordinates, focal length, camera imaging height and width, field of view, etc., to be consistent with the real camera.

[0023] 42) Combining the calculated pose of the real camera, control the movement of the virtual camera to be consistent with the calculated pose of the real camera, thereby obtaining the output of the virtual world from the same perspective as the real camera, including the virtual world RGB image and the virtual world semantic segmentation image.

[0024] Furthermore, the specific method for step five is as follows:

[0025] 51) Design an image extraction method for virtual traffic participants that includes sun shadows;

[0026] 52) Perform image fusion on the background image captured by the real camera and the images of traffic participants in the virtual simulation world.

[0027] Furthermore, the specific method for step 51) is as follows:

[0028] 511) Calculate the solar altitude angle and solar azimuth angle at the time of the experiment based on the latitude and longitude of the experimental site and the time of the experiment, so as to obtain the solar shadow in the virtual simulation world with the same orientation as the real world.

[0029] Formula for calculating solar angle:

[0030] h=arcsin(sin(φ)·sin(δ)+cos(φ)·cos(δ)·cos(H))

[0031] The solar azimuth angle is calculated using the following formula:

[0032]

[0033] In the formula, φ is the latitude of the observation point; δ is the declination angle of the sun; and H is the solar hour angle.

[0034] After generating the sun shadow, extract the corresponding sun shadow portion;

[0035] By converting an RGB image to LAB space, when the sum of the average values ​​of channels A and B is less than a given threshold ab_threshold, pixel values ​​satisfying L≤[mean(L)-std(L) / 3] are classified as shadow regions, and other pixels are classified as non-shadow pixels, where mean(L) represents the mean of the pixel channels and std(L) represents the standard deviation of the pixel channels; when the sum of the average values ​​of channels A and B is greater than the given threshold ab_threshold, the portions of channels L and B below the channel threshold are classified as shadow regions.

[0036] 512) Traffic participants are extracted by combining the semantically segmented image with the original RGB image; based on the semantic segmentation results, the category of the target object is selected, and a binarized template is generated, as shown below:

[0037]

[0038] In the formula, S(x,y) is the label at position (x,y) in the semantic segmentation result; M(x,y) is the generated binarized template;

[0039] The process for obtaining the overall binarized image of the traffic object region and the shadow region first involves calculating the shadow mask to remove the portion overlapping with the traffic object:

[0040]

[0041] In the formula, M is the complement of the traffic object mask; shadow For the shadow mask;

[0042] Next, the de-overlapping shadow mask and vehicle mask are merged:

[0043] M combined =M object ∪M refined_shadow

[0044] In the formula, M combined A masked image of the entire region containing traffic objects and shadows;

[0045] In summary, the entire process can be represented as follows:

[0046]

[0047] Using the generated binarized template M, a pixel-by-pixel masking operation is performed on the original image to extract the image region of the target object, as shown below:

[0048] O(x,y) = I(x,y) × M(x,y);

[0049] In the formula, I(x,y) is the original image; M(x,y) is the masked image of the entire region containing traffic objects and shadows;

[0050] Furthermore, the specific method for step 52) is as follows:

[0051] 521) Gaussian convolution is performed on the masked image of traffic participants containing sun shadows to create a gradual change in the edge regions of the mask. The Gaussian function in two-dimensional space is expressed as:

[0052]

[0053] In the formula, (x,y) is the offset relative to the center pixel; σ is the standard deviation of the normal distribution; and r is the blur radius (r0). 2 =x 2 +y 2 G(x,y) is the weight of position (x,y);

[0054] The Gaussian blur process for a 2D image is as follows: applying Gaussian blur to an image is achieved by convolving the image with a Gaussian kernel matrix. The entire process is represented by the following formula:

[0055]

[0056] In the formula, I(i,j) is the pixel value of the original image at position (i,j); I'(i,j) is the pixel value of the blurred image at position (i,j); G(x,y) is the weight of position (x,y); and k is the radius of the kernel.

[0057] 522) The mask image after Gaussian convolution is alpha-fused with the corresponding RGB image and the foreground image captured by the real camera to achieve a smooth blend between the foreground and background. The specific formula is as follows:

[0058] x output =x foreground *x mask +x background *(1-x mask )

[0059] In the formula, x output The image after fusion; x foreground Foreground part; x background For the background part; x mask For masking; x mask The actual pixel value will be normalized from [0,255] to [0,1]; when x mask =0, the output pixel value belongs to the background; when x = 0, the output pixel value belongs to the background. mask =1, the output pixel belongs to the foreground; when 0 < x mask<1, the output pixel value is the blend of the foreground and background.

[0060] Furthermore, the specific method for step six is ​​as follows:

[0061] 61) The augmented reality fusion image from step five is input into the vehicle's perception layer for processing to test the perception layer's ability to perceive the fusion of real-world scenes and virtual dynamic elements, as well as its adaptability to complex environments. The fusion image, by preserving real-world background details and overlaying dynamic traffic participants from the virtual environment, not only fully reproduces the real driving scenario but also overcomes the problem of separation between perception and decision-making in traditional VIL testing. This allows the perception layer to accurately evaluate object recognition and tracking performance in dynamic environments. The perception layer calculates the motion information of the corresponding traffic participants, including speed and distance. The perception layer then sends the calculated relative speed and relative distance to the decision control module via UDP communication.

[0062] 62) After receiving the perceived signals, the decision control module calculates the corresponding control signals and sends them to the vehicle's drive-by-wire actuators, thereby changing the vehicle's motion information. Simultaneously, the high-precision information acquisition system collects new motion information, enabling the entire test system to enter the next update cycle. Through this process, this test method verifies the comprehensiveness and accuracy of the perception layer while ensuring that the scenario reproduction capability is closer to actual driving conditions, thus improving the safety and reliability of autonomous driving technology.

[0063] The beneficial effects of this invention are as follows:

[0064] 1) This invention can enhance the testing capabilities of the perception layer by using augmented reality technology to integrate the real-world background with the foreground of virtual traffic participants, providing a more complex and realistic testing environment for the perception layer of intelligent vehicles and significantly improving the testing effect of the perception layer algorithm.

[0065] 2) This invention improves the comprehensiveness and repeatability of testing. The fused images from an augmented reality perspective not only retain details of the real world but also incorporate simulated dynamic elements, making the test scenarios richer and more diverse. Furthermore, the testing process exhibits good repeatability, which is helpful for in-depth analysis and improvement of autonomous driving algorithms.

[0066] 3) This invention optimizes testing efficiency and cost-effectiveness. Compared with traditional real vehicle testing, it reduces high costs and long time cycles while improving testing efficiency. By simulating diverse traffic scenarios, more diverse test cases can be completed in a shorter time, reducing resource consumption. Attached Figure Description

[0067] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 This is a schematic diagram of the process of the present invention;

[0069] Figure 2 This is a schematic diagram of a high-precision information acquisition system.

[0070] Figure 3 A time synchronization diagram of camera image information and high-precision motion information;

[0071] Figure 4 Flowchart for extracting solar shadows;

[0072] Figure 5 A schematic diagram of traffic participants in the shadow of the sun;

[0073] Figure 6 This is a schematic diagram of the Alpha fusion principle based on feathering mask;

[0074] Figure 7 An illustration of the fusion of virtual and real images from an augmented reality perspective. Detailed Implementation

[0075] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present invention, and not all of the structures.

[0076] See Figure 1 This embodiment provides a vehicle-in-the-loop testing method based on visual augmented reality, including the following steps:

[0077] Step 1, Refer to Figure 2A high-precision vehicle motion information acquisition system was constructed by combining the RT3002 device with Qianxun Positioning's RTK differential service. The RT3002 device has a built-in high-performance GPS positioning module that receives satellite signals to determine the vehicle's position and speed. It is also equipped with an inertial measurement unit (IMU) consisting of three mutually perpendicular single-axis accelerometers and three mutually perpendicular single-axis gyroscopes to collect longitudinal and lateral acceleration data of the vehicle and calculate attitude information such as heading, pitch, and roll in real time based on the angular velocity of the vehicle's rotation around each coordinate axis. Through an embedded 24th-order Kalman filter algorithm, the RT3000 V3 can automatically correct errors caused by inaccurate antenna installation. By fusing its own acquired positioning data with differential data provided by Qianxun FindCM, the RT3002 achieves centimeter-level positioning accuracy through differential calculation.

[0078] Step Two, Refer to Figure 3 The time synchronization mechanism in ROS is used to synchronously acquire image information from the vehicle-mounted camera and the corresponding vehicle motion information, as detailed below:

[0079] The high-precision motion information acquisition module and camera data are received uniformly through the ROS system. The high-precision motion information acquisition module data is in CAN format, and the camera data acquisition format is sensor_msgs / Image. The high-precision motion information acquisition module includes the vehicle's latitude and longitude information, as well as heading, pitch, and roll angle information.

[0080] The above topics are received uniformly by the TimeSynchronizer in the ROS system. Only when all topics have the same timestamp will a synchronization result callback function be generated, thereby enabling the camera signal to synchronize with the CAN signal output by the RT.

[0081] Step 3: Using the camera's initial extrinsic parameters and the synchronously acquired motion information, calculate the camera's specific position and attitude or pose at each moment, as follows:

[0082] The first step is to calibrate the camera's intrinsic parameters and initial extrinsic parameters. This calibration primarily involves determining the camera's position in a coordinate system with the vehicle's inertial navigation system (RT) as the origin, including the camera's position (x, y, z) and attitude angles (heading, pitch, roll). This invention calibrates the vehicle camera's initial extrinsic parameters by measuring the coordinates of known points on a calibration board and the corresponding pixel coordinates on the camera's image, using a PNP algorithm.

[0083] Secondly, since the high-precision vehicle positioning information obtained is in the form of latitude and longitude, it is not suitable for calculating vehicle translation. Therefore, it is necessary to convert the latitude and longitude information into planar coordinates in the UTM coordinate system. The latitude and longitude are (λ, φ), in radians, and the UTM coordinates are (E, N), in km. The specific conversion formula is as follows:

[0084] k0=0.9996, E0=500km, e=0.0818192, a=6378.137km

[0085]

[0086] λ0=(zoneNumber-1)*6-180+3

[0087]

[0088] In the formula, λ is longitude; λ is latitude; k0 is the standard scale factor; E0 is the eastward shift of the central meridian; N0 is the northward shift of the meridian's starting point; e is the Earth's eccentricity; a is the major axis radius of the Earth's ellipsoid; T is the square of the tangent at latitude; C is the angle between the meridian and the parallel; zoneNumber is the UTM projection zone number; v is the radius of curvature of latitude; λ0 is the central meridian of each zone; A is the difference between longitude and the central meridian multiplied by the cosine of latitude, used to calculate the lateral offset; s is the meridian arc length; E is the eastward coordinate in the UTM coordinate system; N is the northward coordinate in the UTM coordinate system.

[0089] After obtaining the vehicle's position in the planar coordinate system, the translation vector of the vehicle at the selected world coordinate point can be calculated, which can be specifically described as:

[0090] t vehicle_to_world =(x vehicle -x origin ,y vehicle -y origin ,z vehicle -z origin )

[0091] In the formula, (x origin ,y origin ,z origin ) represents the coordinates of the selected world coordinate origin.

[0092] The vehicle's rotation matrix in the world coordinate system can be calculated based on its pitch, roll, and heading angles.

[0093]

[0094] R vechicle_to_world =R z(heading)·R y (pitch)·R x (roll)

[0095] After calculating the vehicle's rotation matrix and translation vector in the world coordinate system using the formula, and combining this with the camera's initial extrinsic parameters from the previous section, the camera's pose in the world coordinate system can be calculated, specifically as follows:

[0096]

[0097] In the formula, R cam_veh and t cam_veh These are the rotation matrix and translation vector of the camera relative to the vehicle, respectively, which can be calculated using the initial parameters of the camera calibration.

[0098] Step 4: Synchronously input the calculated camera pose information into the simulation software, design the camera parameters of the virtual camera to be consistent with those of the real camera, and generate a virtual camera output that is completely consistent with the real camera in time and space, as detailed below:

[0099] Using the calibrated camera intrinsic parameters, the parameters of the virtual camera are set to be consistent with those of the real camera: principal point coordinates, focal length, camera imaging height and width, field of view, etc.; combined with the pose of the real camera calculated in step three, the movement of the virtual camera is controlled to be consistent with the calculated pose of the real camera, so as to obtain the output of the virtual world from the same perspective as the real camera, including the virtual world RGB image and the virtual world semantic segmentation image.

[0100] Step 5: Using real-world camera images as the background and traffic participants provided by a virtual simulation camera as the foreground, generate a fused image from an augmented reality perspective using an image fusion algorithm, as follows:

[0101] First, the solar altitude angle and azimuth angle are calculated based on the latitude and longitude of the test site and the test time to obtain the shadow generation direction that conforms to the real world. The formulas for calculating the solar altitude angle and azimuth angle are as follows:

[0102] h=arcsin(sin(φ)·sin(δ)+cos(φ)·cos(δ)·cos(H))

[0103]

[0104] In the formula, φ is the latitude of the observation point; δ is the declination angle of the sun; H is the solar hour angle (determined by local time and geographical location); h is the calculated solar altitude angle; and A is the calculated solar azimuth angle.

[0105] Once the sun's precise position and angle are obtained, the sun's position parameters in the virtual environment can be set based on this data in 3D modeling and rendering software. The core objective of this process is to ensure that the generated shadows realistically reflect the effects of sunlight in the real world, ensuring that the generated shadow shapes are reasonable and consistent with reality.

[0106] After generating sun shadows, effectively extracting and accurately identifying shadow areas is one of the key steps in achieving realistic lighting effects in virtual scenes. (Refer to...) Figure 4 The process of extracting sun shadows using the LAB color model first converts the RGB image into an equivalent LAB image, then calculates the pixel mean and standard deviation for the L, A, and B channels. The mean of the L channel represents the overall brightness level of the image, while the standard deviation reflects the distribution range of brightness. A lower L value usually indicates that the area is less bright and may be a shadow area.

[0107] The mean values ​​of the A and B channels reflect the color information in an image, especially in shadow areas. Due to the angle of sunlight and the occlusion of objects, shadow areas may exhibit a relatively uniform hue, and the values ​​of the A and B channels are quite similar. When the sum of the mean values ​​of the A and B channels is less than a given threshold ab_threshold, pixel values ​​satisfying L ≤ [mean(L) - std(L) / 3] are classified as shadow areas, and other pixels are classified as non-shadow pixels, where mean(L) represents the mean value of the pixel channel and std(L) represents the standard deviation of the pixel channel. When the sum of the mean values ​​of the A and B channels is greater than the given threshold ab_threshold, the extraction of shadow areas needs to consider the brightness information of both the L and B channels. In this case, the portions of the L and B channels below the channel threshold can be classified as shadow areas. This threshold is usually set as the channel mean value minus 1 / 3 of the channel standard deviation. In this way, pixels with low brightness and colors close to those of shadow areas can be identified more accurately.

[0108] Traffic participant extraction is primarily achieved by combining semantic segmentation images with the original RGB image. Semantic segmentation images provide the category label for each pixel in the image. Based on the semantic segmentation results, the category of the target object is selected, and a binary template is generated. In this template, pixels belonging to the target object are labeled as 1, and other background pixels are labeled as 0. This process can be achieved through a thresholding operation, with the formula:

[0109]

[0110] The process for obtaining the overall binarized image of the traffic object area and the shadow area is as follows:

[0111] First, calculate the shadow mask and remove the portion that overlaps with the traffic object:

[0112]

[0113] Next, the de-overlapping shadow mask and the vehicle mask are merged, as shown in the following formula:

[0114] M combined =M object ∪M refined_shadow

[0115] In the formula, M combined A masked image of the entire area containing traffic objects and shadows.

[0116] In summary, the entire process can be represented as:

[0117]

[0118] Using the generated binarized template M, a pixel-by-pixel masking operation is performed on the original image I to extract the image region of the target object, such as... Figure 5 As shown. The specific steps are as follows:

[0119] O(x,y) = I(x,y) × M(x,y)

[0120] In the formula, I(x,y) is the pixel value at position (x,y) in the original image; O(x,y) is the pixel at position (x,y) in the extracted target object image.

[0121] The mask image M has been obtained. combined After obtaining the corresponding RGB image O(x,y) and the real background image, the next step is to fuse the images. The fusion process is described in [reference needed]. Figure 6 .

[0122] Mask image M combined Perform Gaussian convolution to obtain a binary mask with feathered edges, as shown in the following formula:

[0123]

[0124] In the formula, I(i,j) is the pixel value of the original image at position (i,j); I'(i,j) is the pixel value of the blurred image at position (i,j); G(x,y) is the weight of position (x,y); and k is the radius of the kernel.

[0125] Finally, the virtual world image and the real background image are combined with a feathered mask for alpha fusion to obtain the final augmented reality image, such as... Figure 7 As shown:

[0126] x output =x foreground *xmask +x background *(1-x mask )

[0127] In the formula, x output The image after fusion; x foreground Foreground part; x background For the background part; x mask For masking. The above x mask The actual pixel values ​​will be normalized from [0, 255] to [0, 1]. When The output pixel value belongs to the background; when x mask =1, the output pixel belongs to the foreground; when 0 < x mask <1, the output pixel value is the blend of the foreground and background.

[0128] Step Six: The fused image is input into the vehicle's perception layer for processing to test the perception layer's ability to perceive the fusion of real-world scenes and virtual dynamic elements, as well as its adaptability to complex environments. The fused image, by preserving real-world background details and overlaying dynamic traffic participants from the virtual environment, fully reproduces the real driving scenario, enabling the perception layer to accurately assess object recognition and tracking performance in dynamic environments. The perception results are then transmitted to the decision control layer, generating corresponding decision commands and sending them to the vehicle's actuators. The vehicle adjusts its motion state according to these commands, while new motion information is captured again by the high-precision information acquisition system, ensuring the closed-loop consistency between vehicle behavior and perception decisions, as detailed below:

[0129] 61) The augmented reality fusion image from step five is input into the vehicle's perception layer for processing to test the perception layer's ability to perceive the fusion of real-world scenes and virtual dynamic elements, as well as its adaptability to complex environments. The fusion image, by preserving real-world background details and overlaying dynamic traffic participants from the virtual environment, not only fully reproduces the real driving scenario but also overcomes the problem of separation between perception and decision-making in traditional VIL testing. This allows the perception layer to accurately evaluate object recognition and tracking performance in dynamic environments. The perception layer calculates the motion information of the corresponding traffic participants, including speed and distance. The perception layer then sends the calculated relative speed and relative distance to the decision control module via UDP communication.

[0130] 62) After receiving the perceived signals, the decision control module calculates the corresponding control signals and sends them to the vehicle's drive-by-wire actuators, thereby changing the vehicle's motion information. Simultaneously, the high-precision information acquisition system collects new motion information, enabling the entire test system to enter the next update cycle. Through this process, this test method verifies the comprehensiveness and accuracy of the perception layer while ensuring that the scenario reproduction capability is closer to actual driving conditions, thus improving the safety and reliability of autonomous driving technology.

[0131] In summary, this invention effectively addresses the shortcomings of traditional testing methods for testing the perception layer of intelligent vehicles by introducing key technologies such as high-precision positioning and motion information acquisition, synchronous image and motion information acquisition, accurate calculation of camera pose, generation of consistent virtual camera output, image fusion, and perception layer testing and decision control.

[0132] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A vehicle-in-the-loop testing method based on visual augmented reality, characterized in that, Includes the following steps: Step 1: Build a positioning and motion information acquisition system using RT3002 and Qianxun Positioning RTK service; Step 2: Use the time synchronization mechanism in ROS to synchronously acquire image information from the vehicle-mounted camera and the corresponding vehicle motion information; Step 3: Using the camera's initial extrinsic parameters and the synchronously acquired motion information, calculate the camera's specific position and attitude or pose at each moment. Step 4: Synchronously input the calculated camera pose information into the simulation software, design the camera parameters of the virtual camera to be consistent with those of the real camera, and generate a virtual camera output that is completely consistent with the real camera in time and space. Step 5: Using real-world camera images as the background and traffic participants provided by virtual simulation cameras as the foreground, generate a fused image from an augmented reality perspective using an image fusion algorithm. Step Six: The fused image is input into the vehicle's perception layer for processing to test the perception layer's ability to perceive the fusion of real-world scenes and virtual dynamic elements, as well as its adaptability to complex environments. By preserving background details of the real world and overlaying dynamic traffic participants from the virtual environment, the fused image fully reproduces the real driving scenario, enabling the perception layer to accurately evaluate object recognition and tracking performance in dynamic environments. The perception results are then transmitted to the decision control layer, generating corresponding decision commands and sending them to the vehicle's actuators. The vehicle adjusts its motion state according to these commands, while new motion information is captured again by the high-precision information acquisition system, ensuring the closed-loop consistency between vehicle behavior and perception decisions. The specific method for step five is as follows: 51) Design an image extraction method for virtual traffic participants that includes sun shadows; 52) Perform image fusion between background images captured by real cameras and images of traffic participants in the virtual simulation world; The specific method for step 51) is as follows: 511) Calculate the solar altitude angle and solar azimuth angle at the time of the experiment based on the latitude and longitude of the experimental site and the time of the experiment, so as to obtain the solar shadow in the virtual simulation world with the same orientation as the real world. Formula for calculating solar angle: h=arcsin(sin(φ)·sin(δ)+cos(φ)·cos(δ)·cos(H)) The solar azimuth angle is calculated using the following formula: In the formula, φ is the latitude of the observation point; δ is the declination angle of the sun; and H is the solar hour angle. After generating the sun shadow, extract the corresponding sun shadow portion; By converting an RGB image to LAB space, when the sum of the average values ​​of channels A and B is less than a given threshold ab_threshold, pixel values ​​satisfying L≤[mean(L)-std(L) / 3] are classified as shadow regions, and other pixels are classified as non-shadow pixels, where mean(L) represents the mean of the pixel channels and std(L) represents the standard deviation of the pixel channels; when the sum of the average values ​​of channels A and B is greater than the given threshold ab_threshold, the portions of channels L and B below the channel threshold are classified as shadow regions. 512) Traffic participants are extracted by combining the semantically segmented image with the original RGB image; based on the semantic segmentation results, the category of the target object is selected, and a binarized template is generated, as shown below: In the formula, S(x,y) is the label at position (x,y) in the semantic segmentation result; M(x,y) is the generated binarized template; The process for obtaining the overall binarized image of the traffic object region and the shadow region first involves calculating the shadow mask to remove the portion overlapping with the traffic object: In the formula, M is the complement of the traffic object mask; shadow For the shadow mask; Next, the de-overlapping shadow mask and vehicle mask are merged: M combined =M object ∪M refined_shadow In the formula, M combined A masked image of the entire region containing traffic objects and shadows; In summary, the entire process can be represented as follows: Using the generated binarized template M, a pixel-by-pixel masking operation is performed on the original image to extract the image region of the target object, as shown below: O(x,y) = I(x,y) × M(x,y); In the formula, I(x,y) is the original image; M(x,y) is the masked image of the entire region containing traffic objects and shadows.

2. The vehicle-in-the-loop testing method based on visual augmented reality according to claim 1, characterized in that, The specific method for step two is as follows: 21) The high-precision motion information acquisition module data and camera data are received uniformly through the ROS system. The high-precision motion information acquisition module data is in CAN format, and the camera data acquisition format is sensor_msgs / Image. The high-precision motion information acquisition module includes the vehicle's latitude and longitude information and heading, pitch, and roll angle information. 22) Use the TimeSynchronizer module in the ROS system to synchronize the data from the high-precision motion information acquisition module and the camera data to ensure that the different received signals are aligned at the same timestamp, thereby achieving precise time synchronization of the signals.

3. The vehicle-in-the-loop testing method based on visual augmented reality according to claim 2, characterized in that, The specific method for step three is as follows: 31) The intrinsic and initial extrinsic parameters of the real camera are calibrated. The intrinsic parameter calibration uses Zhang Youding calibration method to obtain the intrinsic parameter matrix of the camera. The extrinsic parameter calibration first measures the world coordinate point and the corresponding pixel coordinate, and then solves the rotation matrix R and translation vector t of the camera in the coordinate system with RT as the origin through PnP algorithm. 32) Calculate the vehicle's translation vector using latitude and longitude information; calculate the vehicle's rotation matrix using heading, pitch, and roll angle information; and calculate the camera's pose in the world coordinate system by combining the calibrated intrinsic parameter matrix of the vehicle camera with the initial extrinsic parameter information.

4. The vehicle-in-the-loop testing method based on visual augmented reality according to claim 1, characterized in that, The specific method for step four is as follows: 41) Using the calibrated camera intrinsic parameter information, set the parameters of the virtual camera: principal point coordinates, focal length, camera imaging height and width, field of view, etc., to be consistent with the real camera. 42) By combining the calculated pose of the real camera, the movement of the virtual camera is controlled to be consistent with the calculated pose of the real camera, thereby obtaining the output of the virtual world from the same perspective as the real camera, including the virtual world RGB image and the virtual world semantic segmentation image.

5. The vehicle-in-the-loop testing method based on visual augmented reality according to claim 1, characterized in that, The specific method for step 52) is as follows: 521) Gaussian convolution is performed on the masked image of traffic participants containing sun shadows to create a gradual change in the edge regions of the mask. The Gaussian function in two-dimensional space is expressed as: In the formula, (x,y) is the offset relative to the center pixel; σ is the standard deviation of the normal distribution; and r is the blur radius (r0). 2 =x 2 +y 2 G(x,y) is the weight of position (x,y); The Gaussian blur process for a 2D image is as follows: applying Gaussian blur to an image is achieved by convolving the image with a Gaussian kernel matrix. The entire process is represented by the following formula: In the formula, I(i,j) is the pixel value of the original image at position (i,j); I ' (i,j) represents the pixel value at position (i,j) in the blurred image; G(x,y) represents the weight at position (x,y); and k represents the radius of the kernel. 522) The mask image after Gaussian convolution is alpha-fused with the corresponding RGB image and the foreground image captured by the real camera to achieve a smooth blend between the foreground and background. The specific formula is as follows: x output =x foreground *x mask +x background *(1-x mask ) In the formula, x output The image after fusion; x foreground Foreground part; x background For the background part; x mask For masking; x mask The actual pixel value will be normalized from [0,255] to [0,1]; when x mask =0, the output pixel value belongs to the background; when x = 0, the output pixel value belongs to the background. mask =1, the output pixel belongs to the foreground; when 0 <x mask <1, the output pixel value is the blend of the foreground and background.

6. The vehicle-in-the-loop testing method based on visual augmented reality according to claim 1, characterized in that, The specific method for step six is ​​as follows: 61) The augmented reality fusion image from step five is input into the vehicle's perception layer for processing to test the perception layer's ability to perceive the fusion of real-world scenes and virtual dynamic elements, as well as its adaptability to complex environments. By preserving background details of the real world and overlaying dynamic traffic participants from the virtual environment, the fusion image enables the perception layer to accurately evaluate object recognition and tracking performance in dynamic environments. The perception layer calculates the motion information, speed, and distance of the corresponding traffic participants. The perception layer then sends the calculated relative speed and relative distance to the decision control module via UDP communication. 62) After receiving the sensed signal, the decision control module calculates the corresponding control signal and sends it to the vehicle's drive-by-wire actuator, thereby changing the vehicle's motion information. At the same time, the high-precision information acquisition system collects new motion information, thus enabling the entire test system to enter the next round of updates.

Images