Vehicle-in-the-loop test method based on visual augmented reality

Abstract

The invention relates to a vehicle in-the-loop test method based on visual augmented reality. Firstly, a positioning and motion information acquisition system of a real vehicle is established, and high-precision motion information of the vehicle can be obtained; then image information and motion information of a vehicle-mounted camera are synchronously collected, and the purpose is to obtain camera output and vehicle motion information under the same timestamp; secondly, calculating the pose of the camera at each moment according to the internal participation initial external parameter information of the camera and the synchronously acquired motion information; synchronously inputting the calculated pose into simulation software, and obtaining virtual camera output consistent with the real camera in time and space; taking a really acquired camera image as a background, taking a traffic participant provided by a virtual simulation camera as a foreground, and obtaining fused image information of an augmented reality view angle through an image fusion algorithm; and finally, the fused image is input to the sensing layer, a sensing result is settled and then input to the decision control layer, a decision instruction is output to an execution mechanism of the vehicle, the vehicle motion information is changed and then is acquired by the high-precision information acquisition system, and thus a round of information circulation is completed.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving testing, and specifically is a vehicle-in-the-loop testing method based on visual augmented reality. Background Art

[0002] In recent years, the development of Advanced Driver Assistance Systems (ADAS) and Autonomous Driving (AD) has significantly improved the convenience and safety of drivers and passengers. Unlike the independent operation mode of human, vehicle and environment in traditional cars, autonomous vehicles constitute a highly coupled human-vehicle-environment-driving task integrated system. Therefore, traditional testing methods (mainly focusing on vehicle dynamic control, passive safety and component performance) can no longer fully meet the complex requirements of autonomous driving systems.

[0003] Traditional real vehicle testing can no longer meet the requirements of autonomous vehicle testing due to its high cost, long cycle, and difficulty in reproducing specific scenarios. In order to solve these problems, various car companies and research institutions have gradually turned to vehicle-in-the-Loop Testing (VIL). VIL testing has the advantages of rich test scenarios, fast calculation speed, high efficiency, low resource consumption, and strong repeatability. It can be embedded in all aspects of vehicle development and become an ideal alternative.

[0004] Traditional VIL testing uses high-speed data communication to map the motion state of the autonomous driving car to be tested in the physical world to the digital twin vehicle in the simulation test scenario in real time. The virtual sensors on the digital twin vehicle perceive the information of virtual traffic participants, and the algorithm to be tested makes decisions based on the received virtual simulation sensor signals. However, the test accuracy of this method is heavily dependent on the accuracy of sensor modeling in the virtual simulation software, and it is difficult to conduct a comprehensive test of the perception layer of intelligent vehicles. Environmental perception is the basis for intelligent vehicles to recognize the environment and understand real scenes. Traditional VIL technology that only targets the decision-making layer lacks the ability to reproduce scenes, resulting in a separation between perception and decision-making, making it difficult to fully test the functional defects of the algorithm.

[0005] In view of the above limitations, there is an urgent need for a new VIL test method that combines the application of real vehicle sensors to fully reproduce the real test scenario and achieve effective evaluation of the perception layer of smart cars. The present invention proposes an innovative test scheme that not only inherits the advantages of traditional VIL testing, but also specifically tests the perception layer of smart vehicles, thereby improving the comprehensiveness and accuracy of the test. This improved test method can better simulate actual driving conditions and ensure the safe and reliable development of autonomous driving technology. Summary of the invention

[0006] To solve the above problems, the present invention provides a vehicle-in-the-loop testing method based on visual augmented reality. First, a real vehicle positioning and motion information acquisition system is established to obtain high-precision motion information of the vehicle; then, the image information and motion information of the on-board camera are synchronously acquired, with the purpose of obtaining the camera output and vehicle motion information at the same timestamp; secondly, the camera's internal and external reference information are combined with the synchronously acquired motion information to calculate the camera's position and posture at each moment; the calculated position and posture are synchronously input into the simulation software to obtain a virtual camera output that is consistent with the real camera in time and space; the real acquired camera image is used as the background, and the traffic participants provided by the virtual simulation camera are used as the foreground, and the fused image information from the augmented reality perspective is obtained through an image fusion algorithm; finally, the fused image is input into the perception layer, and after the perception result is settled, it is input into the decision control layer, and the decision instruction is output to the vehicle's actuator, so that the vehicle motion information is changed and is acquired by the high-precision information acquisition system, thereby completing a round of information circulation.

[0007] The technical solution of the present invention is described as follows in conjunction with the accompanying drawings:

[0008] A vehicle-in-the-loop testing method based on visual augmented reality comprises the following steps:

[0009] Step 1: Use RT3002 and Qianxun Positioning RTK service to build a positioning and motion information collection system that can obtain high-precision vehicle motion information; the system can record the vehicle's position and dynamic parameters in real time and accurately, ensuring the accuracy of the collected vehicle positioning and motion data;

[0010] Step 2: Use the time synchronization mechanism in ROS to synchronously collect the image information of the on-board camera and the corresponding vehicle motion information to ensure the consistency of the camera output and the vehicle motion state at the same timestamp; this step is crucial for subsequent image fusion;

[0011] Step 3: Calculate the specific position and posture (pose) of the camera at each moment by using the initial external parameter information of the camera and the motion information collected synchronously; this process ensures that the behavior of the camera in the virtual environment is completely consistent with that of the camera in the real world;

[0012] Step 4: synchronously input the calculated camera pose information into the simulation software, design the camera parameters of the virtual camera to be consistent with the real camera, and generate a virtual camera output that is completely consistent with the real camera in time and space;

[0013] Step 5: Using the real camera image as the background and the traffic participants provided by the virtual simulation camera as the foreground, an advanced image fusion algorithm is used to generate a fused image from the perspective of augmented reality. This fused image not only retains the details of the real world, but also adds simulated dynamic elements, greatly enriching the complexity and authenticity of the test scene.

[0014] Step 6: Input the fused image into the perception layer of the vehicle for processing to test the perception layer's ability to perceive the fusion of real scenes and virtual dynamic elements and its ability to adapt to complex environments. The fused image not only fully reproduces the real driving scene by retaining the background details of the real world and superimposing the dynamic traffic participants in the virtual environment, but also overcomes the problem of separation between perception and decision-making in traditional VIL testing, allowing the perception layer to accurately evaluate the object recognition and tracking performance in dynamic environments. The perception results are then passed to the decision control layer, which generates corresponding decision instructions and sends them to the vehicle's actuators. The vehicle adjusts its motion state according to these instructions, and the new motion information is captured again by the high-precision information acquisition system to ensure the closed-loop consistency of vehicle behavior and perception decisions.

[0015] Furthermore, the specific method of step 2 is as follows:

[0016] 21) Receive high-precision motion information acquisition module data and camera data uniformly through the ROS system, wherein the high-precision motion information acquisition module data is in CAN format, and the camera data acquisition format is sensor_msgs / Image; the high-precision motion information acquisition module contains the vehicle's latitude and longitude information and heading, pitch, and roll angle information;

[0017] 22) Use the TimeSynchronizer module in the ROS system to time synchronize the data of the above-mentioned high-precision motion information acquisition module and the camera data to ensure that the different signals received are aligned at the same timestamp, thereby achieving accurate time synchronization processing of the signals.

[0018] Furthermore, the specific method of step three is as follows:

[0019] 31) Calibrate the intrinsic parameters and initial extrinsic parameters of the real camera. The intrinsic parameter calibration uses Zhang Youding calibration method to obtain the intrinsic parameter matrix of the camera; the extrinsic parameter calibration first measures the world coordinate points and the corresponding pixel coordinates, and then solves the rotation matrix R and translation vector t of the camera in the coordinate system with RT as the origin through the PnP algorithm;

[0020] 32) Calculate the vehicle's translation vector using the longitude and latitude information; calculate the vehicle's rotation matrix using the heading, pitch, and roll angle information; combine the calibrated vehicle camera's intrinsic parameter matrix with the initial extrinsic parameter information to calculate the camera's position in the world coordinate system.

[0021] Furthermore, the specific method of step 4 is as follows:

[0022] 41) Using the calibrated camera internal parameter information, set the parameters of the virtual camera: principal point coordinates, focal length, camera imaging height and width, field of view, etc. to be consistent with the real camera;

[0023] 42) combining the calculated pose of the real camera, controlling the movement of the virtual camera to be consistent with the calculated pose of the real camera, thereby obtaining the output of the virtual world under the same viewing angle as the real camera, including a virtual world RGB image and a virtual world semantic segmentation image;

[0024] Furthermore, the specific method of step five is as follows:

[0025] 51) Design an image extraction method for virtual traffic participants including sun shadows;

[0026] 52) Perform image fusion on the background image captured by the real camera and the images of traffic participants in the virtual simulation world.

[0027] Furthermore, the specific method of step 51) is as follows:

[0028] 511) Calculate the solar altitude angle and solar azimuth angle at that time according to the longitude and latitude of the experimental site and the test time, so as to obtain the solar shadow with the same orientation as that in the real world in the virtual simulation world;

[0029] The formula for calculating the sun angle is:

[0030] h=arcsin(sin(φ)·sin(δ)+cos(φ)·cos(δ)·cos(H))

[0031] The solar azimuth is calculated by the following formula:

[0032]

[0033] Where φ is the latitude of the observation point; δ is the declination angle of the sun; H is the solar hour angle;

[0034] After generating the sun shadow, extract the corresponding sun shadow part;

[0035] By converting the RGB image into LAB space, when the sum of the average values ​​of the A and B channels is less than the given threshold ab_threshold, the pixel values ​​that satisfy L≤[mean(L)-std(L) / 3] are classified as shadow parts, and other pixels are classified as non-shadow pixels, where mean(L) represents the mean of the pixel channel and std(L) represents the standard deviation of the pixel channel; when the sum of the average values ​​of the A and B channels is greater than the given threshold ab_threshold, the parts of the L and B channels that are below the channel threshold are classified as shadow areas;

[0036] 512) The extraction of traffic participants is obtained through semantic segmentation image and original RGB image; according to the semantic segmentation result, the category of the target object is selected and a binary template is generated, as shown below:

[0037]

[0038] Where S(x,y) is the label at position (x,y) in the semantic segmentation result; M(x,y) is the generated binary template;

[0039] For the overall binary image acquisition process of the traffic object area and the shadow area, first calculate the shadow mask to remove the part overlapping with the traffic object:

[0040]

[0041] In the formula, is the complement of the traffic object mask; M shadow is the mask for the shadow;

[0042] Next, merge the de-overlapped shadow mask and the vehicle mask:

[0043] M combined =M object ∪M refined_shadow

[0044] Where M combined is a mask image of the overall area containing traffic objects and shadows;

[0045] In summary, the whole process is expressed as:

[0046]

[0047] Using the generated binary template M, perform a pixel-by-pixel mask operation on the original image to extract the image area of ​​the target object, as shown below:

[0048] O(x,y)=I(x,y)×M(x,y);

[0049] Where I(x,y) is the original image; M(x,y) is the mask image of the entire area including traffic objects and shadows;

[0050] Further, the specific method of step 52) is as follows:

[0051] 521) Gaussian convolution is performed on the mask image of the traffic participant object containing the sun shadow, so that the edge area of ​​the mask produces a gradient. The Gaussian function in two-dimensional space is expressed as:

[0052]

[0053] Where (x, y) is the offset relative to the center pixel; σ is the standard deviation of the normal distribution; r is the blur radius (r 2 =x 2 +y 2 ); G(x,y) is the weight of the position (x,y);

[0054] The Gaussian blur process of a two-dimensional image is as follows: Applying Gaussian blur on an image is achieved by convolution of the Gaussian kernel matrix with the image. The whole process is expressed using the following formula:

[0055]

[0056] Where I(i,j) is the pixel value of the original image at position (i,j); I'(i,j) is the pixel value of the blurred image at position (i,j); G(x,y) is the weight of position (x,y); k is the radius of the kernel;

[0057] 522) The mask image after Gaussian convolution is alpha-fused with the corresponding RGB image and the foreground image collected by the real camera to achieve smooth fusion between the foreground and the background. The specific formula is:

[0058] x output =x foreground *x mask +x background *(1-x mask )

[0059] In the formula, x output is the fused image; x foreground is the foreground part; x background is the background part; x mask is the mask; x mask The actual calculation of the pixel value will be normalized from [0,255] to [0,1]; when x mask = 0, the output pixel value belongs to the background; when x mask =1, the output pixel belongs to the foreground; when 0<x mask<1, the output pixel value is a fusion of foreground and background.

[0060] Furthermore, the specific method of step six is ​​as follows:

[0061] 61) Input the augmented reality fusion image of step five into the perception layer of the vehicle for processing to test the perception layer's ability to perceive the fusion of real scenes and virtual dynamic elements and its ability to adapt to complex environments. The fusion image not only fully reproduces the real driving scene by retaining the background details of the real world and superimposing the dynamic traffic participants in the virtual environment, but also overcomes the problem of separation between perception and decision-making in traditional VIL testing, allowing the perception layer to accurately evaluate the object recognition and tracking performance in a dynamic environment; the perception layer calculates the motion information, speed, and distance of the corresponding traffic participants; the perception layer sends the calculated relative speed, relative distance, etc. to the decision control module through UDP communication.

[0062] 62) After receiving the perceived signal, the decision control module calculates the corresponding control signal and sends it to the vehicle's wire-controlled actuator, thereby changing the vehicle's motion information. At the same time, the high-precision information acquisition system collects new motion information, thereby completing the next round of updates for the entire test system. Through this process, this test method verifies the comprehensiveness and accuracy of the perception layer, while ensuring that the scene reproduction capability is closer to actual driving conditions, thereby improving the safety and reliability of autonomous driving technology.

[0063] The beneficial effects of the present invention are:

[0064] 1) The present invention can enhance the testing capability of the perception layer, and use augmented reality technology to integrate the real-world background with the foreground of virtual traffic participants, providing a more complex and realistic testing environment for the perception layer of smart cars, and significantly improving the testing effect of the perception layer algorithm;

[0065] 2) The present invention can improve the comprehensiveness and repeatability of the test. The fused image under the augmented reality perspective not only retains the details of the real world, but also adds simulated dynamic elements, making the test scene richer and more diverse. In addition, the test process has good repeatability, which is conducive to in-depth analysis and improvement of autonomous driving algorithms;

[0066] 3) The present invention optimizes test efficiency and cost-effectiveness. Compared with traditional real vehicle testing, it reduces the high cost and the need for a long period of time, while improving test efficiency. By simulating diverse traffic scenarios, more diverse test cases can be completed in a shorter time, reducing resource consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0068] Figure 1 It is a schematic diagram of the process of the present invention;

[0069] Figure 2 This is the schematic diagram of the high-precision information acquisition system;

[0070] Figure 3 It is a time synchronization diagram of camera image information and high-precision motion information;

[0071] Figure 4 Extract flow chart for sun shadow;

[0072] Figure 5 A schematic diagram of an image of a traffic participant with a sun shadow;

[0073] Figure 6 This is the principle diagram of Alpha fusion based on feather mask;

[0074] Figure 7 This is a schematic diagram of an image that shows the fusion of virtuality and reality from an augmented reality perspective. DETAILED DESCRIPTION

[0075] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0076] See also Figure 1 This embodiment provides a vehicle-in-the-loop testing method based on visual augmented reality, comprising the following steps:

[0077] Step 1: Read Figure 2, a high-precision vehicle motion information collection system was built by combining the RT3002 device with the RTK differential service of Qianxun Positioning. The RT3002 device has a built-in high-performance GPS positioning module that can receive satellite signals to determine the position and speed of the vehicle; at the same time, it is also equipped with an inertial measurement unit (IMU) consisting of three mutually perpendicular single-axis accelerometers and three mutually perpendicular single-axis gyroscopes, which is used to collect the vehicle's longitudinal and lateral acceleration data, and calculate the heading, pitch and roll attitude information in real time according to the angular velocity of the carrier rotating around each coordinate axis. Through the embedded 24th-order Kalman filter algorithm, the RT3000 V3 can automatically correct the errors caused by inaccurate antenna installation position; RT3002 can achieve centimeter-level positioning accuracy by fusing the positioning data it obtains with the differential data provided by Qianxun Zhicun-FindCM, and after the differential decomposition process.

[0078] Step 2: See Figure 3 , using the time synchronization mechanism in ROS, synchronously collect the image information of the on-board camera and the corresponding vehicle motion information, as follows:

[0079] The high-precision motion information acquisition module and camera topics are uniformly received through the ROS system, wherein the data of the high-precision motion information acquisition module is in CAN format, and the format of camera data acquisition is sensor_msgs / Image; the high-precision motion information acquisition module contains the latitude and longitude information and heading, pitch, and roll angle information of the vehicle;

[0080] The above topics are uniformly received through the TimeSynchronizer in the ROS system. Only when all topics have the same timestamp will a callback function of the synchronization result be generated, so that the camera signal and the CAN signal output by the RT are synchronized in time.

[0081] Step 3: Use the initial external parameter information of the camera and the motion information collected synchronously to calculate the specific position and posture or position of the camera at each moment, as follows:

[0082] The first is the calibration of the initial external parameters of the camera. The calibration of the initial external parameters of the camera mainly determines the position of the camera in the coordinates with the vehicle inertial navigation system-RT device as the origin, including the camera's position (x, y, z) and attitude angle (heading, pitch, roll). The present invention achieves the calibration of the initial external parameters of the vehicle camera by measuring the coordinates of the known points on the calibration plate and the pixel coordinates on the corresponding camera imaging picture using the pnp algorithm.

[0083] Secondly, since the high-precision information collected by the vehicle is the latitude and longitude information of the vehicle, it is not suitable for calculating the translation of the vehicle, so the longitude and latitude information needs to be converted into plane coordinates in the UTM coordinate system. The longitude and latitude are (λ, φ), the unit is radian, and the UTM coordinates are (E, N), the unit is km. The specific conversion formula is as follows:

[0084] k0=0.9996, E0=500km, e=0.0818192, a=6378.137km

[0085]

[0086] λ0=(zoneNumber-1)*6-180+3

[0087]

[0088] Where λ is the longitude; is the latitude; k0 is the standard scale factor; E0 is the easting value of the central meridian; N0 is the northing value of the meridian starting point; e is the eccentricity of the earth; a is the radius of the major axis of the earth ellipsoid; T is the square of the tangent of the latitude; C is the angle between the longitude and the latitude; zoneNumber is the UTM projection zone number; v is the curvature radius of the latitude; λ0 is the central meridian of each zone; A is the difference between the longitude and the central meridian multiplied by the cosine latitude, which is used to calculate the lateral offset; s is the arc length of the meridian; E is the easting coordinate in the UTM coordinate system; N is the northing coordinate in the UTM coordinate system.

[0089] After obtaining the position of the vehicle in the plane coordinate system, the translation vector of the vehicle at the selected world coordinate point can be calculated, which can be described as:

[0090] t vehicle_to_world =(x vehicle -x origin ,y vehicle -y origin ,z vehicle -z origin )

[0091] In the formula, (x origin ,y origin ,z origin ) is the selected world origin coordinate.

[0092] The rotation matrix of the vehicle in the world coordinate system can be calculated based on the pitch, roll and heading angles of the vehicle:

[0093]

[0094] R vechicle_to_world =R z(heading)·R y (pitch) R x (roll)

[0095] After calculating the rotation matrix and translation vector of the vehicle in the world coordinate system by the formula, combined with the initial external parameters of the camera in the previous section, the position and posture of the camera in the world coordinate system can be calculated, which is specifically expressed as:

[0096]

[0097] In the formula, R cam_veh and t cam_veh They are the rotation matrix and translation vector of the camera relative to the vehicle, which can be calculated using the initial parameters of the camera calibration.

[0098] Step 4: Input the calculated camera pose information into the simulation software synchronously, design the camera parameters of the virtual camera to be consistent with the real camera, and generate a virtual camera output that is completely consistent with the real camera in time and space, as follows:

[0099] Using the calibrated camera intrinsic parameter information, set the parameters of the virtual camera: principal point coordinates, focal length, camera imaging height and width, field of view, etc. are consistent with the real camera; combined with the real camera pose calculated in step 3, control the movement of the virtual camera to be consistent with the calculated real camera pose, so that the output of the virtual world under the same perspective as the real camera can be obtained, including the virtual world RGB image and the virtual world semantic segmentation image.

[0100] Step 5: Using the real camera image as the background and the traffic participants provided by the virtual simulation camera as the foreground, the image fusion algorithm is used to generate a fused image from the perspective of augmented reality, as follows:

[0101] First, the corresponding solar altitude angle and azimuth are calculated according to the latitude and longitude of the test site and the test time, so as to obtain the shadow generation direction that conforms to the real world. The formula for calculating the solar altitude angle and azimuth is as follows:

[0102] h=arcsin(sin(φ)·sin(δ)+cos(φ)·cos(δ)·cos(H))

[0103]

[0104] Where φ is the latitude of the observation point; δ is the declination of the sun; H is the solar hour angle (determined by local time and geographic location); h is the calculated solar altitude angle; and A is the calculated solar azimuth angle.

[0105] Once the accurate position and angle of the sun are obtained, the sun position parameters in the virtual environment can be set based on these data in the 3D modeling and rendering software. The core goal of this process is to ensure that the generated shadows can truly reflect the effects of sunlight in the real world, so as to ensure that the generated shadows are reasonable and in line with the reality.

[0106] After generating the sun shadow, how to effectively extract and accurately identify the shadow area is one of the key steps to achieve realistic lighting effects in virtual scenes. Figure 4 ,The process of extracting sun shadows using the LAB color model,first convert the RGB image into an equivalent LAB image, and calculate the pixel mean and standard deviation of the L, A, and B channels in the image respectively. The mean of the L channel represents the overall brightness level in the image, and the standard deviation reflects the distribution range of the brightness. A lower L value usually indicates that the brightness of the area is lower, which may be a shadow area.

[0107] The mean of the A and B channels can reflect the color information in the image, especially in the shadow area. Due to the angle of sunlight and the occlusion of objects, the shadow part may present a relatively single tone, and the values ​​of the A and B channels are relatively close. When the sum of the mean values ​​of the A and B channels is less than the given threshold ab_threshold, the pixel values ​​that meet L≤[mean(L)-std(L) / 3] are classified as shadow parts, and other pixels are classified as non-shadow pixels, where mean(L) represents the mean of the pixel channel and std(L) represents the standard deviation of the pixel channel; when the sum of the mean values ​​of the A and B channels is greater than the given threshold ab_threshold, at this time, the extraction of the shadow area needs to comprehensively consider the brightness information of the L and B channels. In this case, the part of the L and B channels that is lower than the channel threshold can be classified as the shadow area. The threshold is usually set to the mean of the channel minus 1 / 3 of the standard deviation of the channel. In this way, pixels with low brightness and colors close to the shadow area can be more accurately identified.

[0108] The extraction of traffic participants is mainly obtained by combining semantic segmentation images with original RGB images. Semantic segmentation images can obtain the category label of each pixel in the image. According to the semantic segmentation results, the category of the target object is selected and a binary template is generated. In this template, the pixels belonging to the target object are marked as 1, and the remaining background pixels are marked as 0. This process can be achieved through threshold operation, the formula is:

[0109]

[0110] The process of obtaining the overall binary image of the traffic object area and the shadow area is as follows:

[0111] First, calculate the shadow mask and remove the part that overlaps with traffic objects:

[0112]

[0113] Next, the de-overlapped shadow mask and vehicle mask are merged as follows:

[0114] M combined =M object ∪M refined_shadow

[0115] Where M combined is the mask image of the overall area containing traffic objects and shadows.

[0116] In summary, the whole process can be expressed as:

[0117]

[0118] Using the generated binary template M, the original image I is masked pixel by pixel to extract the image area of ​​the target object, such as Figure 5 The specific operations are as follows:

[0119] O(x,y)=I(x,y)×M(x,y)

[0120] Where I(x,y) is the pixel value at position (x,y) in the original image; O(x,y) is the pixel at position (x,y) in the extracted target object image.

[0121] The mask image M has been obtained combined , the corresponding RGB image O(x,y), and the real background image, the next step is to fuse the images. The fusion process is referred to Figure 6 .

[0122] The mask image M combined Perform Gaussian convolution to obtain the binary mask of edge feathering. The formula is as follows:

[0123]

[0124] Where I(i,j) is the pixel value of the original image at position (i,j); I'(i,j) is the pixel value of the blurred image at position (i,j); G(x,y) is the weight of position (x,y); and k is the radius of the kernel.

[0125] Finally, the virtual world image and the real background image are combined with the feathered mask for Alpha fusion to obtain the final augmented reality image, such as Figure 7 As shown:

[0126] x output =x foreground *xmask +x background *(1-x mask )

[0127] In the formula, x output is the fused image; x foreground is the foreground part; x background is the background part; x mask is the mask. mask The actual calculation of the pixel value will be normalized from [0,255] to [0,1]. The output pixel value belongs to the background; when x mask =1, the output pixel belongs to the foreground; when 0<x mask <1, the output pixel value is a fusion of foreground and background.

[0128] Step 6: Input the fused image into the perception layer of the vehicle for processing to test the perception layer's ability to perceive the fusion of real scenes and virtual dynamic elements and its ability to adapt to complex environments. The fused image fully reproduces the real driving scene by retaining the background details of the real world and superimposing the dynamic traffic participants in the virtual environment, so that the perception layer can accurately evaluate the object recognition and tracking performance in a dynamic environment. The perception results are then passed to the decision control layer, which generates corresponding decision instructions and sends them to the vehicle's actuators. The vehicle adjusts its motion state according to these instructions, and the new motion information is captured again by the high-precision information acquisition system to ensure the closed-loop consistency of vehicle behavior and perception decisions, as follows:

[0129] 61) Input the augmented reality fusion image of step 5 into the perception layer of the vehicle for processing to test the perception layer's perception ability of the fusion scene of real scenes and virtual dynamic elements and its adaptability to complex environments. The fusion image not only fully reproduces the real driving scene by retaining the background details of the real world and superimposing the dynamic traffic participants in the virtual environment, but also overcomes the problem of separation between perception and decision-making in traditional VIL testing, so that the perception layer can accurately evaluate the object recognition and tracking performance in a dynamic environment; the perception layer calculates the movement information, speed, and distance of the corresponding traffic participants; the perception layer sends the calculated relative speed, relative distance, etc. to the decision control module through UDP communication;

[0130] 62) After receiving the perceived signal, the decision control module calculates the corresponding control signal and sends it to the vehicle's wire-controlled actuator, thereby changing the vehicle's motion information. At the same time, the high-precision information acquisition system collects new motion information, thereby completing the next round of updates for the entire test system. Through this process, this test method verifies the comprehensiveness and accuracy of the perception layer, while ensuring that the scene reproduction capability is closer to actual driving conditions, thereby improving the safety and reliability of autonomous driving technology.

[0131] In summary, the present invention effectively solves the problem of insufficient testing of the perception layer of smart cars by traditional testing methods by introducing key technologies such as high-precision positioning and motion information acquisition, synchronized image and motion information acquisition, accurate calculation of camera pose, generation of consistent virtual camera output, image fusion, and perception layer testing and decision control.

[0132] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A vehicle-in-the-loop testing method based on visual augmented reality, characterized in that: The following steps are involved: Step 1: Use RT3002 and Qianxun Positioning RTK service to build a positioning and motion information collection system; Step 2: Use the time synchronization mechanism in ROS to synchronously collect the image information of the on-board camera and the corresponding vehicle motion information; Step 3: Calculate the specific position and posture or position of the camera at each moment by using the initial external parameter information of the camera and combining it with the motion information collected synchronously; Step 4: synchronously input the calculated camera pose information into the simulation software, design the camera parameters of the virtual camera to be consistent with the real camera, and generate a virtual camera output that is completely consistent with the real camera in time and space; Step 5: Using the real camera image as the background and the traffic participants provided by the virtual simulation camera as the foreground, a fused image under the augmented reality perspective is generated through an image fusion algorithm; Step 6: Input the fused image into the vehicle's perception layer for processing to test the perception layer's ability to perceive the fusion of real scenes and virtual dynamic elements and its ability to adapt to complex environments. The fused image fully reproduces the real driving scene by retaining the background details of the real world and superimposing the dynamic traffic participants in the virtual environment, so that the perception layer can accurately evaluate the object recognition and tracking performance in a dynamic environment. The perception results are then passed to the decision control layer, which generates corresponding decision instructions and sends them to the vehicle's actuators. The vehicle adjusts its motion state according to these instructions. At the same time, the new motion information is captured again by the high-precision information acquisition system to ensure the closed-loop consistency of vehicle behavior and perception decisions.

2. A vehicle-in-the-loop testing method based on visual augmented reality according to claim 1, characterized in that: The specific method of step 2 is as follows: 21) Receive high-precision motion information acquisition module data and camera data uniformly through the ROS system, wherein the high-precision motion information acquisition module data is in CAN format, and the camera data acquisition format is sensor_msgs / Image; the high-precision motion information acquisition module contains the vehicle's latitude and longitude information and heading, pitch, and roll angle information; 22) Use the TimeSynchronizer module in the ROS system to time synchronize the data of the above-mentioned high-precision motion information acquisition module and the camera data to ensure that the different signals received are aligned at the same timestamp, thereby achieving accurate time synchronization processing of the signals.

3. A vehicle-in-the-loop testing method based on visual augmented reality according to claim 2, characterized in that: The specific method of step three is as follows: 31) Calibrate the intrinsic parameters and initial extrinsic parameters of the real camera. The intrinsic parameter calibration uses Zhang Youding calibration method to obtain the intrinsic parameter matrix of the camera; the extrinsic parameter calibration first measures the world coordinate points and the corresponding pixel coordinates, and then solves the rotation matrix R and translation vector t of the camera in the coordinate system with RT as the origin through the PnP algorithm; 32) Calculate the vehicle's translation vector using the longitude and latitude information; calculate the vehicle's rotation matrix using the heading, pitch, and roll angle information; combine the calibrated vehicle camera's intrinsic parameter matrix with the initial extrinsic parameter information to calculate the camera's position in the world coordinate system.

4. The vehicle-in-the-loop testing method based on visual augmented reality according to claim 1, characterized in that: The specific method of step 4 is as follows: 41) Using the calibrated camera internal parameter information, set the parameters of the virtual camera: principal point coordinates, focal length, camera imaging height and width, field of view, etc. to be consistent with the real camera; 42) Combined with the calculated pose of the real camera, the movement of the virtual camera is controlled to be consistent with the calculated pose of the real camera, so as to obtain the output of the virtual world under the same perspective as the real camera, including the virtual world RGB image and the virtual world semantic segmentation image.

5. The vehicle-in-the-loop testing method based on visual augmented reality according to claim 1, characterized in that: The specific method of step five is as follows: 51) Design an image extraction method for virtual traffic participants including sun shadows; 52) Perform image fusion on the background image captured by the real camera and the images of traffic participants in the virtual simulation world.

6. A vehicle-in-the-loop testing method based on visual augmented reality according to claim 5, characterized in that: The specific method of step 51) is as follows: 511) Calculate the solar altitude angle and solar azimuth angle at that time according to the longitude and latitude of the experimental site and the test time, so as to obtain the solar shadow with the same orientation as that in the real world in the virtual simulation world; The formula for calculating the sun angle is: h=arcsin(sin(φ)·sin(δ)+cos(φ)·cos(δ)·cos(H)) The solar azimuth is calculated by the following formula: Where φ is the latitude of the observation point; δ is the declination angle of the sun; H is the solar hour angle; After generating the sun shadow, extract the corresponding sun shadow part; By converting the RGB image into LAB space, when the sum of the average values ​​of the A and B channels is less than the given threshold ab_threshold, the pixel values ​​that satisfy L≤[mean(L)-std(L) / 3] are classified as shadow parts, and other pixels are classified as non-shadow pixels, where mean(L) represents the mean of the pixel channel and std(L) represents the standard deviation of the pixel channel; when the sum of the average values ​​of the A and B channels is greater than the given threshold ab_threshold, the parts of the L and B channels that are below the channel threshold are classified as shadow areas; 512) The extraction of traffic participants is obtained through semantic segmentation image and original RGB image; according to the semantic segmentation result, the category of the target object is selected and a binary template is generated, as shown below: Where S(x,y) is the label at position (x,y) in the semantic segmentation result; M(x,y) is the generated binary template; For the overall binary image acquisition process of the traffic object area and the shadow area, first calculate the shadow mask to remove the part overlapping with the traffic object: In the formula, is the complement of the traffic object mask; M shadow is the mask for the shadow; Next, merge the de-overlapped shadow mask and the vehicle mask: M combined =M object ∪M refined_shadow Where M combined is a mask image of the overall area containing traffic objects and shadows; In summary, the whole process is expressed as: Using the generated binary template M, perform a pixel-by-pixel mask operation on the original image to extract the image area of ​​the target object, as shown below: O(x,y)=I(x,y)×M(x,y); Where I(x, y) is the original image; M(x, y) is the mask image of the entire area including traffic objects and shadows.

7. The vehicle-in-the-loop testing method based on visual augmented reality according to claim 5, characterized in that: The specific method of step 52) is as follows: 521) Gaussian convolution is performed on the mask image of the traffic participant object containing the sun shadow, so that the edge area of ​​the mask produces a gradient. The Gaussian function in two-dimensional space is expressed as: Where (x, y) is the offset relative to the center pixel; σ is the standard deviation of the normal distribution; r is the blur radius (r 2 =x 2 +y 2 ); G(x,y) is the weight of the position (x,y); The Gaussian blur process of a two-dimensional image is as follows: Applying Gaussian blur on an image is achieved by convolution of the Gaussian kernel matrix with the image. The whole process is expressed using the following formula: Where I(i,j) is the pixel value of the original image at position (i,j); I'(i,j) is the pixel value of the blurred image at position (i,j); G(x,y) is the weight of position (x,y); k is the radius of the kernel; 522) The mask image after Gaussian convolution is alpha-fused with the corresponding RGB image and the foreground image collected by the real camera to achieve smooth fusion between the foreground and the background. The specific formula is: x output =x foreground *x mask +x background *(1-x mask ) In the formula, x output is the fused image; x foreground is the foreground part; x background is the background part; x mask is the mask; x mask The actual calculation of the pixel value will be normalized from [0,255] to [0,1]; when x mask = 0, the output pixel value belongs to the background; when x mask =1, the output pixel belongs to the foreground; when 0<x mask <1, the output pixel value is a fusion of foreground and background.

8. The vehicle-in-the-loop testing method based on visual augmented reality according to claim 1, characterized in that: The specific method of step six is ​​as follows: 61) The augmented reality fusion image of step 5 is input into the perception layer of the vehicle for processing to test the perception layer's perception ability of the fusion scene of the real scene and the virtual dynamic elements and its adaptability to the complex environment; the fusion image retains the background details of the real world and superimposes the dynamic traffic participants in the virtual environment, so that the perception layer can accurately evaluate the object recognition and tracking performance in the dynamic environment; the perception layer calculates the motion information, speed, and distance of the corresponding traffic participants; the perception layer sends the calculated relative speed and relative distance to the decision control module through UDP communication; 62) After receiving the sensed signal, the decision control module calculates the corresponding control signal and sends it to the vehicle's wire-controlled actuator, thereby changing the vehicle's motion information. At the same time, the high-precision information acquisition system collects new motion information, thereby completing the next round of updates for the entire test system.

Images