An automatic driving test platform based on virtual-real integration technology

Abstract

This invention discloses an autonomous driving test platform based on virtual-real integration technology, belonging to the field of autonomous vehicle testing technology. The platform is based on a three-level linkage of cloud, edge, and terminal, combining real-world environment testing, sandbox environment testing, and virtual simulation technology. The real-world scenario on the vehicle side represents the actual application environment of the autonomous vehicle, collecting environmental information and providing feedback on the environmental information and virtual-real fusion information. In addition to building a virtual autonomous driving scenario, the cloud-based virtual environment fuses information from the real-world scenario and the sandbox scenario to generate virtual-real fusion data. The sandbox environment, as an edge component, acts as a relay space, bridging the cloud-based virtual environment and the vehicle-side real environment, digitally twinning the autonomous vehicle in the real environment into the virtual simulation environment, and transmitting the virtual-real fusion information to the autonomous vehicle. The sandbox environment, virtual environment, and real environment fully interact to achieve data enhancement and obtain optimal testing results.

Description

Technical Field

[0001] This invention relates to the field of autonomous vehicle testing technology, and in particular to an autonomous driving testing platform based on virtual-real integration technology. Background Technology

[0002] In recent years, autonomous driving technology has developed rapidly, and autonomous vehicles have become a trend in the automotive industry. More and more vehicles equipped with autonomous driving technology are entering people's lives and work. Therefore, the reliability of autonomous driving technology has a significant impact on traffic safety. For safety reasons, autonomous vehicles must undergo thorough testing to prove their stability and safety before they can be put on the road. Currently, there are three main types of testing for autonomous vehicles: real-world testing, virtual simulation testing, and hybrid virtual-real testing.

[0003] Real-world testing refers to deploying autonomous vehicles in real-world application environments. These vehicles acquire real-world information to make decisions, thus testing autonomous driving. Real-world testing closely resembles the actual operating environment of autonomous vehicles, offering high confidence levels. However, real-world testing has three main drawbacks: First, it is costly, requiring the deployment of autonomous driving algorithms onto the vehicles, which are typically expensive. Second, it is difficult to test complex or extreme scenarios, such as storms. In reality, storms occur only a few times a year, reducing testing efficiency. Insufficient testing in extreme scenarios limits the testing process, preventing the exposure of potential problems and posing risks to autonomous vehicles. Third, it is highly dangerous; testing on open roads could lead to traffic accidents if autonomous driving decisions malfunction, posing a significant challenge to road safety.

[0004] Virtual simulation testing refers to testing autonomous vehicles in a virtual environment, where they acquire information from the virtual environment to make decisions. The main drawback of virtual simulation testing is its limited confidence level. The results directly depend on the realism of the test scenario reproduction; the fidelity of the sensor models and simulation signals significantly impacts the confidence level of the results. For example, if the friction coefficient of a certain road material under specific weather conditions cannot be determined, the vehicle's dynamic response cannot be accurately calculated, greatly reducing the reliability of the simulation test results.

[0005] Virtual-real fusion testing involves both virtual simulation testing and real-world environment testing, and its development is still in its early stages. Most virtual-real fusion systems are digital twin systems, and some lack real-time interaction between the virtual and real ends. For example, in the virtual-real fusion scheme proposed in "A Teaching Platform Based on Virtual-Real Fusion Testing Technology for Autonomous Driving," information from the virtual environment is transmitted to the central control unit of the autonomous vehicle, which then controls the vehicle based on this information. However, this scheme only considers the virtual scene during testing, lacking the collection of real-world environmental information, resulting in insufficient interaction between the virtual and real ends. In other words, virtual environment information unidirectionally influences the autonomous vehicle, while real-world scene information is not considered in decision-making and does not affect the virtual environment. Some virtual-real fusion schemes can achieve virtual-real interaction, but they require significant resources and real-time performance is difficult to guarantee. For instance, the virtual-real fusion scheme proposed in "Metaverse Data Fusion System" requires the transmission of information such as obstacles and other vehicles in addition to real-world spatial location information. It only uses data fusion methods, resulting in a very large communication volume, requiring substantial communication resources, making real-time performance difficult to guarantee, and placing high demands on the computing platform. Furthermore, even though real vehicles can respond to virtual objects during testing, the limitation of virtual scene fidelity remains. For example, when setting up a rain scene in the virtual environment, if a rain scene is not simultaneously set up in the real environment, the vehicle's behavior in rainy weather in the real environment cannot be reflected. In other words, the vehicle's performance in rainy weather is mainly reflected in the virtual world, which again highlights the limitations of simulation testing. Summary of the Invention

[0006] In order to at least partially solve one of the technical problems existing in the prior art, the purpose of this invention is to provide an autonomous driving test platform based on virtual-real integration technology.

[0007] The technical solution adopted in this invention is:

[0008] An autonomous driving test platform based on virtual-real integration technology includes:

[0009] The vehicle-side real space includes the autonomous vehicle and the real environment. The autonomous vehicle includes vehicle-side sensors, a vehicle-side computing unit, and a vehicle-side control unit. The vehicle-side sensors are used to collect real environment information. The vehicle-side computing unit is used to receive real environment information as well as virtual-real fusion information and sandbox environment information transmitted through the communication framework, perform calculations based on the received information, and issue control commands to the vehicle-side control unit. The vehicle-side control unit is used to control the vehicle's movement according to the received control commands.

[0010] The edge-side sandbox space includes a sandbox vehicle and a sandbox environment constructed at scale from the autonomous vehicle and the real environment. The sandbox vehicle includes sandbox vehicle sensors, an edge computing unit, and a sandbox vehicle control unit. The sandbox vehicle sensors are used to collect sandbox environment information. The edge computing unit is used to transmit the sandbox environment information to the cloud virtual space through a communication framework, and simultaneously forward the virtual-real fusion information from the cloud virtual space to the vehicle-side computing unit. Before the test begins, the sandbox vehicle is placed in a position relative to the sandbox, consistent with the position of the autonomous vehicle in the real environment. During the test, the state of the sandbox vehicle and the autonomous vehicle remains synchronized.

[0011] The cloud-based virtual space includes virtual vehicles and a virtual environment; the virtual vehicles include virtual sensors, a cloud computing unit, and a virtual controller; the virtual sensors are used to collect virtual environment information; the cloud computing unit performs fusion calculations on the three types of environmental information to obtain virtual-real fusion information, and transmits the virtual-real fusion information to the edge sandbox space through a communication framework.

[0012] Furthermore, the communication framework is a communication framework based on data distribution service technology;

[0013] The communication framework comprises four layers: a communication fusion layer, a perception layer, a simulation layer, and a sandbox layer.

[0014] The communication convergence layer, as the core of the communication framework, includes wireless access points, base stations, and gateways. At the physical level, wireless access points are connected to base stations via 5G or Ethernet cables, and gateways are connected to base stations via bearer networks or optical fibers. At the logical level, data is exchanged based on the DDS data bus.

[0015] The perception layer is used to provide data acquisition of real-world scenarios and communication services with the communication fusion layer.

[0016] The simulation layer is used to provide virtual scene generation and communication services with the communication fusion layer;

[0017] The sand table layer is used to provide data acquisition for the sand table scene and communication services with the communication fusion layer.

[0018] Furthermore, the virtual environment includes static components, dynamic components, and virtual weather; wherein, the static components are obtained from the real environment through digital twins; the dynamic components consist of dynamic traffic elements, including virtual pedestrians and virtual vehicles; and the virtual weather is the weather state of the virtual environment.

[0019] Furthermore, the cloud-based virtual space is deployed on a cloud platform;

[0020] Virtual vehicles are obtained by digital twins of autonomous vehicles, but they do not have the ability to drive autonomously in a virtual environment.

[0021] Virtual sensors include simulated LiDAR and simulated cameras.

[0022] Furthermore, the vehicle-mounted sensors include LiDAR and cameras;

[0023] The lidar is used to collect depth and distance information and point cloud data;

[0024] The camera is used to collect visual information and, by combining it with machine vision algorithms, completes the identification of object types;

[0025] The vehicle-mounted computing unit is equipped with a robot operating system. After receiving real environmental information, virtual-real fusion information, and sand table environment information collected from vehicle-mounted sensors, it generates control commands in conjunction with autonomous driving algorithms.

[0026] Furthermore, the foundation of the virtual-real scene fusion problem of the autonomous driving test platform is to solve the matching and fusion problem of multiple sets of point cloud data. The edge sandbox space is selected as the benchmark, and the point clouds in the real space of the vehicle and the virtual space of the cloud are registered with the point clouds in the edge sandbox space. All data of multiple sets of point clouds are distributed, parallel, and paired.

[0027] Furthermore, in the step of pairwise registration of all data from multiple point clouds, a multi-level fusion method involving data, features, and pose is adopted to solve the virtual-real fusion problem:

[0028] For static objects, denoted as set A, the sampling rate is adjusted by downsampling method S, and point cloud data is fused offline using iterative neighbor registration method;

[0029] For slow-moving objects, denoted as set B, the voxel features of the point cloud are first obtained through the feature extraction network F, and then the voxel features are fused.

[0030] For high-speed moving objects, denoted as set C, the object outlines are extracted from the virtual and real point clouds using a 3D detector D, and then the outlines are fused.

[0031] Furthermore, the expression for solving the virtual-real fusion problem using a multi-level fusion method of data, features, and pose is as follows:

[0032] Virtual-Sandbox Space Cloud-Edge Registration:

[0033]

[0034] Real-world sand table space edge registration:

[0035]

[0036] Where R is the cloud edge rotation matrix, t is the cloud edge coordinate translation vector, and q i ∈Q represents the i-th point cloud in the cloud virtual space, p i ∈P represents the i-th point cloud in the edge sandbox space; W is the edge rotation matrix, v is the edge coordinate translation vector, and o i ∈O represents the i-th point cloud in the vehicle's real space, p i Let be the i-th point cloud in the edge sandbox space; n is the total number of point clouds, and ||||2 is the L2 norm.

[0037] Furthermore, for the case where the coordinate translation vector t and the rotation matrix R are coupled in cost functions at different levels, the alternating direction multiplier method is used for decomposition; the cloud edge rotation matrix R = R is obtained by solving. * The cloud edge coordinate translation vector t = t * End rotation selection matrix W = W * The translation vector of the end coordinates is v = v * .

[0038] Furthermore, the pseudocode for the algorithm to fuse the point cloud P of the edge sandbox space and the point cloud Q of the cloud virtual space is as follows:

[0039] 1) Obtain point cloud P and Q using ROS2 Subscriber

[0040] 2) Initialize List F = []

[0041] 3) For i = 1, ..., n:

[0042]

[0043] F = F.append(p i )

[0044] Else:

[0045] F = F.append(R) * q i +t * )

[0046] 4) Use ROS2 Publisher to output the fused point cloud F;

[0047] The pseudocode for the algorithm to fuse the point cloud P of the edge-side sandbox space and the point cloud O of the vehicle-side virtual space is as follows:

[0048] 1) Obtain point cloud P and O using ROS2 Subscriber

[0049] 2) Initialize List G = []

[0050] 3) For i = 1, ..., n:

[0051]

[0052] G = G.append(p i )

[0053] Else:

[0054] G = G.append(W * o i +v * )

[0055] 4) Use ROS2 Publisher to output the fused point cloud G.

[0056] The beneficial effects of this invention are as follows: The platform of this invention is based on a three-level linkage between cloud, edge, and terminal, combining real-world environment testing, sandbox environment testing, and virtual simulation technology. The sandbox environment, virtual environment, and real environment fully interact to achieve data enhancement and obtain optimal testing results. Specifically, the sandbox environment acts as a relay between the vehicle-side environment and the cloud-based virtual environment, reducing communication load and enhancing real-time performance; furthermore, the sandbox environment supplements the real environment, achieving data enhancement. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of the present invention or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 This is an overall framework diagram of an autonomous driving test platform based on virtual-real integration technology in an embodiment of the present invention;

[0059] Figure 2 This is a schematic diagram of the multi-space high-frequency high-precision registration algorithm in an embodiment of the present invention;

[0060] Figure 3 This is a schematic diagram of the communication framework in an embodiment of the present invention;

[0061] Figure 4 These are diagrams illustrating the operational effects of the virtual scene and sandbox scene in embodiments of the present invention.

[0062] Figure 5 This is a diagram illustrating the overall operation of the platform in an embodiment of the present invention. Detailed Implementation

[0063] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0064] In the description of this invention, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., are based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0065] In the description of this invention, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0066] Furthermore, in the description of this invention, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0067] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0068] To address the shortcomings of existing solutions, this invention proposes an autonomous driving test platform based on virtual-real integration technology. This invention offers the following advantages: 1. It combines the richness of virtual simulation testing with the confidence level of real-world testing, introducing a sandbox scenario as a supplement between virtual simulation and real-world scenarios. Scenarios that are difficult to construct in real-world environments can be implemented in the sandbox. Since the sandbox is a physical environment, its scenarios are closer to real-world scenarios, have higher confidence levels, and compensate for the deficiencies of simulation testing. 2. The three scenarios in this invention are interconnected at multiple levels. The sandbox environment acts as an edge terminal, relaying communication between real-world and virtual-world information, reducing communication load and latency. Virtual-world information can be transmitted to the autonomous vehicle, enabling it to respond based on the constructed virtual environment. Real-world scenario information is also transmitted to the virtual scenario, affecting the virtual vehicle's state; this bidirectional communication enhances data. 3. This invention employs a multi-level fusion method for data, features, and pose, pairwise registration of data from different scenarios. The overall approach involves scenario decomposition followed by layered fusion, resulting in smaller data transmission volume, fewer resources required, and higher real-time performance.

[0069] like Figure 1 As shown, this embodiment provides an autonomous driving test platform based on virtual-real integration technology, including:

[0070] The vehicle-side real space includes the autonomous vehicle and the real environment. The autonomous vehicle includes vehicle-side sensors, a vehicle-side computing unit, and a vehicle-side control unit. The vehicle-side sensors are used to collect real environment information. The vehicle-side computing unit is used to receive real environment information as well as virtual-real fusion information and sandbox environment information transmitted through the communication framework, perform calculations based on the received information, and issue control commands to the vehicle-side control unit. The vehicle-side control unit is used to control the vehicle's movement according to the received control commands.

[0071] The edge-side sandbox space includes a sandbox vehicle and a sandbox environment constructed at scale from the autonomous vehicle and the real environment. The sandbox vehicle includes sandbox vehicle sensors, an edge computing unit, and a sandbox vehicle control unit. The sandbox vehicle sensors are used to collect sandbox environment information. The edge computing unit is used to transmit the sandbox environment information to the cloud virtual space through a communication framework, and simultaneously forward the virtual-real fusion information from the cloud virtual space to the vehicle-side computing unit. Before the test begins, the sandbox vehicle is placed in a position relative to the sandbox, consistent with the position of the autonomous vehicle in the real environment. During the test, the state of the sandbox vehicle and the autonomous vehicle remains synchronized.

[0072] The cloud-based virtual space includes virtual vehicles and a virtual environment; the virtual vehicles include virtual sensors, a cloud computing unit, and a virtual controller; the virtual sensors are used to collect virtual environment information; the cloud computing unit performs fusion calculations on the three types of environmental information to obtain virtual-real fusion information, and transmits the virtual-real fusion information to the edge sandbox space through a communication framework.

[0073] This embodiment designs an autonomous driving test platform based on virtual-real integration technology. It achieves data enhancement through three-level linkage between cloud, edge, and terminal, and uses hierarchical data fusion to achieve low-latency, low-load data interaction, enabling efficient and high-confidence autonomous driving testing. The platform comprises three parts: a real-world vehicle scenario, an edge-based sandbox scenario, and a cloud-based virtual scenario. The autonomous vehicle is deployed in the real environment and digitally twinned in the virtual scenario via edge relay communication. During testing, the autonomous vehicle simultaneously receives information from onboard sensors, sandbox vehicle sensors, and simulated sensors attached to the digital twin vehicle. Information from the real, sandbox, and simulated environments is considered during decision-making. Therefore, platform users can easily set up virtual environments to build test scenarios or sandbox environments, then test the performance of the autonomous vehicle in the real environment, and simultaneously evaluate the state of the sandbox vehicle in the sandbox scenario, completing the testing of the autonomous driving algorithm.

[0074] As an alternative implementation, this embodiment employs pairwise registration and multi-level fusion of data, features, and poses to achieve digital twins of autonomous vehicles in virtual scenarios, synchronizing vehicles in the sandbox scenario with real vehicles. Simultaneously, the sandbox scenario and the virtual scenario receive information from the real scenario, determine the vehicle's status information, and update it accordingly.

[0075] Specifically, this embodiment also employs a communication framework based on DDS technology to address the heterogeneous perception across three different scenarios. This framework enables cross-domain, synchronous, and asynchronous data communication, offering high security and allowing for adjustments to communication resource allocation strategies as needed, resulting in high communication efficiency. To reduce communication requirements, information with lower real-time requirements, such as virtual scene object information, is asynchronously fused. Low-latency communication under low-load conditions is achieved by dynamically adjusting the communication strategy. The sandbox environment, virtual environment, and real environment fully interact to enhance data and achieve optimal testing results.

[0076] The platform described above will be explained in detail below with reference to the accompanying drawings and specific embodiments.

[0077] This embodiment designs an autonomous driving test platform based on virtual-real integration technology. It achieves data enhancement through cloud-edge-device three-level linkage and employs hierarchical data fusion to achieve low-latency, low-load data interaction, enabling efficient and high-confidence autonomous driving testing. The detailed description of the technical solution in this embodiment will be divided into the following four parts: system framework, hardware design, software design, and communication framework.

[0078] (I) System Framework

[0079] Figure 1 The diagram shown is the overall system framework of this patent. The system mainly consists of four components: the real-world vehicle scenario (i.e., Figure 1 The actual space on the vehicle side), and the edge-side sandbox scene (i.e. Figure 1 Edge-based sandbox space), cloud-based virtual scene (i.e. Figure 1 The cloud-based virtual space and the communication framework based on DDS technology.

[0080] The on-vehicle real-world scenario component includes the autonomous vehicle and the physical environment (i.e., the real-world environment). An autonomous vehicle mainly consists of three parts: on-vehicle sensors, an on-vehicle computing unit, and an on-vehicle control unit. On-vehicle sensors are responsible for acquiring and perceiving the real-world environment and transmitting this information to the vehicle control and positioning systems. Commonly used on-vehicle sensors include LiDAR and cameras. LiDAR emits a laser beam towards a target; when the beam encounters an object, it returns. The distance from the object to the LiDAR is calculated based on the return time, obtaining depth information. Cameras can collect visual information about the environment, and combined with machine vision algorithms, can perform functions such as object type recognition. The on-vehicle computing unit is equipped with a Robot Operating System (ROS). It receives real-world environmental information from the on-vehicle sensors, as well as virtual-real fusion information and sandbox environment information transmitted through the DDS communication framework. Based on this environmental information and the autonomous driving algorithm, it issues control commands to the on-vehicle control unit. For example, it fuses point cloud information from three spaces and then uses an adaptive Monte Carlo algorithm to calculate the fused pose, localization, and environmental information. This information, combined with the autonomous driving algorithm, then issues control commands (steering, throttle, braking, etc.) to the on-vehicle control unit. The vehicle-side control unit receives control commands from the vehicle-side computing unit and completes vehicle driving according to the commands.

[0081] The edge computing scenario includes a sandbox vehicle and its environment. The sandbox vehicle and environment are constructed by scaling down the autonomous vehicle to the real-world environment. Similarly, the sandbox vehicle is equipped with sensors such as LiDAR and cameras to perceive the sandbox environment. The edge computing unit transmits sandbox information to the cloud computing unit via a communication framework based on DDS technology, and simultaneously forwards virtual-real fusion information from the cloud computing unit to the vehicle-side computing unit. Furthermore, the materials of the sandbox that may affect the vehicle's state should be consistent with the real-world environment; for example, the sandbox road surface material should be consistent with the real-world road surface material. The state of the sandbox vehicle depends on the state of the autonomous vehicle; before testing begins, the sandbox vehicle's position relative to the sandbox should be consistent with the position of the autonomous vehicle in its real-world environment.

[0082] The cloud-based virtual scene is generated as a proportional digital twin of the real scene in the Carla simulator and deployed on a high-performance computing platform. Generally, all components of the virtual scene have the same scaling ratio as their corresponding parts in the physical scene. The virtual scene includes a virtual vehicle and a virtual environment. The virtual vehicle is a digital twin of an autonomous vehicle but does not have the ability to drive autonomously in the virtual environment. The cloud computing unit fuses vehicle status information, sensor information, and virtual vehicle sensor information to obtain the current vehicle position, which is used for virtual vehicle position updates. Specifically, the cloud computing unit receives point cloud information from LiDAR in different spaces and uses the multi-level fusion algorithm described in this embodiment to fuse the point clouds of the edge sandbox space and the cloud virtual space, as well as the edge sandbox space and the vehicle's real space, to obtain the vehicle's position and attitude information in different spaces, and updates the virtual vehicle position based on this information. Virtual sensors are attached to the virtual vehicle and perceive the virtual environment in which the virtual vehicle is currently located, transmitting the acquired virtual environment information to the vehicle control system through a communication framework. Similar to autonomous vehicles, virtual sensors include simulated LiDAR, simulated cameras, etc.

[0083] The virtual environment comprises static components, dynamic components, and virtual weather. Static components are digital twins of the physical environment, and their positions within the virtual scene remain unchanged during system operation, but their appearance can be adjusted according to testing needs. Commonly used static components include roadside trees, buildings, traffic lights, and road condition signs. Dynamic components consist of dynamic traffic elements, including virtual pedestrians and virtual vehicles. These dynamic traffic elements can move along designated or random routes and possess physical properties. When a digital twin vehicle collides with a dynamic traffic element, the cloud computing unit calculates the physical state of both the digital twin vehicle and the dynamic traffic element after the collision and displays the results, such as rollover and shaking after the collision. This calculation helps assess the severity of the accident, thus providing more information for the verification of autonomous driving algorithms. Virtual weather represents the weather conditions in the virtual environment, including sunny, cloudy, rainy, and windy scenarios, which can reproduce various common and extreme scenarios, facilitating thorough testing of autonomous driving algorithms.

[0084] See Figure 4 , Figure 4 These are screenshots showing the virtual and sandbox scenes in operation. The four screens, from top left clockwise, are: virtual sensor information (simulated camera), real scene, real sensor information (camera), and virtual scene. See also... Figure 5 , Figure 5 The test platform is shown in its overall operational effect diagram. The three scenes are arranged in clockwise order from the top left corner: real scene, virtual scene, and sandbox scene.

[0085] (II) Hardware Design

[0086] In a real-world scenario, the autonomous vehicle used in this embodiment is an autonomous vehicle capable of deploying autonomous driving algorithms, such as a Toyota Prius. The autonomous vehicle is equipped with sensors such as LiDAR and binocular cameras, and includes an onboard AI computing platform that can receive and process sensor information and execute autonomous driving algorithms.

[0087] In this embodiment, the vehicles used are scaled-down versions of passenger cars, and the sand table environment is a scaled-down version of the real environment. The following solutions are adopted to address a series of technical problems caused by the scaling down of the sand table vehicles and environment:

[0088] After being scaled up to passenger car dimensions, the model vehicle will be less than 5m long, less than 2m wide, less than 2m high, and have a wheelbase of less than 3m. The chassis uses an Ackermann structure and features a shock-absorbing suspension system, similar to common passenger car structures. The model vehicle's host computer is an artificial intelligence computing platform, connected to sensors such as LiDAR and binocular cameras, and wirelessly connected to the platform frame via Wi-Fi. This AI computing platform receives and processes sensor information, executes autonomous driving algorithms, and issues control commands. The model vehicle's slave computer receives and executes control commands from the host computer, and features speed feedback and speed closed-loop functions. Furthermore, the slave computer is equipped with a remote control receiver, allowing the vehicle to be controlled via a remote control.

[0089] The lanes in the model environment are scaled up to highway dimensions, with a typical city lane width of 3.5m and an urban expressway width of 3.75m. In addition to lanes, the model environment also includes proportionally scaled-down street trees, road signs, traffic lights, buildings, and other static landmarks. Notably, the scale of the model environment is equivalent to that of an autonomous vehicle.

[0090] (III) Software Design

[0091] Multi-space high-frequency high-precision registration algorithm

[0092] The foundation of this embodiment for solving the virtual-real scene fusion problem lies in addressing the matching and fusion of multiple sets of point cloud data to determine the coordinate translation vector t and rotation matrix R. For autonomous driving platforms, high real-time performance is crucial. Performing pairwise registration of all data from multiple point clouds requires computation... Furthermore, as the number of point clouds requiring registration increases, such as by connecting more autonomous vehicles, the required time increases n-fold. 2 The increasing size of the virtual scene poses a significant challenge to real-time performance. To ensure the system's real-time performance, this embodiment selects the sandbox space as the baseline, and registers all other point clouds with the sandbox space point cloud. In this case, the required time increases by (n-1), which is significantly reduced compared to the aforementioned registration method. Specifically, consider the real scene point cloud O, the sandbox space point cloud P, and the virtual scene point cloud Q:

[0093] O={o1,o2,…}, P={p1,p2,…}, Q={q1,q2,…},

[0094] Specifically, taking the fusion of the point cloud P in the sand table space and the point cloud Q in the virtual scene as an example, we calculate the Euclidean transformation:

[0095]

[0096] The above problem can be solved using the Iterative Closet Point (ICP) method published by PJ Besl of General Motors in *IEEE TPAMI*. The first step is to find corresponding points in the virtual and physical point clouds; the second step is to construct an objective function for these corresponding points using the least squares method and perform iterative optimization. However, the ICP method has extremely high complexity and cannot be directly applied to real-time virtual-physical fusion problems. Therefore, see [link to relevant documentation]. Figure 2 This embodiment proposes a multi-level fusion method for data, features, and pose. The overall approach is to decompose the scene and then fuse them layer by layer:

[0097] 1) For static objects (denoted as set A), and considering accuracy requirements and system performance, the sampling rate is adjusted using downsampling methods such as integer multiple downsampling and fractional multiple downsampling (denoted as function S), and point cloud data is fused offline using the ICP method.

[0098] 2) For slow-moving objects (denoted as set B), voxel features of the point cloud are first obtained through feature extraction networks such as convolution and encoders (denoted as function F), and then the voxels are fused. The resolution and number of voxels can be adjusted according to the complexity requirements of this process.

[0099] 3) For high-speed moving objects (denoted as set C), due to their extremely high information update frequency, data or feature fusion methods cannot be used. 3D detectors such as SECOND, PointPillar, and PointRCNN (denoted as function D) are used to extract object bounding boxes from the virtual and real point clouds respectively, and then the bounding boxes are fused.

[0100] Based on the above analysis, in the multi-level virtual-real scene fusion problem of this embodiment, the edge sandbox space and the cloud virtual space take the following forms:

[0101]

[0102] In the above problem, the cloud edge coordinate translation vector t and the cloud edge rotation matrix R are coupled in cost functions at different levels. Therefore, the Alternating Direction Method of Multipliers (ADMM) can be further used for decomposition. The core of the mathematical derivation of ADMM is the construction of the augmented Lagrangian function, which has the following form:

[0103]

[0104] Where R1, R2, R3, t1, t2, t3 are mirror variables; R 12 ,R 13 ,t 12 ,t13 These are dual variables.

[0105] Similarly, for the edge space and the vehicle-side real space, there is the following augmented Lagrangian function, which takes the following form:

[0106]

[0107] Wherein, the end-edge translation vector v and the end-edge rotation matrix W,W1,W2,W3,v1,v2,v3 are mirror variables; W 12 W 13 ,v 12 ,v 13 These are dual variables.

[0108] Alternating optimization of the aforementioned cost function yields a locally optimal rotation and translation matrix upon convergence. After decomposition, data from each level can be fused separately. The fusion of features and bounding boxes requires association, averaging, and pruning operations: ① Clustering algorithms such as DBSCAN and k-means are used to associate virtual and real targets; ② Heatmaps are used to characterize the detection uncertainty of different targets, and pose calibration is performed based on weighted averaging of the heatmaps; ③ The problem of target overlap is addressed using methods such as the Hungarian algorithm and nonmaximum suppression.

[0109] Similarly, by using the above method to register the real scene point cloud O with the sandbox spatial point cloud P, the point cloud registration of each space can be achieved. Let the cloud edge rotation matrix R = R * The cloud edge coordinate translation vector t = t * End rotation selection matrix W = W * The translation vector of the end coordinates is v = v * Finally, parallel point cloud fusion processing is performed on cloud-edge and edge-segment data. The pseudocode for the algorithm that fuses the point cloud P in the edge sandbox space and the point cloud Q in the cloud virtual space is shown in Table 1 below:

[0110] Table 1

[0111]

[0112] On the other hand, the pseudocode for the algorithm that fuses the point cloud P of the edge sandbox space and the point cloud O of the vehicle virtual space is shown in Table 2 below:

[0113] Table 2

[0114]

[0115] (iv) Communication Framework

[0116] The integration of real-world, virtual, and sandbox scenarios relies on high-frequency, high-precision, and synchronized information. Therefore, a wide-coverage, low-latency communication framework is needed to enable information interaction between different scenarios. This invention, based on Data Distribution Service (DDS) technology, constructs a layered communication framework to achieve information interaction. The framework is as follows: Figure 3 As shown, the communication framework consists of four layers: the communication fusion layer, the perception layer, the simulation layer, and the sandbox layer.

[0117] The communication convergence layer is the core of the communication framework, comprising wireless access points, base stations, and gateways. At the physical layer, wireless access points connect to base stations via 5G or Ethernet cables, while gateways connect to fiber optics via a bearer network. At the logical layer, data is exchanged based on the DDS data bus. DDS is a distributed real-time communication middleware employing a publish-subscribe architecture, emphasizing data as its core and providing rich QoS (Quality of Service) policies to ensure real-time, efficient, and flexible data distribution. It is widely used in industrial control, robotics, and other fields. Furthermore, DDS boasts robust security, offering domain-based and intra-domain security protection. Access control and identity verification can exclude malicious access or attackers from the communication framework. Therefore, DDS technology is well-suited for inter-network communication. In practical implementation, a complete solution can be achieved using a DDS-based ROS2 system. First, members requiring communication register nodes in the ROS2 system, transmitting data in the form of topics. Data providers encapsulate data into topics and broadcast them to the DDS system. Data receivers subscribe to these topics, and when the subscribed information is updated, the receiver receives the topic information from DDS.

[0118] The perception layer provides data acquisition from real-world scenarios and communication services with the communication fusion layer. Sensors attached to the intelligent vehicle, such as LiDAR and cameras, collect information and transmit it to the sand table server via wireless communication methods such as Wi-Fi and DSRC. The vehicle also receives information from other spaces, ensuring consistency between the real-world scene and the sand table scene.

[0119] The simulation layer provides virtual scene generation and communication services with the communication convergence layer. The cloud server cluster connects to the communication convergence layer gateway through the core network to enable interaction between virtual scene information and sandbox scene information.

[0120] The sand table layer provides data acquisition for the sand table scene and communication services with the communication integration layer. The sand table server is connected to the gateway via a private network, enabling the interaction between sand table scene information and information from the real and virtual scenes.

[0121] In summary, through the aforementioned communication framework, data from various spaces fully interact through the communication fusion layer, achieving low-latency alignment of information across different scenarios and ensuring the real-time performance of the autonomous driving test platform.

[0122] In summary, compared with existing technical solutions, the present invention has at least the following advantages and beneficial effects:

[0123] (1) This invention combines the richness of virtual simulation testing with the confidence of real environment testing, and introduces sand table scenarios as a supplement between virtual simulation and real scenarios. Some scenarios that are difficult to build in real scenarios can be realized in sand table scenarios. Sand table is a physical environment, and its scenarios are closer to real scenarios with higher confidence, thus making up for the shortcomings of simulation testing.

[0124] (2) The three scenarios in this invention are linked at multiple levels. The sandbox environment acts as the edge terminal, relaying communication between real environment information and virtual environment information, thereby reducing communication load and latency. Virtual environment information can be transmitted to autonomous vehicles, enabling them to respond based on the constructed virtual environment. Real scene information is also transmitted to the virtual scene, affecting the state of the virtual vehicle. Two-way communication achieves data enhancement.

[0125] (3) This invention employs a pairwise spatial registration and a multi-level fusion method of data, features, and pose. The overall approach is to decompose the scene and then fuse it layer by layer, resulting in a small data transmission volume, low resource requirements, and high real-time performance. For static objects, point cloud data is fused offline using the ICP method. For slow-moving objects, voxel features of the point cloud are first obtained through feature extraction networks such as convolution and encoders, and then fusion is performed on the voxels. The resolution and number of voxels can be adjusted according to the complexity requirements. For high-speed moving objects, due to their extremely high information update frequency, data or feature fusion methods cannot be used. 3D detectors such as SECOND, PointPillar, and PointRCNN (denoted as function D) are used to extract object bounding boxes from the virtual and real point clouds respectively, and then the bounding boxes are fused.

[0126] In the foregoing description of this specification, references to terms such as "one embodiment," "another embodiment," or "some embodiments" indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of the present invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0127] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0128] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. An autonomous driving test platform based on virtual-real integration technology, characterized in that, include: The vehicle-side real space includes the autonomous vehicle and the real environment. The autonomous vehicle includes vehicle-side sensors, a vehicle-side computing unit, and a vehicle-side control unit. The vehicle-side sensors are used to collect information about the real environment. The vehicle-side computing unit is used to receive real-world environmental information and virtual-real fusion information and sand table environment information transmitted through the communication framework, perform calculations based on the received information, and issue control commands to the vehicle-side control unit; the vehicle-side control unit is used to control the vehicle's movement according to the received control commands. The edge-side sandbox space includes a sandbox vehicle and a sandbox environment constructed by scaling down autonomous vehicles and real-world environments. The sandbox vehicle includes sandbox vehicle sensors, an edge computing unit, and a sandbox vehicle control unit. The sandbox vehicle sensors are used to collect sandbox environment information. The edge computing unit is used to transmit sand table environment information to the cloud virtual space through the communication framework, and at the same time forward the virtual-real fusion information from the cloud virtual space to the vehicle computing unit; wherein, before the test begins, the sand table vehicle is placed in a position relative to the sand table and the position of the autonomous vehicle in the real environment; during the test, the state of the sand table vehicle and the autonomous vehicle remains synchronized. The cloud-based virtual space includes virtual vehicles and a virtual environment; the virtual vehicles include virtual sensors, a cloud computing unit, and a virtual controller; the virtual sensors are used to collect virtual environment information; the cloud computing unit performs fusion calculations on the three types of environmental information to obtain virtual-real fusion information, and transmits the virtual-real fusion information to the edge sandbox space through a communication framework.

2. The autonomous driving test platform based on virtual-real integration technology according to claim 1, characterized in that, The communication framework is a communication framework based on data distribution service technology; The communication framework includes a communication fusion layer, a perception layer, a simulation layer, and a sandbox layer. The communication convergence layer, as the core of the communication framework, includes wireless access points, base stations, and gateways; At the physical level, wireless access points are connected to base stations, and gateways are connected to base stations; at the logical level, data is exchanged based on the DDS data bus. The perception layer is used to provide data acquisition of real-world scenarios and communication services with the communication fusion layer. The simulation layer is used to provide virtual scene generation and communication services with the communication fusion layer; The sand table layer is used to provide data acquisition for the sand table scene and communication services with the communication fusion layer.

3. The autonomous driving test platform based on virtual-real integration technology according to claim 1, characterized in that, The virtual environment comprises static components, dynamic components, and virtual weather; wherein, the static components are obtained from the real environment through digital twins; the dynamic components consist of dynamic traffic elements, including virtual pedestrians and virtual vehicles; and the virtual weather is the weather state of the virtual environment.

4. The autonomous driving test platform based on virtual-real integration technology according to claim 1, characterized in that, The cloud-based virtual space is deployed on a cloud platform; Virtual vehicles are obtained by digital twins of autonomous vehicles, but they do not have the ability to drive autonomously in a virtual environment. Virtual sensors include simulated LiDAR and simulated cameras.

5. The autonomous driving test platform based on virtual-real integration technology according to claim 1, characterized in that, The vehicle-mounted sensors include lidar and cameras; The lidar is used to collect depth and distance information and point cloud data; The camera is used to collect visual information and, by combining it with machine vision algorithms, completes the identification of object types; The vehicle-mounted computing unit is equipped with a robot operating system. After receiving real environmental information, virtual-real fusion information, and sand table environment information collected from vehicle-mounted sensors, it generates control commands in conjunction with autonomous driving algorithms.

6. The autonomous driving test platform based on virtual-real integration technology according to claim 1, characterized in that, The foundation of the virtual-real scene fusion problem of the autonomous driving test platform is to solve the matching and fusion problem of multiple sets of point cloud data. The edge sandbox space is selected as the benchmark, and the point clouds in the real space of the vehicle and the virtual space of the cloud are registered with the point clouds in the edge sandbox space. All data of multiple sets of point clouds are distributed, parallel, and paired.

7. The autonomous driving test platform based on virtual-real integration technology according to claim 6, characterized in that, In the step of pairwise registration of all data from multiple point clouds, a multi-level fusion method involving data, features, and pose is adopted to solve the virtual-real fusion problem: For static objects, denoted as set A, the sampling rate is adjusted by downsampling method S, and point cloud data is fused offline using iterative neighbor registration method; For slow-moving objects, denoted as set B, the voxel features of the point cloud are first obtained through the feature extraction network F, and then the voxel features are fused. For high-speed moving objects, denoted as set C, the object outlines are extracted from the virtual and real point clouds using a 3D detector D, and then the outlines are fused.

8. The autonomous driving test platform based on virtual-real integration technology according to claim 7, characterized in that, The expression for solving the virtual-real fusion problem using a multi-level fusion method of data, features, and pose is as follows: Virtual-Sandbox Space Cloud-Edge Registration: Real-world sand table space edge registration: Where R is the cloud edge rotation matrix, t is the cloud edge coordinate translation vector, and q i ∈Q represents the i-th point cloud in the cloud virtual space, p i ∈P represents the i-th point cloud in the edge sandbox space; W is the edge rotation matrix, v is the edge coordinate translation vector, and o i ∈O represents the i-th point cloud in the vehicle's real space, p i Let be the i-th point cloud in the edge sandbox space; n is the total number of point clouds, and ||2| is the L2 norm.

9. An autonomous driving test platform based on virtual-real integration technology according to claim 8, characterized in that, For cases where the coordinate translation vector t and the rotation matrix R are coupled to cost functions at different levels, the alternating direction multiplier method is used for decomposition; the cloud edge rotation matrix R = R is obtained by solving. * The cloud edge coordinate translation vector t = t * End rotation selection matrix W = W * The translation vector of the end coordinates is v = v * .

10. An autonomous driving test platform based on virtual-real integration technology according to claim 6, characterized in that, The pseudocode for the algorithm to fuse the point cloud P of the edge sandbox space and the point cloud Q of the cloud virtual space is as follows: 1) Obtain point cloud P and Q using ROS2 Subscriber 2) Initialize List F = [] 3) For i = 1, ..., n: F=F.append(p i ) Else: F=F.append(R * q i +t * ) 4) Use ROS2 Publisher to output the fused point cloud F; The pseudocode for the algorithm to fuse the point cloud P of the edge-side sandbox space and the point cloud O of the vehicle-side virtual space is as follows: 1) Obtain point cloud P and O using ROS2 Subscriber 2) Initialize List G = [] 3) For i = 1, ..., n: G=G.append(p i ) Else: G=G.append(W * o i +v * ) 4) Use ROS2 Publisher to output the fused point cloud G.

Images