Autonomous driving test platform based on virtual-real integration technology

Abstract

An autonomous driving test platform based on virtual-real integration technology, which platform belongs to the technical field of autonomous car testing. On the basis of cloud-edge-end three-level linkage, the platform combines real environment testing and sand table environment testing with virtual simulation technology. A vehicle-end real scene is an actual environment in which an autonomous vehicle is applied; and environmental information is collected, and feedback is provided for the environmental information and virtual-real fusion information. In a cloud-end virtual environment, in addition to building an autonomous driving virtual scene, data fusion is also performed on information from a real scene and a sand table scene, so as to generate virtual-real fusion data. A sand table environment is used as an edge portion to provide a relay space, which is used for bridging the cloud-end virtual environment to a vehicle-end real environment, digitally twinning the autonomous vehicle in the actual environment to a virtual simulation environment, and transmitting the virtual-real fusion information to the autonomous vehicle. The sand table environment, the virtual environment and the real environment fully interact with one another to implement data enhancement, thereby obtaining an optimal test effect.

Description

An autonomous driving test platform based on virtual-reality integration technology Technical Field

[0001] The present invention relates to the field of autonomous driving vehicle testing technology, and in particular to an autonomous driving test platform based on virtual-reality integration technology. Background Art

[0002] In recent years, autonomous driving technology has developed rapidly, and self-driving cars have become a trend in the automotive industry. More and more vehicles equipped with autonomous driving technology are entering people's daily lives. Therefore, the reliability of autonomous driving technology has a significant impact on traffic safety. For safety reasons, autonomous driving cars must undergo thorough testing to prove their stability and safety before they can be put on the road. Currently, there are three main types of testing for autonomous vehicles: real-world testing, virtual simulation testing, and virtual-reality fusion testing.

[0003] Real-world testing involves deploying autonomous vehicles in a real-world application environment, where they utilize real-world information to make decisions. Real-world testing closely resembles the actual operating environment of autonomous vehicles, providing a high level of confidence. However, real-world testing has three drawbacks: First, testing costs are high. Testing requires deploying autonomous driving algorithms to autonomous vehicles, which are typically expensive. Second, it is difficult to test complex or extreme scenarios, such as stormy weather. In reality, stormy weather only occurs a few times a year, reducing testing efficiency. The inability to test in extreme scenarios also limits testing, preventing potential issues from being exposed, and thus creating risks for autonomous vehicles. Third, testing is highly dangerous. If testing is conducted on open roads, any issues with autonomous driving decisions could lead to traffic accidents, posing a significant challenge to road safety.

[0004] Virtual simulation testing involves testing a virtual autonomous vehicle in a virtual environment, obtaining information about the environment to make decisions, and completing the test. The primary drawback of virtual simulation testing is its limited confidence. Virtual simulation test results directly depend on the authenticity of the test scenario, and the fidelity of sensor models and simulation signals significantly impacts the confidence of the test results. For example, if the friction coefficient of a road material cannot be determined under specific weather conditions, the vehicle's dynamic response cannot be accurately calculated, significantly reducing the credibility of the simulation test results.

[0005] Virtual-reality fusion testing involves both virtual simulation and real-world testing, and its research and development is still in its infancy. Most virtual-reality fusion systems are digital twin systems, and some lack real-time interaction between the virtual and real worlds. For example, the virtual-reality fusion solution proposed in "A Teaching Platform Based on Virtual-Reality Fusion Testing Technology for Autonomous Driving" transmits information from the virtual environment to the autonomous vehicle's central control unit, which then controls the vehicle based on this information. During testing, this solution only considers the virtual scene and lacks real-world information, resulting in insufficient interaction between the virtual and real worlds. In other words, virtual environment information only influences the autonomous vehicle in a one-way manner, while real-world information is not considered in decision-making and has no impact on the virtual environment. Some virtual-reality fusion solutions can achieve virtual-reality interaction, but they require significant resources and are difficult to guarantee real-time performance. For example, the virtual-reality fusion solution proposed in "Metaverse Data Fusion System" requires not only real-world spatial location information but also information such as obstacles and other vehicles. Using only data fusion methods, this approach results in significant communication traffic and resource requirements, making real-time performance difficult to guarantee and placing high demands on the computing platform. Furthermore, even if the real vehicle can react to virtual objects during testing, it still doesn't overcome the limitations of virtual scene fidelity. For example, if a rainy scene is set up in the virtual environment without also being set up in the real environment, the vehicle's behavior in the rainy environment won't be reflected. In other words, the vehicle's performance in the rain is primarily reflected in the virtual world, which again points to the limitations of simulation testing.

[0006] Summary of the Invention

[0007] In order to solve one of the technical problems existing in the prior art to at least a certain extent, the purpose of the present invention is to provide an autonomous driving test platform based on virtual-reality integration technology.

[0008] The technical solution adopted in the present invention is:

[0009] An autonomous driving test platform based on virtual-reality integration technology, including:

[0010] The vehicle-side real space includes the autonomous vehicle and the real environment. The autonomous vehicle includes a vehicle-side sensor, a vehicle-side computing unit, and a vehicle-side control unit. The vehicle-side sensor is used to collect real environment information. The vehicle-side computing unit is used to receive the real environment information and virtual-reality fusion information and sandbox environment information transmitted through the communication framework, perform calculations based on the received information, and issue control commands to the vehicle-side control unit. The vehicle-side control unit is used to control the vehicle according to the received control commands.

[0011] The edge-side sandbox space includes a sandbox vehicle and a sandbox environment constructed by scaling down the autonomous vehicle and the real environment in equal proportions. The sandbox vehicle includes sandbox vehicle sensors, an edge-side computing unit, and a sandbox vehicle control unit. The sandbox vehicle sensors are used to collect sandbox environment information. The edge-side computing unit is used to transmit the sandbox environment information to the cloud-based virtual space via a communication framework, and simultaneously forward the virtual-reality fusion information from the cloud-based virtual space to the vehicle-side computing unit. Before the test begins, the sandbox vehicle is placed in a position relative to the sandbox consistent with the autonomous vehicle's position in the real environment. During the test, the states of the sandbox vehicle and the autonomous vehicle remain synchronized.

[0012] The cloud virtual space includes a virtual vehicle and a virtual environment; the virtual vehicle includes a virtual sensor, a cloud computing unit and a virtual controller; the virtual sensor is used to collect virtual environment information; the cloud computing unit performs fusion calculations on the three types of environment information to obtain virtual-reality fusion information, and transmits the virtual-reality fusion information to the edge sandbox space through a communication framework.

[0013] Furthermore, the communication framework is a communication framework based on data distribution service technology;

[0014] The communication framework consists of four layers: communication fusion layer, perception layer, simulation layer, and sandbox layer;

[0015] The communication convergence layer, as the core of the communication framework, includes wireless access points, base stations, and gateways. At the physical level, wireless access points are connected to base stations via 5G or network cables, and gateways are connected to base stations via bearer networks or optical fibers. At the logical level, data is exchanged based on the DDS data bus.

[0016] The perception layer is used to provide data collection of real scenes and communication services with the communication fusion layer;

[0017] The simulation layer is used to provide virtual scene generation and communication services with the communication fusion layer;

[0018] The sandbox layer is used to provide data collection of sandbox scenes and communication services with the communication fusion layer.

[0019] Furthermore, the virtual environment includes static components, dynamic components and virtual weather; wherein, the static components are obtained from the real environment through digital twins; the dynamic components are composed of dynamic traffic elements, and the dynamic traffic elements include virtual pedestrians and virtual vehicles; the virtual weather is the weather status of the virtual environment.

[0020] Furthermore, the cloud virtual space is deployed on a cloud platform;

[0021] The virtual vehicle is obtained from the digital twin of the autonomous vehicle, but does not have the ability to drive autonomously in the virtual environment;

[0022] Virtual sensors include simulated lidar and simulated cameras.

[0023] Furthermore, the vehicle-side sensor includes a laser radar and a camera;

[0024] The laser radar is used to collect depth distance information and point cloud data;

[0025] The camera is used to collect visual information and identify the object type by combining it with a machine vision algorithm;

[0026] The vehicle-side computing unit is deployed with a robot operating system, which generates control commands in combination with the autonomous driving algorithm after receiving real environment information, virtual-reality fusion information, and sandbox environment information collected from vehicle-side sensors.

[0027] Furthermore, the basis of the virtual and real scene fusion problem of the autonomous driving test platform is to solve the matching and fusion problem of multiple sets of point cloud data. The edge sandbox space is selected as the benchmark, and the point clouds in the real space on the vehicle side and the virtual space on the cloud side are aligned with the point clouds in the edge sandbox space. All data of the multiple sets of point clouds are distributed, parallel, and pairwise aligned.

[0028] Furthermore, in the step of pairwise registration of all data of multiple point clouds, a multi-level fusion method of data, features, and pose is used to solve the virtual-real fusion problem:

[0029] For static objects, denoted as set A, the sampling rate is adjusted by the downsampling method S, and the point cloud data is fused using the iterative neighbor registration method;

[0030] For slow-moving objects, denoted as set B, the voxel features of the point cloud are first obtained through the feature extraction network F, and then the voxel features are fused;

[0031] For high-speed moving objects, denoted as set C, the 3D detector D is used to extract the object bounding boxes from the virtual and real point clouds respectively, and then the bounding boxes are fused.

[0032] Furthermore, the expression of using multi-level fusion of data, features, and posture to solve the virtual-real fusion problem is as follows:

[0033] Virtual-sandbox space cloud-edge registration:

[0034] Real-sandbox space end-to-edge registration:

[0035] Among them, R is the cloud edge rotation matrix, t is the cloud edge coordinate translation vector, qi ∈Q is the i-th point cloud in the cloud virtual space, p i ∈P is the i-th point cloud in the edge sandbox space; W is the edge rotation matrix, v is the edge coordinate translation vector, o i ∈O is the i-th point cloud in the real space of the vehicle, p i is the i-th point cloud in the edge sandbox space; n is the total number of point clouds, and ‖‖2 is the L2 norm.

[0036] Furthermore, in the case where the coordinate translation vector t and the rotation matrix R are coupled in the cost function at different levels, the alternating direction multiplication method is used to decompose the cost function; the cloud edge rotation matrix R = R * , cloud edge coordinate translation vector t=t * , end-edge rotation matrix W = W * , end-side coordinate translation vector v = v * .

[0037] Furthermore, the pseudo code of the algorithm for fusing the point cloud P in the edge sandbox space and the point cloud Q in the cloud virtual space is as follows:

[0038] 1) Use ROS2 Subscriber to obtain point cloud P, Q

[0039] 2) Initialize List F = []

[0040] 3) For i=1,…,n:

[0041] If

[0042] F=F.append(p i )

[0043] Else:

[0044] F=F.append(R * q i +t * )

[0045] 4) Use ROS2 Publisher to output the fused point cloud F;

[0046] The pseudo code of the algorithm for fusing the point cloud P in the edge sandbox space and the point cloud O in the vehicle-side virtual space is as follows:

[0047] 1) Use ROS2 Subscriber to obtain point cloud P,O

[0048] 2) Initialize List G = []

[0049] 3) For i=1,…,n:

[0050] If

[0051] G=G.append(p i )

[0052] Else:

[0053] G=G.append(W * o i +v * )

[0054] 4) Use ROS2 Publisher to output the fused point cloud G.

[0055] The beneficial effects of this invention are as follows: Based on a three-level linkage between cloud, edge, and end, the platform combines real-world testing, sandbox testing, and virtual simulation technology. The sandbox, virtual, and real-world environments fully interact to achieve data enhancement and achieve optimal testing results. The sandbox environment acts as a relay between the vehicle-side environment and the cloud-based virtual environment, reducing communication load and enhancing real-time performance. Furthermore, the sandbox environment complements the real-world environment, achieving data enhancement. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following introduction is made to the drawings of the embodiments of the present invention or the related technical solutions in the prior art. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative work.

[0057] FIG1 is an overall framework diagram of an autonomous driving test platform based on virtual-reality integration technology according to an embodiment of the present invention;

[0058] FIG2 is a schematic diagram of a multi-space high-frequency and high-precision registration algorithm according to an embodiment of the present invention;

[0059] FIG3 is a schematic diagram of a communication framework according to an embodiment of the present invention;

[0060] FIG4 is a diagram showing the operation effects of a virtual scene and a sandbox scene according to an embodiment of the present invention;

[0061] FIG5 is a diagram showing the overall operation effect of the platform in an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and are not to be construed as limiting the present invention. The step numbers in the following embodiments are provided for ease of explanation only and do not limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0063] In the description of the present invention, it should be understood that descriptions involving orientations, such as up, down, front, back, left, right, etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, they cannot be understood as limitations on the present invention.

[0064] In the description of the present invention, "several" means one or more, "many" means more than two, "greater than," "less than," and "exceed" are understood to exclude the number itself, while "above," "below," and "within" are understood to include the number itself. The use of "first" and "second" in the description is solely for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, implicitly specifying the number of the indicated technical features, or implicitly specifying the order of the indicated technical features.

[0065] Furthermore, in the description of this invention, unless otherwise specified, "plurality" refers to two or more. "And / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0066] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, and connecting should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0067] To address the shortcomings of existing solutions, the present invention proposes an autonomous driving test platform based on virtual-reality integration technology. This invention offers the following advantages: 1. It combines the scenario richness of virtual simulation testing with the confidence of real-world testing. It introduces a sandbox scenario as a complement to the virtual simulation and real-world scenarios. This allows scenarios that are difficult to construct in real-world scenarios to be realized in the sandbox scenario. Since the sandbox is a physical environment, its scenarios are closer to real-world scenarios, providing higher confidence and thus compensating for the shortcomings of simulation testing. 2. The three scenarios in this invention are multi-level interconnected. The sandbox environment acts as the edge, relaying communication between real-world and virtual-world information, reducing communication load and latency. Virtual-world information can be transmitted to the autonomous vehicle, enabling it to provide feedback based on the constructed virtual environment. Real-world information is also transmitted to the virtual scene, influencing the state of the virtual vehicle. This two-way communication enables data enhancement. 3. This invention employs a multi-level fusion method for data, features, and pose, by pairwise registration of different scene data. The overall concept is to decompose the scene and then fuse it layer by layer. This results in a small amount of transmitted data, minimal resource requirements, and high real-time performance.

[0068] As shown in FIG1 , this embodiment provides an autonomous driving test platform based on virtual-reality integration technology, including:

[0069] The vehicle-side real space includes the autonomous vehicle and the real environment. The autonomous vehicle includes a vehicle-side sensor, a vehicle-side computing unit, and a vehicle-side control unit. The vehicle-side sensor is used to collect real environment information. The vehicle-side computing unit is used to receive the real environment information and virtual-reality fusion information and sandbox environment information transmitted through the communication framework, perform calculations based on the received information, and issue control commands to the vehicle-side control unit. The vehicle-side control unit is used to control the vehicle according to the received control commands.

[0070] The edge-side sandbox space includes a sandbox vehicle and a sandbox environment constructed by scaling down the autonomous vehicle and the real environment in equal proportions. The sandbox vehicle includes sandbox vehicle sensors, an edge-side computing unit, and a sandbox vehicle control unit. The sandbox vehicle sensors are used to collect sandbox environment information. The edge-side computing unit is used to transmit the sandbox environment information to the cloud-based virtual space via a communication framework, and simultaneously forward the virtual-reality fusion information from the cloud-based virtual space to the vehicle-side computing unit. Before the test begins, the sandbox vehicle is placed in a position relative to the sandbox consistent with the autonomous vehicle's position in the real environment. During the test, the states of the sandbox vehicle and the autonomous vehicle remain synchronized.

[0071] The cloud virtual space includes a virtual vehicle and a virtual environment; the virtual vehicle includes a virtual sensor, a cloud computing unit and a virtual controller; the virtual sensor is used to collect virtual environment information; the cloud computing unit performs fusion calculations on the three types of environment information to obtain virtual-reality fusion information, and transmits the virtual-reality fusion information to the edge sandbox space through a communication framework.

[0072] This embodiment designs an autonomous driving test platform based on virtual-reality integration technology. It achieves data enhancement through three-level linkage between cloud, edge, and end, and uses data hierarchical fusion to achieve low-latency, low-load data interaction, thereby achieving efficient and high-confidence autonomous driving testing. This platform consists of three parts: the real scene on the vehicle side, the sandbox scene on the edge side, and the virtual scene on the cloud side. The autonomous driving vehicle is deployed in the real environment and communicates through the edge side relay to form a digital twin in the virtual scene. During the test, the autonomous driving vehicle simultaneously receives information from the on-board sensors, the sensors on the sandbox vehicle side, and the simulated sensors attached to the digital twin vehicle. In the decision-making process, information from the real, sandbox, and simulated environments will all be considered. Therefore, platform users can easily set up a virtual environment to build a test scene or sandbox environment, and then test the performance of the autonomous driving vehicle in the real environment, while evaluating the status of the sandbox vehicle in the sandbox scene, completing the test of the autonomous driving algorithm.

[0073] As an optional implementation, this embodiment uses pairwise registration for different scenarios, integrating data, features, and pose at multiple levels to create a digital twin of the autonomous vehicle in the virtual scene and synchronize the vehicle in the sandbox scene with the real vehicle. Simultaneously, the sandbox and virtual scenes receive information from the real scene to determine and update the vehicle's status.

[0074] Specifically, to address the heterogeneous perception of the three different scenarios, this embodiment also utilizes a communication framework based on DDS technology. This enables cross-domain, synchronous, and asynchronous data communication, providing high security and adaptable communication resource allocation strategies based on demand, resulting in high communication efficiency. To reduce communication requirements, asynchronous fusion of information with low real-time requirements, such as information about objects in virtual scenes, is employed. Dynamic adjustment of communication strategies enables low-latency communication under low-load conditions. Sandbox, virtual, and real-world environments fully interact to achieve data enhancement, resulting in optimal testing results.

[0075] The above platform is explained in detail below with reference to the accompanying drawings and specific embodiments.

[0076] This embodiment designs an autonomous driving test platform based on virtual-reality integration technology. It achieves data enhancement through a three-level cloud-edge-device linkage and employs hierarchical data fusion to achieve low-latency, low-load data exchange, enabling efficient and high-confidence autonomous driving testing. The detailed explanation of the technical solution for this embodiment is divided into four parts: system framework, hardware design, software design, and communication framework.

[0077] (1) System framework

[0078] Figure 1 shows the overall system framework diagram of this patent. The system mainly includes four components: the real scene on the vehicle side (i.e. the real space on the vehicle side in Figure 1), the sandbox scene on the edge side (i.e. the sandbox space on the edge side in Figure 1), the virtual scene on the cloud side (i.e. the virtual space on the cloud side in Figure 1) and the communication framework based on DDS technology.

[0079] The on-board real-world scene encompasses the autonomous vehicle and the physical environment (also known as the real environment). Autonomous vehicles primarily consist of on-board sensors, a computing unit, and a control unit. On-board sensors are responsible for acquiring and sensing the real environment and transmitting this information to the vehicle control system and positioning system. Common on-board sensors include lidar (LiDAR) and cameras. LiDAR emits a laser beam at a target, which then returns when it encounters an object. The return time is used to calculate the object's distance from the radar and obtain depth and distance information. Cameras collect visual information of the environment and, combined with machine vision algorithms, can perform functions such as object type recognition. The on-board computing unit, equipped with the Robot Operating System (ROS), receives real-world environment information from on-board sensors, as well as virtual-reality fusion information and sandbox environment information transmitted via the DDS communication framework. Based on this environmental information and combined with the autonomous driving algorithm, it issues control commands to the on-board control unit. For example, it fuses point cloud information from three spatial locations within the information and then uses an adaptive Monte Carlo algorithm to calculate the fused pose, positioning, and environmental information. This information, combined with the autonomous driving algorithm, issues control commands (steering, throttle, braking, etc.) to the on-board control unit. The vehicle-side control unit receives control commands from the vehicle-side computing unit and completes vehicle driving according to the commands.

[0080] The edge-side sandbox scene consists of a sandbox vehicle and a sandbox environment. The sandbox vehicle and environment are constructed by scaling down the autonomous vehicle and the real scene in equal proportions. Similarly, the sandbox vehicle is equipped with sensors such as lidar and cameras to enable perception of the sandbox environment. The edge-side computing unit transmits the sandbox information to the cloud-side computing unit via a communication framework based on DDS technology, and simultaneously forwards the virtual-reality fusion information from the cloud-side computing unit to the vehicle-side computing unit. In addition, the material of the sandbox that may affect the vehicle's status should be consistent with the real scene, such as the material of the sandbox road surface should be consistent with the road surface in the real scene. The status of the sandbox vehicle depends on the status of the autonomous vehicle. Before the test begins, the position of the sandbox vehicle relative to the sandbox should be consistent with the position of the autonomous vehicle in the real environment.

[0081] The cloud-based virtual scene is generated by a scaled digital twin of the real scene in the Carla simulator and deployed on a high-performance computing platform. Generally, all virtual scene components are scaled to the same extent as their physical counterparts. The virtual scene includes a virtual vehicle and a virtual environment. The virtual vehicle is a digital twin of the autonomous vehicle, but does not have the ability to autonomously navigate the virtual environment. The cloud-based computing unit fuses vehicle status information with sensor information and virtual vehicle sensor information to obtain the current vehicle position, which is used to update the virtual vehicle's position. Specifically, the cloud-based computing unit receives lidar point cloud information from different spaces and, using the multi-level fusion algorithm described in this embodiment, fuses the point clouds from the edge sandbox space with the cloud-based virtual space, and from the edge sandbox space with the vehicle's real space, obtaining the vehicle's position and posture information in each space. This information is then used to update the virtual vehicle's position. Virtual sensors are attached to the virtual vehicle and sense the virtual environment in which the virtual vehicle is currently located. This virtual environment information is transmitted to the vehicle control system via a communication framework. Similar to autonomous vehicles, virtual sensors include simulated lidar, simulated cameras, and other sensors.

[0082] The virtual environment consists of static components, dynamic components, and virtual weather. The static components are digital twins of the physical environment. Their positions in the virtual scene remain unchanged during system operation, but their appearance can be adjusted based on testing requirements. Common static components include roadside trees, buildings, traffic lights, and road condition signs. The dynamic components consist of dynamic traffic elements, such as virtual pedestrians and vehicles. These dynamic traffic elements can move along either a planned or random route and possess physical properties. When the digital twin vehicle collides with a dynamic traffic element, the cloud-based computing unit calculates the physical state of the digital twin vehicle and the dynamic traffic element after the collision and displays the results, such as post-collision rollover and tremors. These results help assess the severity of the accident, providing additional information for autonomous driving algorithm validation. Virtual weather refers to the weather conditions in the virtual environment, including sunny, cloudy, rainy, and windy scenarios. This allows for the reproduction of a variety of common and extreme scenarios, facilitating thorough testing of autonomous driving algorithms.

[0083] See Figure 4, which shows the virtual and sandbox scenes in action. The four screens, clockwise from the top left, are: virtual sensor information (simulated camera), real scene, real sensor information (camera), and virtual scene. See Figure 5, which shows the overall test platform in action. The three scenes, clockwise from the top left, are: real scene, virtual scene, and sandbox scene.

[0084] (2) Hardware Design

[0085] In real-world scenarios, the autonomous vehicle used in this embodiment is a vehicle capable of deploying autonomous driving algorithms, such as a Toyota Prius. The autonomous vehicle is equipped with sensors such as lidar and binocular cameras, and includes an onboard artificial intelligence computing platform that receives and processes sensor information and executes the autonomous driving algorithm.

[0086] In the sandbox environment, the vehicle used in this embodiment is a sandbox vehicle scaled down by a passenger car, and the sandbox environment is a real environment scaled down by the same proportion. The following solutions are adopted to solve a series of technical problems caused by the reduction of the sandbox vehicle and the sandbox environment:

[0087] When the sandbox vehicle is scaled up to passenger car size, it will be less than 5 meters long, less than 2 meters wide, less than 2 meters high, and have a wheelbase of less than 3 meters. The sandbox vehicle chassis utilizes an Ackerman structure with shock-absorbing suspension, similar to the structure of a typical passenger car. The sandbox vehicle's host computer is an artificial intelligence computing platform connected to sensors such as lidar and binocular cameras, and wirelessly connected to the platform framework via Wi-Fi. The AI ​​computing platform receives and processes sensor information, executes autonomous driving algorithms, and issues control commands. The sandbox vehicle's slave computer receives and executes control commands from the host computer, providing speed feedback and closed-loop speed control. Furthermore, the slave computer is equipped with a remote control receiver, allowing the vehicle to be controlled via a remote controller.

[0088] The lanes in the sandbox environment are scaled up to highway size: a typical urban lane is 3.5 meters wide, while an urban expressway is 3.75 meters wide. In addition to the lanes, the sandbox environment also features proportionally scaled static landmarks such as street trees, street signs, traffic lights, and buildings. Specifically, the sandbox environment's scale is equivalent to that of the autonomous vehicle.

[0089] (3) Software Design

[0090] Multi-space high-frequency and high-precision registration algorithm

[0091] The basis of the virtual and real scene fusion problem in this embodiment is to solve the matching and fusion problem of multiple sets of point cloud data, so as to determine the coordinate translation vector t and the rotation matrix R. For the autonomous driving platform, the system has high real-time requirements. If all the data of multiple sets of point clouds are to be registered pairwise, it is necessary to calculate When the number of point clouds that need to be registered increases, such as when more autonomous vehicles are connected, the time required increases by n. 2 Growth poses a great challenge to real-time performance. To ensure the real-time performance of the system, this embodiment selects the sandbox space as the benchmark, and the remaining point clouds are aligned with the sandbox space point cloud. At this time, the time required increases by (n-1), which is significantly reduced compared with the above alignment method. Specifically, consider the real scene point cloud O, the sandbox space point cloud P and the virtual scene point cloud Q: O = {o1,o2,…}, P = {p1,p2,…}, Q = {q1,q2,…},

[0092] Specifically, taking the fusion of the sandbox space point cloud P and the virtual scene point cloud Q as an example, the Euclidean transformation is calculated:

[0093] The above problem can be solved by using the iterative closet point registration method (ICP) published by General Motors PJBesl in "IEEE TPAMI". The first step is to find the corresponding points in the virtual space point cloud and the physical space point cloud; the second step is to construct the objective function by the least squares method for the corresponding points and perform iterative optimization. However, the ICP method is extremely complex and cannot be directly used for real-time virtual and real fusion problems. Therefore, referring to Figure 2, this embodiment proposes a multi-level fusion method of data, features, and posture. The overall idea is to decompose the scene and then fuse it in layers:

[0094] 1) For static objects (denoted as set A), the sampling rate is adjusted through downsampling methods such as integer downsampling and fractional downsampling (denoted as function S) based on the accuracy requirements and system performance, and the point cloud data is fused offline using the ICP method.

[0095] 2) For slow-moving objects (denoted as set B), voxel features of the point cloud are first obtained through a feature extraction network (denoted as function F), including convolution and encoders. Voxel fusion is then performed. The resolution and number of voxels can be adjusted according to the complexity required.

[0096] 3) For high-speed moving objects (denoted as set C), data or feature fusion methods cannot be used because their information update frequency is far too high. 3D detectors such as SECOND, PointPillar, and PointRCNN (denoted as function D) are used to extract object bounding boxes from both virtual and real point clouds, and then fusion is performed on these bounding boxes.

[0097] Based on the above analysis, in the multi-level virtual and real scene fusion problem of this embodiment, the edge sandbox space and the cloud virtual space are as follows:

[0098] In the above problem, the cloud edge coordinate translation vector t and the cloud edge rotation matrix R are coupled in cost functions at different levels. Therefore, the alternating direction method of multipliers (ADMM) can be used to further decompose the problem. The core of the ADMM mathematical derivation is to construct an augmented Lagrangian function, which has the following form:

[0099] Among them, R1, R2, R3, t1, t2, t3 are mirror variables; R12, R 13 ,t12,t13 are dual variables.

[0100] Similarly, for the edge space and the vehicle real space, there is the following augmented Lagrangian function, which has the following form:

[0101] Among them, the end-side coordinate translation vector v and the end-side rotation matrix W, W1, W2, W3, v1, v2, v3 are mirror variables; W12, W13, v12, v 13 is the dual variable.

[0102] By alternately optimizing the above cost function and converging to obtain the locally optimal rotation and translation matrix, the decomposition can be performed on the data at each level. The fusion of features and bounding boxes requires association, averaging, and pruning: ① Using clustering algorithms such as DBSCAN and k-means to associate virtual and real targets; ② Using heat maps to characterize the detection uncertainty of different targets, and performing weighted average pose calibration based on the heat maps; ③ Addressing target overlap using methods such as the Hungarian algorithm and non-maximum suppression.

[0103] Similarly, the above method is used to register the real scene point cloud O with the sandbox space point cloud P, and the point cloud registration of each space can be achieved. The cloud edge rotation matrix R = R * , cloud edge coordinate translation vector t=t * , end-edge rotation matrix W = W * , end-side coordinate translation vector v = v * Finally, parallel point cloud fusion processing is performed on the cloud-edge and edge-segment. The pseudo code of the algorithm for fusing the point cloud P in the edge sandbox space and the point cloud Q in the cloud virtual space is shown in Table 1 below:

[0104] Table 1

[0105] On the other hand, the pseudo code of the algorithm for fusing the point cloud P in the edge sandbox space and the point cloud O in the vehicle virtual space is shown in Table 2 below:

[0106] Table 2

[0107] (4) Communication framework

[0108] The integration of real-world, virtual, and sandbox scenarios relies on high-frequency, high-precision synchronization of information. Therefore, a wide-coverage, low-latency communication framework is required to enable information exchange between these scenarios. This paper, based on Data Distribution Service (DDS) technology, constructs a layered communication framework to achieve information exchange. The framework is shown in Figure 3. The communication framework consists of four layers: communication fusion layer, perception layer, simulation layer, and sandbox layer.

[0109] The communication convergence layer is the core of the communication framework and includes wireless access points, base stations, and gateways. Physically, wireless access points connect to base stations via 5G or Ethernet cables, while gateways connect to optical fibers via the bearer network. Logically, data is exchanged over the DDS data bus. DDS is a distributed real-time communication middleware that utilizes a publish-subscribe architecture. It prioritizes data and offers a rich set of QoS policies, ensuring real-time, efficient, and flexible data distribution. It is widely used in fields such as industrial control and robotics. Furthermore, DDS offers robust security, providing both domain-based and intra-domain security. It can exclude malicious access or attackers from the communication framework through permission management and identity authentication. Therefore, DDS technology is well-suited for inter-network communication. In practical implementation, the ROS2 system based on DDS technology can be used as a complete solution. First, members requiring communication register nodes in the ROS2 system and transmit data in the form of topics. Data providers encapsulate data into topics and broadcast them to the DDS system. Data receivers subscribe to these topics and receive updates from the DDS topic when their subscribed information is updated.

[0110] The perception layer collects data from real-world scenarios and provides communication services with the communication fusion layer. The information collected by sensors attached to the smart car, such as lidar and cameras, is transmitted to the sandbox server via wireless communication methods such as Wi-Fi and DSRC. The server also receives information from other spaces to ensure motion consistency between the real scene and the sandbox.

[0111] The simulation layer provides virtual scene generation and communication services with the communication fusion layer. The cloud server cluster is connected to the communication fusion layer gateway through the core network to enable the interaction of virtual scene information and sandbox scene information.

[0112] The sandbox layer provides data collection for sandbox scenarios and provides communication services with the communication fusion layer. The sandbox server is connected to the gateway via a dedicated network, enabling interaction between sandbox scenario information and real and virtual scenario information.

[0113] In summary, through the above communication framework, data from each space can fully interact through the communication fusion layer, achieving low-latency alignment of information from each scene and ensuring the real-time performance of the autonomous driving test platform.

[0114] In summary, the present invention has at least the following advantages and beneficial effects compared to existing technical solutions:

[0115] (1) The present invention combines the scene richness of virtual simulation testing with the confidence of real environment testing, and introduces a sandbox scene as a supplement between virtual simulation and real scenes. Some scenes that are difficult to construct in real scenes can be realized in the sandbox scene. The sandbox belongs to the physical environment, and its scenes are closer to the real scene, with higher confidence, which makes up for the shortcomings of simulation testing.

[0116] (2) The three scenarios in this invention are multi-level linked. The sandbox environment acts as the edge, relaying the communication between real and virtual environment information, reducing communication load and latency. Virtual environment information can be transmitted to the autonomous vehicle, allowing it to provide feedback based on the constructed virtual environment. Real scene information is also transmitted to the virtual scene, affecting the state of the virtual vehicle. Bidirectional communication enables data enhancement.

[0117] (3) The present invention adopts a pairwise spatial registration and multi-level fusion method of data, features, and posture. The overall idea is to decompose the scene and then fuse it layer by layer, so the amount of data transmitted is small, the required resources are small, and the real-time performance is high. For static objects, the ICP method is used to fuse point cloud data offline. For low-speed moving objects, the voxel features (Voxel) of the point cloud are first obtained through feature extraction networks such as convolution and encoder, and then the voxels are fused. This process can adjust the resolution and number of voxels according to the complexity requirements. For high-speed moving objects, since the frequency of their information update is much too high, data or feature fusion methods cannot be used. 3D detectors such as SECOND, PointPillar, and PointRCNN (denoted as function D) are used to extract the object bounding boxes from the virtual and real point clouds respectively, and then the bounding boxes are fused.

[0118] In the above description of this specification, reference to the terms "one embodiment / example," "another embodiment / example," or "certain embodiments / examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0119] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

[0120] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. An autonomous driving test platform based on virtual-real integration technology, characterized in that: include: The vehicle-side real space includes an autonomous driving vehicle and a real environment. The autonomous driving vehicle includes a vehicle-side sensor, a vehicle-side computing unit, and a vehicle-side control unit. The vehicle-side sensor is used to collect real environment information. The vehicle-side computing unit is used to receive real environment information and virtual-reality fusion information and sandbox environment information transmitted through the communication framework, perform calculations based on the received information, and issue control commands to the vehicle-side control unit; the vehicle-side control unit is used to control the vehicle driving based on the received control commands; The edge sandbox space includes a sandbox vehicle and a sandbox environment that are constructed by scaling down the autonomous driving vehicle and the real environment in equal proportions. The sandbox vehicle includes a sandbox vehicle sensor, an edge computing unit, and a sandbox vehicle control unit. The sandbox vehicle sensor is used to collect sandbox environment information. The edge computing unit is used to transmit the sandbox environment information to the cloud virtual space through the communication framework, and forward the virtual-reality fusion information from the cloud virtual space to the vehicle computing unit; wherein, before the test starts, the position of the sandbox vehicle relative to the sandbox is consistent with the real environment position of the autonomous driving vehicle; During the test, the status of the sandbox vehicle and the autonomous driving vehicle remain synchronized; The cloud virtual space includes a virtual vehicle and a virtual environment; the virtual vehicle includes a virtual sensor, a cloud computing unit and a virtual controller; the virtual sensor is used to collect virtual environment information; the cloud computing unit performs fusion calculations on the three types of environment information to obtain virtual-reality fusion information, and transmits the virtual-reality fusion information to the edge sandbox space through a communication framework.

2. The autonomous driving test platform based on virtual-real integration technology according to claim 1, characterized in that: The communication framework is a communication framework based on data distribution service technology; The communication framework includes a communication fusion layer, a perception layer, a simulation layer, and a sandbox layer; The communication fusion layer, as the core of the communication framework, includes wireless access points, base stations and gateways; At the physical level, wireless access points are connected to base stations, and gateways are connected to base stations; at the logical level, data is exchanged based on the DDS data bus; The perception layer is used to provide data collection of real scenes and communication services with the communication fusion layer; The simulation layer is used to provide virtual scene generation and communication services with the communication fusion layer; The sandbox layer is used to provide data collection of sandbox scenes and communication services with the communication fusion layer.

3. The autonomous driving test platform based on virtual-real integration technology according to claim 1, characterized in that: The virtual environment includes static components, dynamic components and virtual weather; wherein, the static components are obtained from the real environment through digital twins; the dynamic components are composed of dynamic traffic elements, and the dynamic traffic elements include virtual pedestrians and virtual vehicles; the virtual weather is the weather state of the virtual environment.

4. The autonomous driving test platform based on virtual-real integration technology according to claim 1, characterized in that: The cloud virtual space is deployed on a cloud platform; The virtual vehicle is obtained from the digital twin of the autonomous vehicle, but it does not have the ability to drive autonomously in the virtual environment; Virtual sensors include simulated lidar and simulated cameras.

5. The autonomous driving test platform based on virtual-real integration technology according to claim 1, characterized in that: The vehicle-side sensor includes a laser radar and a camera; The laser radar is used to collect depth distance information and point cloud data; The camera is used to collect visual information and identify the object type by combining with machine vision algorithms; The vehicle-side computing unit is deployed with a robot operating system, which generates control commands in combination with the autonomous driving algorithm after receiving real environment information, virtual-reality fusion information, and sandbox environment information collected from vehicle-side sensors.

6. The autonomous driving test platform based on virtual-real integration technology according to claim 1, characterized in that: The basis of the virtual-real scene fusion problem of the autonomous driving test platform is to solve the matching and fusion problem of multiple sets of point cloud data. The edge sandbox space is selected as the benchmark, and the point clouds in the real space of the vehicle and the virtual space of the cloud are aligned with the point clouds in the edge sandbox space. All data of multiple sets of point clouds are distributed, parallel, and pairwise aligned.

7. The autonomous driving test platform based on virtual-real integration technology according to claim 6, characterized in that: In the step of pairwise registration of all data of multiple groups of point clouds, a multi-level fusion method of data, features, and posture is used to solve the virtual-real fusion problem: For static objects, denoted as set A, the sampling rate is adjusted by downsampling method S, and the point cloud data is fused under the iterative neighbor registration method; For slow-moving objects, denoted as set B, the voxel features of the point cloud are first obtained through the feature extraction network F, and then the voxel features are fused; For high-speed moving objects, denoted as set C, the 3D detector D is used to extract the object bounding boxes from the virtual and real point clouds respectively, and then the bounding boxes are fused.

8. The autonomous driving test platform based on virtual-real integration technology according to claim 7, characterized in that: The expression of using multi-level fusion of data, features, and posture to solve the virtual-real fusion problem is as follows: Virtual-sandbox space cloud edge registration: Real-sandbox space end-to-edge registration: Among them, R is the cloud edge rotation matrix, t is the cloud edge coordinate translation vector, q i ∈Q is the i-th point cloud in the cloud virtual space, p i ∈P is the i-th point cloud in the edge sandbox space; W is the edge rotation matrix, v is the edge coordinate translation vector, o i ∈O is the i-th point cloud in the real space of the vehicle, p i is the i-th point cloud in the edge sandbox space; n is the total number of point clouds, and ‖‖2 is the L2 norm.

9. The autonomous driving test platform based on virtual-real integration technology according to claim 8, characterized in that: In the case where the coordinate translation vector t and the rotation matrix R are coupled in the cost functions at different levels, the alternating direction multiplication method is used to decompose them; the cloud edge rotation matrix R = R * , cloud edge coordinate translation vector t = t * , the end-edge rotation matrix W = W * , end-side coordinate translation vector v = v * .

10. The autonomous driving test platform based on virtual-real integration technology according to claim 6, characterized in that: The pseudo code of the algorithm for fusing the point cloud P in the edge sandbox space and the point cloud Q in the cloud virtual space is as follows: The pseudo code of the algorithm for fusing the point cloud P in the edge sandbox space and the point cloud O in the vehicle virtual space is as follows:

Images