Apoll car automatic driving test platform, construction method and system and test method

Abstract

The invention discloses an Apoll car automatic driving test platform, a construction method and system and a test method, and the construction method comprises the steps: constructing an edge node, which comprises the steps: deploying a simulator server at the edge node, and calculating an inference algorithm at the edge end; an Apoll trolley industrial personal computer is constructed and comprises an Apoll automatic driving platform carried by an Apoll trolley, and a coordinate system conversion module and a bridging module are further deployed; and constructing a user side, including constructing a scene generator and a scene display interface. According to the method, rich road scene materials provided by Carla are utilized, and a complex road scene is constructed at the edge end in cooperation with a CyberRT platform of the Apoll car; a to-be-measured algorithm is deployed at an edge end, and the edge end is used for providing a data interface for testing; and the edge end outputs an algorithm analysis result and accesses the Apoll real vehicle to carry out real vehicle testing, so that the method has a wide application prospect.

Description

Technical Field

[0001] The present invention relates to an Apollo car road test technology, and in particular to an Apollo car automatic driving test platform, a construction method and system, and a test method. Background Art

[0002] Autonomous driving technology has emerged in the wave of intelligent cars. With its outstanding advantages, it is considered to be the key to the future transportation field. It can not only greatly improve traffic safety and reduce traffic congestion, but also expand diversified travel options and provide more possibilities for future travel. Baidu Apollo car is an autonomous driving car developed by Baidu. It uses advanced perception, control and navigation technologies, and provides rich data channels that can be used for autonomous driving system testing and verification on actual roads. In order to ensure the safe operation of autonomous driving cars in various complex road conditions, a lot of testing and verification is required. However, there are many difficulties and risks in testing on actual roads, including safety risks, time costs and resource limitations. Therefore, virtual simulation environment combined with real car testing has become a key component of autonomous driving system testing and verification.

[0003] There are many simulation simulators used for autonomous driving, such as CarSim, Carla, SUMO, AirSim, TORCS, etc., which have enriched the application of autonomous driving simulation scenarios to a certain extent. However, they also have some defects. For example, Carsim's calculation is too complicated and the simulation speed is slow. It is mainly used for vehicle engineering problems such as vehicle mechanical structure, conduction and power system, energy consumption, etc., and is rarely used for the research of autonomous driving algorithms; Carla has a slow simulation speed and a complex dependence on the system; SUMO's driving model is too simple, without considering the physical model and underlying control module of the vehicle, but using a slider engine, that is, it can only customize the longitudinal control, and only supports lateral movement between different lanes, and it is completed instantly. However, Carla's open source, diverse environmental simulation, high customizability, sensor simulation, real-time data collection, and developer-friendly features have been widely used in academia and the business community. The simulator is more complicated to build complex scenes, and these simulation simulators, including Carla, have difficulties in docking with Baidu's autonomous driving platform. In the early stages of autonomous driving algorithm development, most algorithms were developed based on Python. Before optimization, the performance requirements for hardware were relatively high, and the vehicle-side hardware alone may not meet the requirements. In addition, it is difficult for a virtual vehicle to completely reproduce the properties of a real vehicle, so it is difficult to completely reproduce the braking response of a real vehicle by relying solely on virtual vehicle testing.

[0004] Therefore, the prior art has the following deficiencies:

[0005] 1. Construction of complex virtual road scenes for Apollo cars: The current powerful simulation software requires a considerable learning cost for the construction of complex road scenes. The relatively simple road scene construction cannot meet the virtual-reality test of real car autonomous driving in some aspects. In addition, some simulation software is not open source and has a high cost of use. In addition, there is no official solution for the current simulation software to connect with the Apollo autonomous driving platform. The unofficial Carla solution for connecting with the Apollo autonomous driving platform requires that it is based on the CyberRT image and is in the same network segment as the Apollo autonomous driving platform. The deployment is relatively complex and has strict network requirements.

[0006] 2. Insufficient terminal computing power makes autonomous driving algorithm testing cumbersome: In the initial stage of algorithm testing, which has not yet been optimized or in some specific test scenarios, the required computing power is high, and the hardware performance of terminals such as the Apollo car cannot meet it.

[0007] 3. Virtual vehicles cannot completely replicate all the properties of real vehicles: Virtual tests of autonomous driving that are independent of real vehicles generally require setting the relevant properties of the vehicle, but often the properties of virtual vehicles cannot completely match the real vehicles, and any tiny variables in autonomous driving may lead to serious consequences. Summary of the invention

[0008] Purpose of the invention: The purpose of the present invention is to provide a convenient method for constructing a virtual-reality autonomous driving test platform for Baidu Apollo car based on edge computing, so as to solve the problems of relatively complex deployment of virtual-reality test for Apollo car, high cost of constructing virtual road scenes and insufficient performance of on-board hardware; an Apollo car autonomous driving test platform, construction system and testing method are also provided.

[0009] Technical solution: A method for constructing an Apollo car autonomous driving test platform of the present invention comprises the following steps:

[0010] Build edge nodes, including deploying simulation simulator servers and edge computing inference algorithms on edge nodes. The simulation simulator server includes multiple actors, perception devices, and Apollo virtual cars. The perception devices are set on the Apollo virtual car, and the data streams of the perception devices are obtained for edge computing inference algorithms to calculate inferences, and the results of the inference calculations are pushed to the Apollo autonomous driving platform.

[0011] The Apollo car industrial computer is built, including the Apollo autonomous driving platform carried by the Apollo car itself. The coordinate system conversion module and the bridge module are also deployed. The coordinate system conversion module converts the GNSS information obtained by the Apollo car in the real world into coordinates in the simulator world; the bridge module is used to realize the communication function between the simulator server on the edge node and the edge computing and reasoning algorithm and the Apollo autonomous driving platform, which is specifically divided into two parts: writing messages to the Apollo channel and reading messages from the Apollo channel;

[0012] Constructing the user end includes constructing a scene generator and a scene display interface. The scene generator generates a specified virtual scene from the pre-written scene code, and the scene display interface intuitively displays the interface in the virtual scene.

[0013] Furthermore, before deploying the simulator server, it is necessary to determine the map for the Apollo autonomous driving platform; the map for the Apollo autonomous driving platform is converted from the map that comes with the simulator server or the map edited by yourself, and the missing files are generated in the Apollo container, specifically:

[0014] Create a folder with the map name in the apollo / modules / map / data directory;

[0015] Use generate_routing_topo_graph.sh provided in the Apollo container to generate routing_map related files;

[0016] Use the sim_map_generator provided in the Apollo container to generate sim_map related files.

[0017] Furthermore, the method for obtaining the data stream of the sensing device is:

[0018] Connect to the simulator server through API;

[0019] Get the specified sensing device from the model library of the simulation simulator server;

[0020] Set the relevant parameters of the perception device and its position and direction information relative to the Apollo virtual car, and place the perception device on the Apollo virtual car;

[0021] The data stream of the perception device is obtained through the data interface of the perception device provided by the simulation simulator server.

[0022] Furthermore, data stream acquisition modules and websocket modules are deployed on edge nodes. The data stream acquisition module provides the data stream of the sensing device to the edge computing and reasoning algorithm for computation and reasoning, and the websocket module transmits the computation and reasoning results to the Apollo car industrial computer for the car to perform related actions.

[0023] The edge computing and reasoning algorithms obtain input and output results in a unified way. Depending on the algorithm functions, they are started by specifying startup parameters when creating a container to determine the data flow that the algorithm needs to obtain and the calculation and reasoning results are transmitted to the channel specified by Apollo through the websocket module and the bridge module.

[0024] Furthermore, in the coordinate system conversion module, there is scaling according to the environment in which the Apollo car is located, specifically:

[0025] When converting the real world coordinate system to the virtual world coordinate system, set the value in the z direction to be constant at 0;

[0026] Choose a road that contains an intersection for testing;

[0027] Plan the scope of the Apollo car in the real world, obtain the coordinates of the Apollo car's starting position, and obtain the coordinates of the preset intersection in the real world; correspondingly obtain the coordinates of the Apollo virtual car's starting position in the virtual world, and the coordinates of the intersection;

[0028] Taking the intersection as the origin, the coordinates of the real-world intersection are marked as (x r ,y r ), the virtual world crossroads coordinates are marked as (x v ,y v ), calculate the x-axis or y-axis difference between the starting position of the real world coordinate system and the crossroad coordinate, recorded as Δw1, and the x-axis or y-axis difference between the starting position and the crossroad coordinate in the virtual world, recorded as Δw2, and calculate the ratio of the real world coordinate difference to the virtual world coordinate difference;

[0029] The real world coordinates (x, y, z) are converted to virtual world coordinates: 0).

[0030] Furthermore, the Apollo channel writes messages and the Apollo channel reads messages, specifically:

[0031] The Apollo channel write message writes the edge node's calculation and reasoning results and the virtual world coordinates converted from the real world coordinates into the Apollo channel, including:

[0032] Obtain the calculation and reasoning results through the websocket service provided by the edge node, and write the calculation and reasoning results to the / apollo / perception / obstacles and / apollo / perception / traffic_light channels;

[0033] Call the coordinate system conversion module on the Apollo car's industrial computer to convert the coordinates of the real world coordinate system obtained by the Apollo car into the coordinates of the virtual world coordinate system and write them into the / apollo / localization / pose channel of Apollo;

[0034] The Apollo channel reads messages including: obtaining the current position and posture information of the Apollo car, and pushing the changes of the position and posture information of the Apollo car to the virtual world through the websocket service, and synchronizing the position and posture information of the Apollo virtual car in the virtual world.

[0035] Furthermore, when the simulation simulator server adopts the simulation simulator Carla server, a scene generator and a scene display interface are constructed, specifically including:

[0036] The virtual scene generator constructs the virtual scene code through Scenario Runner, initializes the positions of each actor, writes the actor's behavior and trigger conditions; through the websocket service, continuously obtains the position and posture information of the Apollo car in the virtual scene, and synchronously changes the position and posture of the Apollo virtual car through the set_transform method provided by the API of the simulator server;

[0037] The scene display interface connects to the simulation simulator server through the simulation simulator server API and obtains the scene video stream.

[0038] An Apollo car automatic driving test platform of the present invention comprises: an edge node, an Apollo car industrial computer and a user end, the edge node comprises a simulation simulator server end and an edge end computing and reasoning module, the simulation simulator server end comprises multiple actors, sensing devices and an Apollo virtual car, and the sensing device is arranged on the Apollo virtual car; the Apollo car industrial computer comprises an Apollo automatic driving platform carried by the Apollo car itself, and also comprises a coordinate system conversion module and a bridge module, and the user end comprises a scene generator module and a scene display interface; the scene generator module generates a virtual scene with multiple actors, sensing devices and the Apollo virtual car according to requirements, and displays it on the scene display interface, the edge end computing and reasoning module performs computing and reasoning according to the data stream of the sensing device, the bridge module is used to realize the communication function between the simulation simulator server end on the edge node and the edge end computing and reasoning algorithm and the Apollo automatic driving platform, and the coordinate system conversion module converts the GNSS information acquired by the Apollo car in the real world into coordinates in the simulation simulator world.

[0039] An Apollo car automatic driving test platform construction system of the present invention includes:

[0040] The edge node construction unit is used to deploy the simulation simulator server and edge computing inference algorithm on the edge node. The simulation simulator server includes multiple actors, perception devices and Apollo virtual car. The perception device is set on the Apollo virtual car, and the data stream of the perception device is obtained to the edge computing inference algorithm for inference, and the inference calculation result is pushed to the Apollo autonomous driving platform;

[0041] The Apollo car industrial computer construction unit is used to deploy the Apollo autonomous driving platform carried by the Apollo car itself. The coordinate system conversion module and the bridge module are also deployed. The coordinate system conversion module converts the GNSS information obtained by the Apollo car in the real world into coordinates in the simulator world; the bridge module realizes the communication function between the simulator server on the edge node and the edge computing and reasoning algorithm and the Apollo autonomous driving platform, which is specifically composed of two parts: Apollo channel writing messages and Apollo channel reading messages;

[0042] The user-side construction unit is used to construct a scene generator and a scene display interface. The scene generator generates a specified virtual scene from a pre-written scene code, and the scene display interface intuitively displays the interface in the virtual scene.

[0043] The present invention provides an Apollo car virtual and real combined automatic driving test method, comprising:

[0044] The edge node runs the simulation simulator server and edge computing inference algorithm programs, including:

[0045] (1) Download the simulator server image;

[0046] (2) Run the simulation simulator server image, map the port, and determine the startup parameters;

[0047] (3) Run the edge computing and reasoning algorithm program and continuously monitor the sensors of the Apollo virtual car;

[0048] Operate the Apollo car industrial computer, including:

[0049] (1) Launch Boostrap of the Apollo autonomous driving platform;

[0050] (2) Log in to Apollo's Dreamview interface, select the appropriate vehicle model, select the imported map, and open the GPS, Canbus, and Planning modules;

[0051] (3) Run the coordinate system conversion process in the container of the Apollo autonomous driving platform to ensure that the converted car coordinates are in the correct position on the map;

[0052] (4) Run the bridge process in the container of the Apollo autonomous driving platform. When running the process, determine the IP address of the simulator server and the server port information of the developed websocket or grpc;

[0053] Select the host connected to the Apollo car industrial computer and the simulation simulator server network to run the user-side program, including:

[0054] (1) Generate a virtual scene Run the program to generate a virtual scene, ensure that the IP address and port of the simulation simulator server and the IP address and port of the car position information server are correct, initialize the actor and the position of the Apollo virtual car, and generate a virtual scene;

[0055] (2) Set the destination of the Apollo car in the Dreamview interface;

[0056] (3) Run the scene display program, ensure the IP address and port of the simulation simulator server, generate the audience perspective camera, and view the virtual scene image through the audience perspective camera image stream.

[0057] Beneficial effects: Compared with the prior art, the advantages of the present invention are: users can build complex virtual road scenes more conveniently, reduce learning costs, and simplify the construction of complex virtual road scenes; at the same time, the official docking and integration with the Apollo autonomous driving platform can be more convenient, avoiding the network and deployment complexity of unofficial solutions; by migrating part of the computing reasoning to the edge nodes, the problem of insufficient terminal computing power is compensated, which helps to conduct algorithm testing more flexibly in the initial testing phase or specific scenarios without optimizing the algorithm efficiency prematurely. The edge nodes provide a convenient interface, allowing the test algorithm to access data and output results, and realize plug-in testing; in addition, in the virtual test, the position information of the real vehicle is projected into the virtual world through certain rules, which solves the problem that the virtual vehicle cannot completely replicate the properties of the real vehicle; this combination of virtual and real methods helps to simulate actual driving scenarios more accurately, taking into account the consequences that may be caused by small variables.

[0058] Compared with the existing technology, the present invention can conveniently construct virtual scenes, integrate edge computing, and combine the virtual and the real, thereby improving the convenience, flexibility, and accuracy of the autonomous driving test platform and reducing the cost of the virtual and the real test, thereby better supporting the autonomous driving development and testing work based on the Baidu Apollo car. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 is a flow chart of the method of the present invention;

[0060] Figure 2 This is the architecture diagram of Baidu Apollo's virtual-real autonomous driving test platform.

[0061] Figure 3 This is the communication flow chart of the Baidu Apollo car virtual-real autonomous driving test platform.

[0062] Figure 4 This is a diagram of the Dreamview interface;

[0063] Figure 5 Schematic diagram of a virtual scene and a result presented by Dreamview, where (a) is a schematic diagram of a virtual scene and (b) is a schematic diagram of a result presented by Dreamview. DETAILED DESCRIPTION

[0064] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0065] like Figure 1 As shown, a method for constructing an Apollo car automatic driving test platform of the present invention comprises the following steps:

[0066] Build edge nodes, including deploying simulation simulator servers and edge computing inference algorithms on edge nodes. The simulation simulator server includes multiple actors, perception devices, and Apollo virtual cars. The perception devices are set on the Apollo virtual car, and the data streams of the perception devices are obtained for edge computing inference algorithms to calculate inferences, and the results of the inference calculations are pushed to the Apollo autonomous driving platform.

[0067] Before deploying the simulator server, you need to determine the map for the Apollo autonomous driving platform. The map for the Apollo autonomous driving platform is converted from the map that comes with the simulator server or the map you edited yourself. The missing files are generated in the Apollo container, specifically:

[0068] Create a folder with the map name in the apollo / modules / map / data directory;

[0069] Use generate_routing_topo_graph.sh provided in the Apollo container to generate routing_map related files;

[0070] Use the sim_map_generator provided in the Apollo container to generate sim_map related files.

[0071] The method to obtain the data stream of the sensing device is:

[0072] Connect to the simulator server through API;

[0073] Get the specified sensing device from the model library of the simulation simulator server;

[0074] Set the relevant parameters of the perception device and its position and direction information relative to the Apollo virtual car, and place the perception device on the Apollo virtual car;

[0075] The data stream of the perception device is obtained through the data interface of the perception device provided by the simulation simulator server.

[0076] On the edge node, a data stream acquisition module and a websocket module are also deployed. The data stream acquisition module provides the data stream of the perception device to the edge computing and reasoning algorithm for computation and reasoning, and the websocket module transmits the computation and reasoning results to the Apollo car industrial computer for the car to perform related actions.

[0077] The edge computing and reasoning algorithms obtain input and output results in a unified way. Depending on the algorithm functions, they are started by specifying startup parameters when creating a container to determine the data flow that the algorithm needs to obtain and the calculation and reasoning results are transmitted to the channel specified by Apollo through the websocket module and the bridge module.

[0078] The Apollo car industrial computer is built, including the Apollo autonomous driving platform carried by the Apollo car itself. The coordinate system conversion module and the bridge module are also deployed. The coordinate system conversion module converts the GNSS information obtained by the Apollo car in the real world into coordinates in the simulator world; the bridge module is used to realize the communication function between the simulator server on the edge node and the edge computing and reasoning algorithm and the Apollo autonomous driving platform, which is specifically divided into two parts: writing messages to the Apollo channel and reading messages from the Apollo channel;

[0079] In the coordinate system conversion module, there is scaling according to the environment in which the Apollo car is located, specifically:

[0080] When converting the real world coordinate system to the virtual world coordinate system, set the value in the z direction to be constant at 0;

[0081] Choose a road that contains an intersection for testing;

[0082] Plan the scope of the Apollo car in the real world, obtain the coordinates of the Apollo car's starting position, and obtain the coordinates of the preset intersection in the real world; correspondingly obtain the coordinates of the Apollo virtual car's starting position in the virtual world, and the coordinates of the intersection;

[0083] Taking the intersection as the origin, the coordinates of the real-world intersection are marked as (x r ,y r ), the virtual world crossroads coordinates are marked as (x v ,y v ), calculate the x-axis or y-axis difference between the starting position of the real world coordinate system and the crossroad coordinate, recorded as Δw1, and the x-axis or y-axis difference between the starting position and the crossroad coordinate in the virtual world, recorded as Δw2, and calculate the ratio of the real world coordinate difference to the virtual world coordinate difference;

[0084] The real world coordinates (x, y, z) are converted to virtual world coordinates: 0).

[0085] Apollo channel write messages and Apollo channel read messages, specifically:

[0086] The Apollo channel write message writes the edge node's calculation and reasoning results and the virtual world coordinates converted from the real world coordinates into the Apollo channel, including:

[0087] Obtain the calculation and reasoning results through the websocket service provided by the edge node, and write the calculation and reasoning results to the / apollo / perception / obstacles and / apollo / perception / traffic_light channels;

[0088] Call the coordinate system conversion module on the Apollo car's industrial computer to convert the coordinates of the real world coordinate system obtained by the Apollo car into the coordinates of the virtual world coordinate system and write them into the / apollo / localization / pose channel of Apollo;

[0089] The Apollo channel reads messages including: obtaining the current position and posture information of the Apollo car, and pushing the changes of the position and posture information of the Apollo car to the virtual world through the websocket service, and synchronizing the position and posture information of the Apollo virtual car in the virtual world.

[0090] Constructing the user end includes constructing a scene generator and a scene display interface. The scene generator generates a specified virtual scene from the pre-written scene code, and the scene display interface intuitively displays the interface in the virtual scene.

[0091] An Apollo car autonomous driving test platform constructed by the above method includes: an edge node, an Apollo car industrial computer and a user end, the edge node includes a simulation simulator server and an edge computing and reasoning module, the simulation simulator server includes multiple actors, perception devices and an Apollo virtual car, and the perception device is set on the Apollo virtual car; the Apollo car industrial computer includes the Apollo autonomous driving platform carried by the Apollo car itself, and also includes a coordinate system conversion module and a bridge module, and the user end includes a scene generator module and a scene display interface; the scene generator module generates a virtual scene with multiple actors, perception devices and Apollo virtual cars according to needs, and displays it on the scene display interface, the edge computing and reasoning module performs computing and reasoning according to the data stream of the perception device, the bridge module is used to realize the communication function between the simulation simulator server on the edge node and the edge computing and reasoning algorithm and the Apollo autonomous driving platform, and the coordinate system conversion module converts the GNSS information obtained by the Apollo car in the real world into coordinates in the simulation simulator world.

[0092] The present invention provides an Apollo car virtual and real combined automatic driving test method, which is characterized by comprising:

[0093] The edge node runs the simulation simulator server and edge computing inference algorithm programs, including:

[0094] (1) Download the simulator server image;

[0095] (2) Run the simulation simulator server image, map the port, and determine the startup parameters;

[0096] (3) Run the edge computing and reasoning algorithm program and continuously monitor the sensors of the Apollo virtual car;

[0097] Operate the Apollo car industrial computer, including:

[0098] (1) Launch Boostrap of the Apollo autonomous driving platform;

[0099] (2) Log in to Apollo's Dreamview interface, select the appropriate vehicle model, select the imported map, and open the GPS, Canbus, and Planning modules;

[0100] (3) Run the coordinate system conversion process in the container of the Apollo autonomous driving platform to ensure that the converted car coordinates are in the correct position on the map;

[0101] (4) Run the bridge process in the container of the Apollo autonomous driving platform. When running the process, determine the IP address of the simulator server and the server port information of the developed websocket or grpc;

[0102] Select the host connected to the Apollo car industrial computer and the simulation simulator server network to run the user-side program, including:

[0103] (1) Generate a virtual scene Run the program to generate a virtual scene, ensure that the IP address and port of the simulation simulator server and the IP address and port of the car position information server are correct, initialize the actor and the position of the Apollo virtual car, and generate a virtual scene;

[0104] (2) Set the destination of the Apollo car in the Dreamview interface;

[0105] (3) Run the scene display program, ensure the IP address and port of the simulation simulator server, generate the audience perspective camera, and view the virtual scene image through the audience perspective camera image stream.

[0106] Example:

[0107] The present invention is applied to the virtual-real road test of the Apollo car, and proposes a method for building a virtual-real autonomous driving test platform based on edge computing and using Carla (Car Learning to Act) to build a virtual road scene. This method uses the rich road scene materials provided by Carla, and cooperates with the CyberRT platform of the Apollo car itself to build a complex road scene at the edge; the algorithm to be tested is deployed at the edge, and the data interface provided by the edge is used for testing; the edge outputs the algorithm analysis results and connects them to the Apollo real car for real car testing; this innovative technology has broad application prospects and provides more convenient, lower-cost, and safer technical support for autonomous driving testing.

[0108] The present invention uses Carla and its extended Scenario Runner to construct complex road scenarios, and solves the docking problem between Carla and the Apollo autonomous driving platform through the C / S architecture; and adopts the edge computing solution to enable the initial unoptimized algorithm code or part of the inference calculation to be performed on the edge node, thereby solving the problem of insufficient vehicle-side configuration; and utilizes the rich data channels provided by the Apollo autonomous driving platform and the data interfaces provided by the edge nodes, in conjunction with actual measurements of real Apollo cars, to solve the problem that it is difficult for the Apollo virtual car to reproduce the properties of the real Apollo car.

[0109] like Figure 1 As shown, a method for constructing an Apollo car automatic driving test platform of the present invention comprises the following steps:

[0110] S1. In this embodiment, the simulation simulator server adopts the simulation simulator Carla server. The edge node mainly deploys the simulation simulator Carla server and the edge computing and reasoning algorithm of autonomous driving, and sets the sensing devices such as radar, camera and audience camera on the Apollo virtual car. The edge node provides the data interface of Carla, provides the data of camera and radar to the algorithm computing and reasoning, and pushes the result of the inference calculation to the Apollo autonomous driving platform through the interface. The specific Carla and edge computing and reasoning are as follows:

[0111] (1) To achieve better image rendering effects, edge nodes need to be equipped with Nvidia graphics cards and run without an interface based on Kubernetes containers. Before deploying Carla, you need to determine the map to be used. The map can be the one that comes with Carla or the one you edit yourself. The details are as follows:

[0112] (11) If you use the map that comes with Carla, you can get it directly from GitHub. Just search for Carla_apollo_maps on GitHub. A complete Apollo map includes the files base_map.bin, base_map.txt, base_map.xml, routing_map.bin, routing_map.bin, sim_map.bin and sim_map.txt.

[0113] (12) The missing files in the acquired map can be generated based on Apollo by following the steps below in the Apollo container:

[0114] (121) Create a folder with the map name in the apollo / modules / map / data directory;

[0115] (122) Use generate_routing_topo_graph.sh provided in the Apollo container to generate routing_map related files;

[0116] (123) Use the sim_map_generator provided in the Apollo container to generate sim_map related files.

[0117] (13) If you need a custom map, you can generate it by following the steps below:

[0118] (131) Use roadrunner to draw custom maps;

[0119] (132) Export the drawn map into Apollo format and Carla format respectively;

[0120] (133) The exported Apollo format lacks map files. Generate the missing files according to the above generation steps and place the map in the map folder of the Apollo container;

[0121] (134) The exported Carla format map is imported into Carla according to the official instructions.

[0122] (2) The reasoning on the edge computing side is also deployed in a containerized manner based on Kubernetes. The specific working principle is as follows:

[0123] (21) A module is established on the edge node to obtain the data stream of the virtual world camera, radar and other sensing devices. This module provides a data interface for algorithm analysis. The specific process of obtaining the data stream of the sensing device is as follows:

[0124] (211) Connect to the simulation simulator Carla server through the Python API provided by Carla;

[0125] (212) Get the specified sensing device from Carla's Library, such as sensor.camera.rgb, sensor.other.radar, etc.

[0126] (213) Set the relevant parameters of the sensing device and its position and direction information relative to the Apollo virtual car, and place it on the Apollo virtual car.

[0127] (214) The data stream of the sensing device is obtained through the LISTEN method of the sensing device provided by Carla.

[0128] (22) The edge node deploys a websocket module, and the algorithm obtains data for analysis. The analysis results are transmitted to the Apollo car's industrial computer via websocket for the Apollo car to perform related actions;

[0129] (23) The algorithm to be tested (i.e., the computational reasoning algorithm for autonomous driving) obtains input and outputs results in a unified way. Depending on the algorithm function, it is started by specifying startup parameters when creating a container to determine the data flow that the algorithm needs to obtain and the analysis results are transmitted to the channel specified by Apollo through websocket.

[0130] In addition to a series of components related to the Apollo autonomous driving platform carried by Apollo itself, the S2 and Apollo car industrial computers mainly deploy coordinate system conversion modules and bridge modules. The specific principles are as follows:

[0131] (1) The main function of the coordinate system conversion module is to convert the GNSS information obtained by the Apollo car in the real world into coordinates in the Carla world. The scaling may occur according to the environment in which the Apollo car is located. The specific steps are as follows:

[0132] (11) In the real world, it is often difficult to find or build a road with the same slope as the virtual world. Due to the difficulty of surveying and mapping, it is also difficult to construct a virtual map with the same slope as the real world in the virtual world. Because the present invention is the movement of the Apollo virtual car in the virtual world driven by the real Apollo car, the transformation of the road height from the real world to the virtual world can be ignored, that is, when the real world coordinate system is converted to the virtual world coordinate system, the value in the z direction is set to be constant at 0.

[0133] (12) Since the test of complex road scenes usually manifests itself in the complexity of pedestrians and vehicles on the road, a road containing a crossroad is usually selected for testing. In Carla, the present invention selects four roads at the crossroad corresponding to the roads in the southeast, northwest, and northeast respectively;

[0134] (13) Plan the scope of the Apollo car in the real world, obtain the coordinates of the Apollo car's starting point, and obtain the coordinates of the preset intersection in the real world; correspondingly obtain the coordinates of the actual position of the Apollo virtual car in the virtual world and the coordinates of the intersection;

[0135] (14) Taking the intersection as the far point, the coordinates of the real-world intersection are marked as (x r ,y r )The virtual world crossroads is the origin and is denoted as (x v ,y v ), calculate the difference between the starting position of the real world coordinate system and the coordinates of the intersection, recorded as Δw1 (should be the difference of a single x-axis or y-axis, the coordinates of the other axis are the same), and the difference between the starting position and the intersection in the virtual world, recorded as Δw2. Recorded as the ratio of the difference in real world coordinates to the difference in virtual world coordinates;

[0136] (15) Through the above, we can obtain the conversion of real world coordinates to virtual world coordinates. For example, if the coordinates of the real world are (x, y, z), then the coordinates of the virtual world are ( 0).

[0137] (2) The main function of the bridge module is to bridge Carla on the edge node and calculate the inference results to the Apollo autonomous driving platform. It is mainly divided into two parts: writing messages to the Apollo channel and reading messages from the Apollo channel. Both use the Python API provided by Apollo to write and read the channel. The specific reading and writing of the Apollo channel are as follows:

[0138] (21) The main task of Apollo channel writing is to write the calculation and reasoning results of the edge nodes and the virtual world coordinates converted from the real world coordinates into the Apollo channel. Its working principle is as follows:

[0139] (211) Obtain the calculation and reasoning results through the websocket service provided by the edge node, and write the calculation and reasoning results into channels such as / apollo / perception / obstacles and / apollo / perception / traffic_light;

[0140] (212) The coordinate system conversion module on the Apollo car's industrial computer is called to convert the coordinates of the real world coordinate system obtained by the Apollo car into the coordinates of the virtual world coordinate system, and write them into the / apollo / localization / pose channel of Apollo.

[0141] (22) The main task of Apollo channel reading is to obtain the current position and posture information of the Apollo car, and push the changes of the Apollo car's position information and posture information to the virtual world through the websocket service, and synchronize the position information and posture information of the Apollo virtual car in the virtual world.

[0142] S3. The user end is mainly divided into two parts: the scene generator and the scene display interface. The main function of the scene generator is to generate a specified virtual scene through a pre-written scene code; the main function of the scene display interface is to provide a display interface for users to intuitively see the pictures in the virtual scene. The details are as follows:

[0143] (1) The virtual scene generator mainly constructs the virtual scene code through Scenario Runner, and initializes the position of each actor, writes the actor behavior and trigger conditions through the methods provided by it; through the websocket service provided above, it continuously obtains the position information and posture information of the Apollo virtual car in the virtual scene, and synchronously changes the position and posture of the virtual car through the set_transform method provided by Carla's API.

[0144] (2) The scene display interface connects to the simulation simulator Carla server through the Carla API and obtains the scene video stream. The main principles are as follows:

[0145] (21) Define the World class to represent the world in the CARLA simulator. It contains information about cars, sensors, and other entities. Create a CARLA client, obtain map information, and initialize various sensors (IMU, camera, etc.).

[0146] (22) Define the KeyboardControl class to process keyboard input, detect key events, switch user perspectives, record simulation data, etc.;

[0147] (23) Define the HUD class to display information related to the simulator status, such as server FPS, vehicle location, number of nearby vehicles and pedestrians, etc.;

[0148] (24) Define the CameraManager class to manage camera sensors, add multiple camera-type sensing devices, and record camera data from multiple angles;

[0149] (25) Define the game_loop method, which is the main game loop. In the loop, it handles keyboard events, updates the world state, renders the screen, etc.

[0150] (26) The startup interface shows how to control the car based on keyboard input, display HUD information, and render the camera image on the screen. The script uses the pygame library to implement window display and user input processing.

[0151] The Apollo car virtual and real combined autonomous driving test platform is built, such as Figure 2 As shown in the figure, it mainly includes three components: edge node, Apollo car industrial computer and user end.

[0152] The present invention migrates part of the computational reasoning to the edge node to make up for the problem of insufficient computing power. It only needs to focus on the accuracy of the algorithm, avoiding the optimization of the efficiency of the algorithm in the early stage. In addition, the edge segment provides a convenient interface for the algorithm to be tested to access data and output results, realizing plug-in testing. Using a real car with a virtual scene, the position of the real car in the real world is projected to the position in the virtual world through certain rules to realize the movement of the car to meet the needs of scene changes, avoiding the problem that the virtual vehicle cannot completely replicate the real vehicle.

[0153] An Apollo car automatic driving test platform construction system of the present invention includes:

[0154] The edge node construction unit is used to deploy the simulation simulator server and edge computing inference algorithm on the edge node. The simulation simulator server includes multiple actors, perception devices and Apollo virtual car. The perception device is set on the Apollo virtual car, and the data stream of the perception device is obtained to the edge computing inference algorithm for inference, and the inference calculation result is pushed to the Apollo autonomous driving platform;

[0155] The Apollo car industrial computer construction unit is used to deploy the Apollo autonomous driving platform carried by the Apollo car itself. The coordinate system conversion module and the bridge module are also deployed. The coordinate system conversion module converts the GNSS information obtained by the Apollo car in the real world into coordinates in the simulator world; the bridge module realizes the communication function between the simulator server on the edge node and the edge computing and reasoning algorithm and the Apollo autonomous driving platform, which is specifically composed of two parts: Apollo channel writing messages and Apollo channel reading messages;

[0156] The user-side construction unit is used to construct a scene generator and a scene display interface. The scene generator generates a specified virtual scene from a pre-written scene code, and the scene display interface intuitively displays the interface in the virtual scene.

[0157] like Figure 3 As shown, the communication process of the Apollo car virtual-real combined autonomous driving test platform constructed by the present invention is as follows:

[0158] (1) Open Apollo's autonomous driving platform DreanView, start the GPS module, and obtain real-world GPS information into Apollo's GNSS channel.

[0159] (2) Use the API officially provided by Apollo to obtain real-world coordinates from the GNSS channel as input to the coordinate conversion module.

[0160] (3) Use the coordinate conversion module to continuously convert the real world coordinate system into the virtual world coordinate system and write it into Apollo's location channel.

[0161] (4) The bridge module provides a websocket service interface, which continuously reads the coordinates of the current virtual world and the posture of the real car from the channel for push.

[0162] (5) The scene generator starts a thread that continuously obtains virtual position and posture push from the websocket service interface provided by the bridge module to synchronize the position and posture of the virtual car.

[0163] (6) After receiving the position and posture information of the virtual car, the position of the virtual scene actor and the trigger process are initialized to generate a virtual scene.

[0164] (7) The edge node generates the corresponding perception device and binds the virtual car through the API provided by Carla, and continuously obtains the data stream of these sensors.

[0165] (8) The edge node performs computational reasoning on the data stream obtained in step (7) and pushes the computational reasoning results through websocket.

[0166] (9) Obtain the calculation and reasoning results through the websocket service provided in step (8), and write the calculation and reasoning results into Apollo channels, such as obstacles, traffic_light, etc.

[0167] (10) At this time, the user terminal runs the scene display interface, connects to Carla of the edge node through the official API provided by Carla, and generates a camera from the audience's perspective.

[0168] (11) Obtain the video stream from the viewer’s perspective from the edge node and display it on the user’s terminal through pygame.

[0169] (12) At this time, the driving path is planned on the Apollo autonomous driving platform based on the imported virtual scene map, and a series of tests are started.

[0170] The method of the present invention for performing an autonomous driving test using the Apollo car virtual-real autonomous driving test platform constructed above comprises the following steps:

[0171] 1. The edge node runs the simulation simulator Carla server and some inference calculation programs:

[0172] (1) Download the Carla server image of the simulation simulator. Since the current Scenario Runner only supports Carla version 0.9.13, download the carlasim / carla:0.9.13 image;

[0173] (2) Run the simulation simulator Carla server image docker, map ports 2000-2002, and use the startup parameters of Carla, -perfernvidia and -RenderOffScreen, which respectively indicate the use of nvidia graphics card and running without interface.

[0174] (3) Run the inference computing program and continuously monitor the sensors of the Apollo virtual car.

[0175] 2. Apollo car industrial computer:

[0176] (1) Start the Boostrap of the Apollo autonomous driving platform carried by the Apollo car;

[0177] (2) Log in to Apollo's Dreamview interface, select the appropriate vehicle model, select the imported Carla map, and open the GPS, Canbus, Planning and other modules. Figure 3 ;

[0178] (3) Run the coordinate system conversion process in the container of the Apollo autonomous driving platform to ensure that the converted car coordinates are in the correct position on the map, such as Figure 4 The car is on the lane on the map;

[0179] (4) Run the bridge process in the container of the Apollo autonomous driving platform. When running the process, you need to determine the IP address of the simulation simulator Carla server and the server port of the developed websocket or grpc.

[0180] 3. Select the host connected to the Apollo car industrial computer and the simulation simulator Carla server to run the user program:

[0181] (1) Generate a virtual scene. Run the program to generate a virtual scene, ensure that the IP address and port of the Carla server of the simulation simulator and the IP address and port of the car position information server are correct, initialize the actors and the position of the Apollo virtual car, and generate a virtual scene.

[0182] (2) Set the destination of the Apollo car in the Dreamview interface;

[0183] (3) Run the scene display program, ensure the IP address and port of the simulation simulator Carla server, generate the audience perspective camera, and view the virtual scene through the audience perspective camera image stream. The final virtual scene and Dreamview presentation results are as follows: Figure 5 As shown in (a) and (b).

Claims

1. A method for constructing an Apollo car autonomous driving test platform, characterized in that: The following steps are involved: Build edge nodes, including deploying simulation simulator servers and edge computing inference algorithms on edge nodes. The simulation simulator server includes multiple actors, perception devices, and Apollo virtual cars. The perception devices are set on the Apollo virtual car, and the data streams of the perception devices are obtained for edge computing inference algorithms to calculate inferences, and the results of the inference calculations are pushed to the Apollo autonomous driving platform. The Apollo car industrial computer is built, including the Apollo autonomous driving platform carried by the Apollo car itself. The coordinate system conversion module and the bridge module are also deployed. The coordinate system conversion module converts the GNSS information obtained by the Apollo car in the real world into coordinates in the simulator world; the bridge module is used to realize the communication function between the simulator server on the edge node and the edge computing and reasoning algorithm and the Apollo autonomous driving platform, which is specifically divided into two parts: writing messages to the Apollo channel and reading messages from the Apollo channel; Constructing the user end includes constructing a scene generator and a scene display interface. The scene generator generates a specified virtual scene from the pre-written scene code, and the scene display interface intuitively displays the interface in the virtual scene.

2. The method for constructing an Apollo car automatic driving test platform according to claim 1, characterized in that: Before deploying the simulator server, you need to determine the map for the Apollo autonomous driving platform. The map for the Apollo autonomous driving platform is converted from the map that comes with the simulator server or the map you edited yourself. The missing files are generated in the Apollo container, specifically: Create a folder with the map name in the apollo / modules / map / data directory; Use generate_routing_topo_graph.sh provided in the Apollo container to generate routing_map related files; Use the sim_map_generator provided in the Apollo container to generate sim_map related files.

3. The method for constructing an Apollo car automatic driving test platform according to claim 1, characterized in that: The method to obtain the data stream of the sensing device is: Connect to the simulator server through API; Get the specified sensing device from the model library of the simulation simulator server; Set the relevant parameters of the perception device and its position and direction information relative to the Apollo virtual car, and place the perception device on the Apollo virtual car; The data stream of the perception device is obtained through the data interface of the perception device provided by the simulation simulator server.

4. The method for constructing an Apollo car automatic driving test platform according to claim 1, characterized in that: On the edge node, a data stream acquisition module and a websocket module are also deployed. The data stream acquisition module provides the data stream of the perception device to the edge computing and reasoning algorithm for computation and reasoning, and the websocket module transmits the computation and reasoning results to the Apollo car industrial computer for the car to perform related actions. The edge computing and reasoning algorithms obtain input and output results in a unified way. Depending on the algorithm functions, they are started by specifying startup parameters when creating a container to determine the data flow that the algorithm needs to obtain and the calculation and reasoning results are transmitted to the channel specified by Apollo through the websocket module and the bridge module.

5. The method for constructing an Apollo car automatic driving test platform according to claim 1, characterized in that: In the coordinate system conversion module, there is scaling according to the environment in which the Apollo car is located, specifically: When converting the real world coordinate system to the virtual world coordinate system, set the value in the z direction to be constant at 0; Choose a road that contains an intersection for testing; Plan the scope of the Apollo car in the real world, obtain the coordinates of the Apollo car's starting position, and obtain the coordinates of the preset intersection in the real world; correspondingly obtain the coordinates of the Apollo virtual car's starting position in the virtual world, and the coordinates of the intersection; Taking the intersection as the origin, the coordinates of the real-world intersection are marked as (x r ,y r ), the virtual world crossroads coordinates are marked as (x v ,y v ), calculate the x-axis or y-axis difference between the starting position of the real world coordinate system and the crossroad coordinate, recorded as Δw1, and the x-axis or y-axis difference between the starting position and the crossroad coordinate in the virtual world, recorded as Δw2, and calculate the ratio of the real world coordinate difference to the virtual world coordinate difference; The real world coordinates (x, y, z) are converted to virtual world coordinates as follows:

6. The method for constructing an Apollo car automatic driving test platform according to claim 1, characterized in that: Apollo channel write messages and Apollo channel read messages, specifically: The Apollo channel write message writes the edge node's calculation and reasoning results and the virtual world coordinates converted from the real world coordinates into the Apollo channel, including: Obtain the calculation and reasoning results through the websocket service provided by the edge node, and write the calculation and reasoning results to the / apollo / perception / obstacles and / apollo / perception / traffic_light channels; Call the coordinate system conversion module on the Apollo car's industrial computer to convert the coordinates of the real world coordinate system obtained by the Apollo car into the coordinates of the virtual world coordinate system and write them into the / apollo / localization / pose channel of Apollo; The Apollo channel reads messages including: obtaining the current position and posture information of the Apollo car, and pushing the changes of the position and posture information of the Apollo car to the virtual world through the websocket service, and synchronizing the position and posture information of the Apollo virtual car in the virtual world.

7. The method for constructing an Apollo car automatic driving test platform according to claim 1, characterized in that: When the simulation simulator server uses the simulation simulator Carla server, a scene generator and scene display interface are constructed, including: The virtual scene generator constructs the virtual scene code through Scenario Runner, initializes the positions of each actor, writes the actor's behavior and trigger conditions; through the websocket service, continuously obtains the position and posture information of the Apollo car in the virtual scene, and synchronously changes the position and posture of the Apollo virtual car through the set_transform method provided by the API of the simulator server; The scene display interface connects to the simulation simulator server through the simulation simulator server API and obtains the scene video stream.

8. An Apollo car automatic driving test platform, characterized in that: include: Edge node, Apollo car industrial computer and user end, the edge node includes a simulation simulator server and an edge computing and reasoning module, the simulation simulator server includes multiple actors, perception devices and Apollo virtual cars, and the perception devices are set on the Apollo virtual car; the Apollo car industrial computer includes the Apollo autonomous driving platform carried by the Apollo car itself, and also includes a coordinate system conversion module and a bridge module, the user end includes a scene generator module and a scene display interface; the scene generator module generates a virtual scene with multiple actors, perception devices and Apollo virtual cars according to needs, and displays it on the scene display interface, the edge computing and reasoning module performs computing and reasoning according to the data stream of the perception device, the bridge module is used to realize the communication function between the simulation simulator server on the edge node and the edge computing and reasoning algorithm and the Apollo autonomous driving platform, and the coordinate system conversion module converts the GNSS information obtained by the Apollo car in the real world into coordinates in the simulation simulator world.

9. An Apollo car autonomous driving test platform construction system, characterized in that: include: The edge node construction unit is used to deploy the simulation simulator server and edge computing inference algorithm on the edge node. The simulation simulator server includes multiple actors, perception devices and Apollo virtual car. The perception device is set on the Apollo virtual car, and the data stream of the perception device is obtained to the edge computing inference algorithm for inference, and the inference calculation result is pushed to the Apollo autonomous driving platform; The Apollo car industrial computer construction unit is used to deploy the Apollo autonomous driving platform carried by the Apollo car itself. The coordinate system conversion module and the bridge module are also deployed. The coordinate system conversion module converts the GNSS information obtained by the Apollo car in the real world into coordinates in the simulator world; the bridge module realizes the communication function between the simulator server on the edge node and the edge computing and reasoning algorithm and the Apollo autonomous driving platform, which is specifically composed of two parts: Apollo channel writing messages and Apollo channel reading messages; The user-side construction unit is used to construct a scene generator and a scene display interface. The scene generator generates a specified virtual scene from a pre-written scene code, and the scene display interface intuitively displays the interface in the virtual scene.

10. A virtual and real combined automatic driving test method for an Apollo car, characterized in that: include: The edge node runs the simulation simulator server and edge computing inference algorithm programs, including: (1) Download the simulator server image; (2) Run the simulation simulator server image, map the port, and determine the startup parameters; (3) Run the edge computing and reasoning algorithm program and continuously monitor the sensors of the Apollo virtual car; Operate the Apollo car industrial computer, including: (1) Launch Boostrap of the Apollo autonomous driving platform; (2) Log in to Apollo's Dreamview interface, select the appropriate vehicle model, select the imported map, and open the GPS, Canbus, and Planning modules; (3) Run the coordinate system conversion process in the container of the Apollo autonomous driving platform to ensure that the converted car coordinates are in the correct position on the map; (4) Run the bridge process in the container of the Apollo autonomous driving platform. When running the process, determine the IP address of the simulator server and the server port information of the developed websocket or grpc; Select the host connected to the Apollo car industrial computer and the simulation simulator server network to run the user-side program, including: (1) Generate a virtual scene Run the program to generate a virtual scene, ensure that the IP address and port of the simulation simulator server and the IP address and port of the car position information server are correct, initialize the actor and the position of the Apollo virtual car, and generate a virtual scene; (2) Set the destination of the Apollo car in the Dreamview interface; (3) Run the scene display program, ensure the IP address and port of the simulation simulator server, generate the audience perspective camera, and view the virtual scene image through the audience perspective camera image stream.

Images