Network communication automatic driving simulation test system and test method based on middleware

Abstract

The embodiment of the invention discloses a middleware-based network communication automatic driving simulation test system and test method, the system comprises a simulation system and an algorithm module, and the simulation system comprises a simulator and middleware; the simulator comprises a simulation engine assembly and a simulation client, and the simulation engine assembly is used for rendering a virtual environment and a sensor model, generating sensor synthetic data and transmitting the sensor synthetic data to the simulation client; and the simulation client transmits the main sensor data to the middleware in a Plugin mode, and transmits the other sensor data to the to-be-tested piece in a Proxy mode. According to the method, the problem that the quality of the synthesized image in the automatic driving closed loop HiL test system is reduced due to ISP processing of the to-be-tested piece is solved, the image does not need to be injected into a board card, the system complexity is reduced, the hardware cost and the labor cost are reduced, and the test efficiency is improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of autonomous driving technology, and specifically to a middleware-based network communication autonomous driving simulation test system and test method, electronic equipment, and storage medium. Background Art

[0002] With the continuous development of autonomous driving technology, the demand for end-to-end algorithm testing and verification is increasing. A very important part of end-to-end algorithm testing and verification is the HiL simulation test system. The HiL simulation test system (Hardware-in-the-Loop) is an important testing tool in the fields of autonomous driving, automotive electronics, etc., used to verify the functions and performance of hardware devices (such as sensors, controllers, actuators, etc.) in a virtual environment. Its core concept is to connect real hardware devices to the virtual simulation environment and test whether the hardware's response under various operating conditions meets expectations by simulating vehicle operation scenarios and external environments.

[0003] The existing HiL simulation test system has significant deficiencies in terms of the uncontrollability of the inverse ISP algorithm, the difficulty of system maintenance and adaptation, and is difficult to meet the requirements of efficient testing in complex scenarios. Specifically, Figure 1 As shown in the figure, the HiL simulation test solution based on the video injection board usually receives RGB images from the simulation software. This will cause the ISP module on the domain controller SoC to process the image twice when it is injected into the domain controller, which will further degrade the image quality. At the same time, the inverse ISP algorithm cannot achieve 100% of the original image and is difficult to develop. When the ISP algorithm changes, the corresponding inverse ISP algorithm needs to be re-debugged, which brings additional development costs. Secondly, the introduction of the injection board increases the debugging issues of drivers such as Serializer and Deserializer, which increases the maintenance and debugging costs of the HiL system. Summary of the Invention

[0004] In order to overcome the shortcomings of the existing technology, the purpose of the embodiments of the present invention is to provide a middleware-based network communication autonomous driving simulation test system and test method, electronic equipment and storage medium, which solves the problems of image quality distortion caused by dual ISP image processing and complex and difficult HiL system debugging, and reduces system costs.

[0005] To solve the above problems, the first aspect of the embodiments of the present invention discloses a middleware-based network communication autonomous driving simulation test system, which includes a simulation system and an algorithm module. The simulation system includes a simulator and middleware;

[0006] The simulator includes a simulation engine component and a simulation client. The simulation engine component is used for virtual environment rendering and sensor model rendering, generating sensor synthesis data, and transmitting the sensor synthesis data to the simulation client. The sensor synthesis data includes main sensor data and other sensor data generated by a camera model, a millimeter wave radar model, and a lidar model. The simulation client transmits the main sensor data to the middleware in a plug-in manner and transmits other sensor data to the device under test in a proxy manner.

[0007] The middleware is used to receive the main sensor data and transmit the main sensor data to the algorithm module, as well as receive the control signal transmitted by the algorithm module and transmit the control signal to the simulation client through the CAN module;

[0008] The algorithm module is used to sense the sensor synthesized data, generate a control signal after making a decision based on the sensor synthesized data, transmit the control signal to the middleware, and the middleware transmits the control signal to the simulator.

[0009] Furthermore, the simulation client includes a configurator and a runner;

[0010] The configurator is used to configure test cases and load configuration files on demand before the simulation is started to achieve parameter setting and modular combination; the runner is used to start simulation tasks and manage data communication processes.

[0011] Furthermore, the sensor model is configured with 7 cameras of different resolutions and 2 laser radars to send camera images or laser radar point clouds to the algorithm module at a frequency of 30FPS.

[0012] Furthermore, the simulation engine component is configured with four RTX 4090D graphics cards for virtual environment rendering and sensor model rendering. The sensor model rendering includes real-time rendering calculations of camera models and radar models. The real-time rendering calculations of the radar model include: loading the geometric data of the three-dimensional scene, selecting a scanning mode according to the type of lidar model, generating scanning rays, using an accelerated data structure to quickly find the nearest intersection of the scanning rays and the three-dimensional scene, calculating the three-dimensional coordinates of the intersection, and converting the calculated three-dimensional coordinates into point cloud data.

[0013] Furthermore, the virtual environment rendering includes: loading static models and dynamic 3D models including roads, buildings, and vehicles to construct a basic geometric scene; importing a map containing lane lines, traffic signs, and traffic lights, defining the road topology, and setting environmental variables including lighting, weather, and time; performing lighting and material rendering; simulating the behavior of dynamic objects to generate natural scene interactions.

[0014] Furthermore, the simulation client transmits the main sensor data to the middleware in a plugin manner, and transmits other sensor data to the device under test in a proxy manner, including:

[0015] The simulation client encapsulates the generated main sensor data according to the test requirements of the DUT, transmits it to the middleware of the DUT via 10 Gigabit Ethernet and the vehicle Ethernet switch, and sends other sensor data to the DUT via the CAN board.

[0016] A second aspect of an embodiment of the present invention discloses a testing method for a middleware-based network communication autonomous driving simulation test system, which includes the following steps:

[0017] Configure the test scenario and sensor model system and start the system;

[0018] Performing virtual environment rendering and sensor model rendering through the simulation engine component to achieve simulation environment rendering and obtain sensor synthesis data; the sensor synthesis data includes main sensor data and other sensor data generated by the camera model, millimeter wave radar model and lidar model;

[0019] The transmission of the main sensor data generated by the camera model, millimeter-wave radar model and lidar model is processed by plugin mode and published to the middleware according to the interface standard required by the middleware; other sensor data is transmitted to the device under test by proxy mode;

[0020] Transmit the received sensor synthesized data to the algorithm module;

[0021] The algorithm module generates a control signal based on the sensor synthesis data, and transmits the control signal to the middleware, and the middleware transmits the control signal to the simulator through the CAN bus.

[0022] Furthermore, the algorithm module includes a perception algorithm module and a decision algorithm module. When testing the obstacle avoidance effect, the target distance between the front obstacle and the vehicle is output through the perception algorithm module, and the brake signal is output through the decision algorithm module. The brake signal is fed back to the simulation engine based on the CAN signal. The simulation engine updates the position of the simulated vehicle's brake pedal based on the feedback pressure value, and updates the position of the simulated vehicle after braking to verify the obstacle avoidance effect.

[0023] The third aspect of an embodiment of the present invention discloses an electronic device, comprising: a memory storing executable program code; a processor coupled to the memory; the processor calls the executable program code stored in the memory to execute the test method of the middleware-based network communication autonomous driving simulation test system disclosed in the second aspect of an embodiment of the present invention.

[0024] The fourth aspect of an embodiment of the present invention discloses a computer-readable storage medium, characterized in that it stores a computer program, wherein the computer program enables a computer to execute the testing method of the middleware-based network communication autonomous driving simulation test system disclosed in the second aspect of an embodiment of the present invention.

[0025] Compared with the prior art, the embodiments of the present invention have the following advantages:

[0026] The present invention's middleware-based network communication autonomous driving simulation test system and test method implements closed-loop HiL testing through Ethernet-based domain controller-oriented middleware. The middleware manages simulated synthetic data and transmits it to the application layer algorithm module. It does not use a video injection board, thus avoiding image distortion caused by ISP processing on the underlying driver. At the same time, it streamlines the HiL system link, reduces Ser / Deser debugging and development work, and solves the problems of image quality distortion caused by dual ISP image processing and the complexity and difficulty of HiL system debugging.

[0027] This invention solves the problem of degraded quality of synthetic images in closed-loop HiL test systems for autonomous driving due to ISP processing of the device under test. It improves the architecture of closed-loop HiL test systems for autonomous driving, eliminates the need for image injection boards, and reduces system complexity. It also eliminates the need for video injection boards in autonomous driving HiL test systems, reducing hardware and labor costs by 30%. It also optimizes the HiL test system architecture, reduces the driver debugging and development work caused by the injection boards, and improves test efficiency by 48%. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic diagram of a system in the background technology of the present invention;

[0029] Figure 2 This is a schematic diagram of the architecture logic of a middleware-based network communication autonomous driving simulation test system provided by one embodiment of the present invention;

[0030] Figure 3 1 is a schematic diagram of a middleware-based network communication autonomous driving simulation test system provided by one embodiment of the present invention;

[0031] Figure 4 This is a flowchart of a middleware-based network communication autonomous driving simulation test method provided by one embodiment of the present invention;

[0032] Figure 5 It is a structural diagram of an electronic device disclosed in one embodiment of the present invention. DETAILED DESCRIPTION

[0033] This specific implementation manner is merely an explanation of an embodiment of the present invention, and it is not a limitation of the embodiment of the present invention. After reading this specification, those skilled in the art may make non-creative modifications to the embodiment as needed, but as long as it is within the scope of the claims of the embodiment of the present invention, it is protected by patent law.

[0034] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the embodiments of the present invention.

[0035] The term "comprise" and any variations thereof in the specification and claims of this application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus that includes a series of steps or units is not necessarily limited to those steps or units expressly listed, but may include other steps or units not expressly listed or inherent to such process, method, product or apparatus.

[0036] In the embodiments of the present invention, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present invention should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0037] The present invention implements closed-loop HiL testing through Ethernet-based middleware for domain controllers. The middleware manages simulated synthetic data and transmits it to the application layer algorithm module. Video injection boards are not used, avoiding image distortion caused by ISP processing on the underlying driver. At the same time, the present invention streamlines the HiL system link, reduces Ser / Deser debugging and development work, reduces system costs by 30%, and improves test efficiency by 48%. It solves the problems of image quality distortion caused by dual ISP image processing and the complexity and difficulty of HiL system debugging.

[0038] The professional terms in the present invention (including professional terms in the background technology, the patent scheme of the present invention, and the drawings) are explained as follows:

[0039] The FPD-Link III serializer-deserializer (SerDes) supports both single-ended and differential signal topologies. The FPD-Link III serializer transmits high-speed video data and control data to the deserializer in the forward direction, while the deserializer transmits low-speed reverse channel control data back to the serializer. FPD-Link III is a full-duplex, bidirectional communication link that shares the same transmission medium.

[0040] GMSL is a video transmission interface / standard based on SerDes technology. Based on SerDes technology, high-speed audio and video, as well as control signals can be transmitted through a single cable. The intelligent driving system only needs to use the GMSL interface chip to achieve signal transmission and two-way communication between the SoC and the camera and screen through a single cable.

[0041] Ser:Serializer (serializer) + Des:Deserializer (deserializer), stands for "serial deserializer".

[0042] The Image Signal Processor (ISP) is the key link between the image sensor and the terminal display. Traditional ISPs for automotive cameras use a series of digital image processing algorithms to optimize the raw signal output by the image sensor, addressing factors such as image noise, brightness, and color. This enables basic processing such as camera image correction, color restoration, image enhancement, and 3A statistics.

[0043] The Automated Driving Control Unit (ADCU) is specifically designed for intelligent driving systems. It receives data from multiple sensors, such as cameras, millimeter-wave radar, lidar, and cloud data transmission (V2X). It also acquires vehicle dynamics data (such as vehicle speed and pedal signals) through the VCU. The ADCU supports customized control strategies and decision execution for all inputs, and its outputs provide driving status feedback and execute various intelligent driving functions on the vehicle.

[0044] HiL test system (Hardware-in-the-Loop test system), hardware-in-the-loop system, simulates the input and output of the real physical world through a simulation environment, allowing the hardware under test (such as chips, controllers) to interact with "virtual objects" in a virtual environment, thereby completing closed-loop testing.

[0045] The plugin method is a program written in accordance with a certain standard application program interface. Through the plugin method, new functional modules or features can be added to the system without modifying the main program code.

[0046] The Proxy method is a structural design pattern that provides a proxy for other objects to control access to the object and plays the role of an intermediary between the client and the real object.

[0047] Example 1:

[0048] Please refer to Figure 2-5 As shown, a network communication autonomous driving simulation test system based on middleware of the present invention includes a simulation system and an algorithm module, wherein the simulation system includes a simulator and middleware;

[0049] Specifically, the simulator includes a simulation engine component and a simulation client. The simulation engine component is used for virtual environment rendering and sensor model rendering, generating sensor synthesis data, and transmitting the sensor synthesis data to the simulation client. The sensor synthesis data includes main sensor data and other sensor data generated by the camera model, millimeter wave radar model, and lidar model. The simulation client transmits the main sensor data to the middleware in a plug-in manner, and transmits other sensor data to the device under test in a proxy manner.

[0050] The test system architecture logic diagram of the present invention is as follows Figure 2 As shown, a 10V1R7L physical sensor configuration (i.e., 10 cameras, 1 millimeter-wave radar, and 7 lidars) is preferred. The system uses a plugin-based data processing model to achieve modular integration and stable call access for various sensor models. GNSS and ultrasonic sensors are connected to the system via proxies, simplifying data simulation and communication paths.

[0051] Preferably, the simulation client transmits the main sensor data to the middleware in a plugin manner, and transmits other sensor data to the device under test in a proxy manner, including:

[0052] The simulation client encapsulates the generated main sensor data according to the test requirements of the DUT, transmits it to the DUT's middleware via 10 Gigabit Ethernet and the vehicle Ethernet switch, and sends other sensor data to the DUT via the CAN board.

[0053] The other sensor data may specifically include GNSS and ultrasonic sensor data;

[0054] To support the rendering and data generation of highly complex sensor models, the system is equipped with four RTX4090D graphics cards, dedicated to real-time rendering and calculation of camera and radar models. The generated primary sensor data is packaged according to the DUT's test requirements and transmitted to the middleware via 10 Gigabit Ethernet and an onboard Ethernet switch. Other sensor data accessed via a proxy is sent to the middleware via a CAN card connected to the PCIe interface, establishing a control bus communication link. After receiving the sensor data, the middleware returns control signals via the CAN bus, enabling closed-loop simulation and test verification.

[0055] The middleware is used to receive the main sensor data and transmit the main sensor data to the algorithm module, as well as receive the control signal transmitted by the algorithm module and transmit the control signal to the simulation client through the CAN module;

[0056] Among them, the lowest level data generated by the original sensor model is used to simulate the original signal acquisition process of the real sensor;

[0057] The algorithm module is used to perceive the sensor synthesized data, and generate a control signal after making decisions and planning paths based on the sensor synthesized data, and transmit the control signal to the middleware, which then transmits the control signal to the simulator.

[0058] Specifically, the algorithm module includes a perception algorithm module and a decision algorithm module. The perception algorithm module is used to convert the sensor-synthesized data into machine-understandable environmental information. The decision algorithm module is used to generate a computational process for reasonable behavioral decisions, such as forward movement, steering, braking, and path planning, based on the environmental information provided by the perception algorithm module and its own state, such as speed, position, and mission objectives.

[0059] In a specific embodiment, the algorithm module can be designed at the application layer and implemented by the ADCU microprocessor.

[0060] Wherein, the simulation client includes a configurator and a runner;

[0061] The configurator is used to configure test cases and load configuration files on demand before the simulation is started to achieve parameter setting and modular combination; the runner is used to start simulation tasks and manage data communication processes.

[0062] The specific application embodiments of the middleware-based network communication autonomous driving simulation test system of the present invention are as follows:

[0063] The simulation test system of the present invention is as follows Figure 3As shown, it includes the rendering station simulator and the middleware for the device under test. The simulator is divided into two parts: the simulation engine and the simulation client. The simulation client mainly consists of the configurator and the runner. The simulation engine component provides the underlying calculation for the HiL system simulation environment rendering, including virtual environment rendering and sensor model rendering. The configurator provides the function of configuring the test cases of the HiL test system, and the runner provides the function of starting the simulation process.

[0064] The simulation engine component and the simulation client will transmit data through memory and binary methods. The simulation engine component will transmit data between modules based on TCP.

[0065] The sensor model's data is sent in two forms: injected into the domain controller via standard Ethernet and CAN bus communications, and returned via CAN signals to the ADAS as control signals to complete the closed-loop HiL test. Standard Ethernet needs to be converted to in-vehicle Ethernet to achieve final communication, so this solution uses 802.1AS for time synchronization.

[0066] The proxy-based approach places less emphasis on data stability and allows for rapid data processing in this simple mode. For time-sensitive camera / lidar and millimeter-wave radar data, since the perception algorithm has high requirements for data input stability and quality, it is deployed in a plugin format, enabling the most stable delivery of high-quality sensor data. The data repackaging method will depend on the requirements of the DUT's middleware, supporting various protocols such as Proto and RosMsg.

[0067] exist Figure 3 In the system, the simulation engine component is responsible for rendering and generating autonomous driving test scenarios and various sensor models to ensure the authenticity and controllability of the simulation environment.

[0068] On the client side, the system development architecture is divided into two core modules: the Configurator and the Runner. The Configurator loads configuration files for different simulation modules on demand before simulation starts, enabling flexible parameter settings and modular combination. The Runner is responsible for starting simulation tasks and managing data communication processes.

[0069] Within the launcher, the system's data transmission methods are categorized into two types: proxy mode and plugin mode. The proxy mode primarily parses and forwards control signals and processes vehicle status information, such as CAN signals and status feedback, offering advantages such as a simple communication structure and low latency. The plugin mode, targeted at time-sensitive data transmission, encapsulates the raw data from camera, millimeter-wave radar, and lidar models, and publishes it according to the interface standards required by the DUT's middleware, ensuring data format and protocol consistency.

[0070] In this embodiment, the middleware is designed into the DUT. The middleware and CAN communication module in the DUT receive the perception data published by the simulation system and pass it to the application-layer algorithm module. After the algorithm module completes the perception, decision-making, and path planning processes, the generated control signal is returned to the simulation system via the CAN bus, completing the closed-loop testing and verification process of perception, decision-making, and control.

[0071] As a specific embodiment, the sensor model is configured with 7 cameras of different resolutions and 2 laser radars to send camera images or laser radar point clouds to the algorithm module at a frequency of 30FPS.

[0072] As a specific embodiment, the simulation engine component is configured with a four-way RTX 4090D graphics card for virtual environment rendering and sensor model rendering, wherein the sensor model rendering includes real-time rendering calculation of camera model and radar model;

[0073] Specifically, the real-time rendering calculation of the radar model includes: loading the geometric data of the three-dimensional scene, selecting a scanning mode according to the type of the lidar model, generating scanning rays, using an accelerated data structure to quickly find the nearest intersection point between the scanning rays and the three-dimensional scene, calculating the three-dimensional coordinates of the intersection point, and converting the calculated three-dimensional coordinates into point cloud data.

[0074] Specifically, the virtual environment rendering includes: loading static models and dynamic 3D models including roads, buildings, and vehicles to build a basic geometric scene; importing a map containing lane lines, traffic signs, and traffic lights, defining the road topology, and setting environmental variables including lighting, weather, and time; performing lighting and material rendering; simulating the behavior of dynamic objects to generate natural scene interactions.

[0075] Example 2:

[0076] Please refer to Figure 4 As shown, Figure 4 The present invention shows a middleware-based network communication autonomous driving simulation test method, which uses the above-mentioned test system and specifically includes the following steps:

[0077] Step S1: Configure the test scenario and sensor model system and start the system;

[0078] In this step, the test scenarios may specifically include emergency braking scenarios, cornering scenarios, and CAN bus failure test scenarios.

[0079] Step S2: Rendering a virtual environment and a sensor model using a simulation engine component to achieve simulation environment rendering and obtain sensor synthesis data; the sensor synthesis data includes primary sensor data and other sensor data generated by a camera model, a millimeter-wave radar model, and a lidar model;

[0080] In this step, the virtual environment rendering includes: loading static models and dynamic 3D models including roads, buildings, and vehicles to build a basic geometric scene; importing a map containing lane lines, traffic signs, and traffic lights, defining the road topology, and setting environmental variables including lighting, weather, and time; performing lighting and material rendering; simulating the behavior of dynamic objects to generate natural scene interactions.

[0081] The sensor model rendering includes real-time rendering calculations of the camera model and the radar model. The real-time rendering calculations of the camera model can generate highly realistic synthetic images by accurately modeling the camera's optical characteristics, imaging process, and noise distribution.

[0082] The real-time rendering calculation of the radar model includes: loading the geometric data of the three-dimensional scene, selecting a scanning mode according to the type of the lidar model, generating scanning rays, using an acceleration data structure to quickly find the nearest intersection point between the scanning rays and the three-dimensional scene, calculating the three-dimensional coordinates of the intersection point, and converting the calculated three-dimensional coordinates into point cloud data.

[0083] Step S3: The transmission of the main sensor data generated by the camera model, millimeter wave radar model and lidar model is processed by plugin mode and published to the middleware according to the interface standard required by the middleware;

[0084] In this embodiment, the primary sensor data may specifically include a synthetic image and point cloud data rendered by a sensor model.

[0085] Step S4: The middleware transmits the received data to the algorithm module;

[0086] Specifically, the middleware is the middleware of the device under test, and the algorithm module includes a perception algorithm module and a decision algorithm module.

[0087] The perception algorithm module is used to convert the sensor-synthesized data into machine-understandable environmental information.

[0088] Specifically, the following steps are included:

[0089] Step S41: Acquire sensor synthesized data and preprocess the sensor synthesized data. The preprocessing may specifically include: using a filtering algorithm (such as Gaussian filtering, median filtering, bilateral filtering) to remove sensor noise (such as camera salt and pepper noise, point cloud outliers), and perform calibration and synchronization, such as calibrating the sensor, correcting sensor inherent errors such as camera lens distortion and lidar non-uniformity, aligning the timestamps and spatial coordinate systems of multi-sensor data, such as projecting the camera image and lidar point cloud into the same coordinate system through an extrinsic parameter matrix.

[0090] Step S42: Extract feature information representing environmental characteristics from the preprocessed data, such as edges, corners, target contours, etc.

[0091] Step S43: Identify targets based on feature information, such as vehicles, pedestrians, and obstacles, and determine their locations, categories, and attributes;

[0092] Step S44: Associating the same target in consecutive frame data to achieve cross-frame tracking and output the target's motion trajectory;

[0093] Step S45: performing high-level semantic analysis on the perceived scene, constructing a structured representation of the environment, and obtaining a geometric map or semantic map of the constructed environment;

[0094] Step S46: Output the perception results such as target location, category, and trajectory, and convert them into a format that can be received by the decision algorithm module and send them to the decision algorithm module.

[0095] Among them, the decision algorithm module is a calculation process used to generate reasonable behavioral decisions such as moving forward, turning, braking, and path planning based on the environmental information provided by the perception algorithm module and its own status, such as speed, position, and task objectives.

[0096] Step S5: The algorithm module generates a control signal based on the data, transmits the control signal to the middleware, and the middleware transmits the control signal to the simulator via the CAN bus.

[0097] The control signals may specifically include steering angle (±45°), accelerator pedal (0-100%), brake pressure (0-2000kPa), etc., and the control signals are mapped via CAN frames. For example, the steering angle (±45°) is transmitted via the first two bytes of the 0x010 data segment.

[0098] During specific implementation, when testing the obstacle avoidance effect, the target distance between the front obstacle and the vehicle is output through the perception algorithm module, and the braking signal is output through the decision algorithm module. The braking signal is fed back to the simulation engine based on the CAN signal. The simulation engine updates the position of the simulated vehicle's brake pedal based on the feedback pressure value, and updates the position of the simulated vehicle after braking to verify the obstacle avoidance effect.

[0099] For example, the perceived target distance is 5 meters from a stationary vehicle ahead, the control signal is expected to be brake pressure = 100%, throttle = 0%, and the closed-loop result verifies that the vehicle stops within 3 meters.

[0100] Among them, the perception algorithm module is used to analyze perception data and obtain the perception results of sensor data generated by the camera and LiDAR sensor. The perception data may include the target position, distance, type, etc. of the obstacle.

[0101] Specifically, when the HiL system of the present invention is running, the main test scenarios and sensor model system of the simulation system are first configured. After starting the system, the simulation engine component will render the simulation environment and generate sensor synthetic data. The simulation client processes the data through proxy and plugin based on the test requirements.

[0102] For example, the sensor model is a camera model. Taking the camera model data processing as an example, in both Proxy and Plugin modes, the camera sensor model performs imaging according to the material rendering-optical lens simulation-CMOS simulation and image signal processing (ISP) process. HDR images, RAW images, and RGBA images after ISP can be extracted from the memory at different stages. For the Proxy proxy mode, a general class function is designed that enables any target to obtain image data through this proxy without having to process the rendering engine port and data encoding. For the Plugin plug-in mode, it is a highly customized data processing mode, which is usually edited directly for the original data according to the target to be tested to ensure the stability of data transmission and reduce link delay.

[0103] As a specific embodiment, when the device under test is the Horizon Journey 6 (Horizon Journey 6 is a high-performance automotive-grade AI chip, mainly used for autonomous driving visual perception tasks), when processed based on the Plugin method, the specific steps include: sensor model instantiation, sensor model configuration, asynchronous reading and data packaging, and final transmission. Asynchronous reading will allow a certain amount of time overlap between the rendering engine and data transmission to ensure the stability of data transmission. At the same time, based on the requirements of the Journey 6 middleware, in the asynchronous processing function, the serial image data is packaged and encoded in the Protobuf format (full name Protocol Buffers, a lightweight and efficient structured data serialization format developed by Google). In the sending function, it is sent based on the sending interface provided by the Journey 6 middleware, and the stability of the sending function is ensured through a reasonable process lock mechanism.

[0104] In this embodiment, the Plugin-based approach can improve data transmission stability by more than 24%, and can effectively improve the reliability of the data link in time-sensitive data transmission.

[0105] In the above implementation process, the main sensor model data will follow the plugin method, be packaged in a specified format, and passed to the Ethernet conversion module via standard Ethernet. Finally, it will be sent to the domain controller middleware for reception. The middleware then passes the data to the application layer algorithm module. This eliminates the need for board debugging, optimizes the traditional HiL test solution system architecture, and avoids the problem of reduced quality of synthetic image RGB data due to ISP processing on the underlying link of the domain controller.

[0106] In specific implementation, the method of the present invention can be tested and verified on multiple domain controller middleware. Under the sensor configuration of 7 cameras with different resolutions and 2 lidars, it can stabilize the injection of 30FPS and meet the algorithm requirements of the algorithm module.

[0107] Example 3:

[0108] See also Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of the present invention. Figure 5 As shown, the electronic device may include:

[0109] A memory 310 storing executable program code;

[0110] a processor 320 coupled to the memory 310;

[0111] Among them, the processor 320 calls the executable program code stored in the memory 310 to execute some or all steps in the testing method of the middleware-based network communication autonomous driving simulation test system in Example 1.

[0112] An embodiment of the present invention discloses a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute some or all of the steps in a testing method of a middleware-based network communication autonomous driving simulation test system in embodiment one.

[0113] An embodiment of the present invention also discloses a computer program product, wherein when the computer program product runs on a computer, it enables the computer to execute some or all of the steps in the testing method of a middleware-based network communication autonomous driving simulation test system in embodiment one.

[0114] An embodiment of the present invention also discloses an application publishing platform, wherein the application publishing platform is used to publish a computer program product. When the computer program product runs on a computer, the computer executes some or all of the steps in a testing method of a middleware-based network communication autonomous driving simulation test system in embodiment one.

[0115] In various embodiments of the present invention, it should be understood that the size of the serial numbers of the processes does not necessarily mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0116] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the objectives of this embodiment as needed.

[0117] In addition, the functional units in the embodiments of the present invention may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The integrated unit may be implemented in the form of hardware or software functional units.

[0118] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-accessible memory. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a memory and includes several requests for causing a computer device (which can be a personal computer, server, or network device, specifically a processor in the computer device) to execute some or all of the steps of the method described in each embodiment of the present invention.

[0119] In the embodiments provided herein, it should be understood that "B corresponding to A" means that B is associated with A and B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B based solely on A; B can also be determined based on A and / or other information.

[0120] Those skilled in the art will appreciate that some or all of the steps in the various methods of the embodiments may be performed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, including a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0121] The above is a detailed introduction to the test method, device, electronic device and storage medium of a middleware-based network communication autonomous driving simulation test system disclosed in an embodiment of the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A middleware-based network communication autonomous driving simulation test system, characterized in that: It includes a simulation system and an algorithm module, wherein the simulation system includes a simulator and middleware; The simulator includes a simulation engine component and a simulation client. The simulation engine component is used for virtual environment rendering and sensor model rendering, generating sensor synthesis data, and transmitting the sensor synthesis data to the simulation client. The sensor synthesis data includes main sensor data and other sensor data generated by a camera model, a millimeter wave radar model, and a lidar model. The simulation client transmits the main sensor data to the middleware in a plug-in manner and transmits other sensor data to the device under test in a proxy manner. The middleware is used to receive the main sensor data and transmit the main sensor data to the algorithm module, as well as receive the control signal transmitted by the algorithm module and transmit the control signal to the simulation client through the CAN module; The algorithm module is used to sense the sensor synthesized data, generate a control signal after making a decision based on the sensor synthesized data, transmit the control signal to the middleware, and the middleware transmits the control signal to the simulator.

2. The middleware-based network communication autonomous driving simulation test system according to claim 1, characterized in that: The simulation client includes a configurator and a runner; The configurator is used to configure test cases and load configuration files on demand before simulation starts to achieve parameter setting and modular combination; The runner is used to start simulation tasks and manage data communication processes.

3. The middleware-based network communication autonomous driving simulation test system according to claim 1, characterized in that: The sensor model is configured with 7 cameras of different resolutions and 2 lidars to send camera images or lidar point clouds to the algorithm module at a frequency of 30FPS.

4. The middleware-based network communication autonomous driving simulation test system according to claim 1, characterized in that: The simulation engine component is equipped with four RTX 4090D graphics cards for virtual environment rendering and sensor model rendering, including real-time rendering calculations of camera models and radar models; The real-time rendering calculation of the radar model includes: loading the geometric data of the three-dimensional scene, selecting a scanning mode according to the type of the lidar model, generating scanning rays, using an acceleration data structure to quickly find the nearest intersection point between the scanning rays and the three-dimensional scene, calculating the three-dimensional coordinates of the intersection point, and converting the calculated three-dimensional coordinates into point cloud data.

5. The middleware-based network communication autonomous driving simulation test system according to claim 1, characterized in that: The virtual environment rendering includes: loading static models and dynamic 3D models including roads, buildings, and vehicles to build a basic geometric scene; importing a map containing lane lines, traffic signs, and traffic lights, defining the road topology, and setting environmental variables including lighting, weather, and time; performing lighting and material rendering; and simulating the behavior of dynamic objects to generate natural scene interactions.

6. The middleware-based network communication autonomous driving simulation test system according to claim 1, characterized in that: The simulation client transmits the main sensor data to the middleware in a plugin manner and transmits other sensor data to the device under test in a proxy manner, including: The simulation client encapsulates the generated main sensor data according to the test requirements of the DUT, transmits it to the middleware of the DUT via 10 Gigabit Ethernet and the vehicle Ethernet switch, and sends other sensor data to the DUT via the CAN board.

7. A test method for a middleware-based network communication autonomous driving simulation test system according to any one of claims 1 to 6, characterized in that: It includes the following steps: Configure the test scenario and sensor model system and start the system; Performing virtual environment rendering and sensor model rendering through the simulation engine component to achieve simulation environment rendering and obtain sensor synthesis data; the sensor synthesis data includes main sensor data and other sensor data generated by the camera model, millimeter wave radar model and lidar model; The transmission of the main sensor data generated by the camera model, millimeter-wave radar model and lidar model is processed by plugin mode and published to the middleware according to the interface standard required by the middleware; other sensor data is transmitted to the device under test by proxy mode; Transmit the received sensor synthesized data to the algorithm module; The algorithm module generates a control signal based on the sensor synthesis data, and transmits the control signal to the middleware, and the middleware transmits the control signal to the simulator through the CAN bus.

8. The middleware-based network communication autonomous driving simulation test method according to claim 7, characterized in that: The algorithm module includes a perception algorithm module and a decision algorithm module. When testing the obstacle avoidance effect, the target distance between the front obstacle and the vehicle is output through the perception algorithm module, and the brake signal is output through the decision algorithm module. The brake signal is fed back to the simulation engine based on the CAN signal. The simulation engine updates the position of the simulated vehicle's brake pedal based on the feedback pressure value, and updates the position of the simulated vehicle after braking to verify the obstacle avoidance effect.

9. An electronic device, characterized in that: It includes: a memory storing executable program code; a processor coupled to the memory; the processor calls the executable program code stored in the memory to execute the test method of the middleware-based network communication autonomous driving simulation test system as described in any one of claims 7-8.

10. A computer-readable storage medium, characterized in that It stores a computer program, wherein the computer program enables a computer to execute the testing method of the middleware-based network communication autonomous driving simulation test system as described in any one of claims 7-8.

Images