Method and apparatus for simulating vehicle end sensor data based on roadside sensors

Abstract

The application relates to the technical field of vehicle control, in particular to a method and device for simulating vehicle end sensor data based on roadside sensors, wherein the method comprises the following steps: generating roadside three-dimensional scene data of roadside sensors based on roadside sensor data of the roadside sensors; generating vehicle body roadside three-dimensional scene data of the roadside three-dimensional scene data in a vehicle body coordinate system based on the roadside three-dimensional scene data and a vehicle body coordinate system of vehicle end sensors; and rendering the vehicle body roadside three-dimensional scene data to obtain simulation data of the roadside sensors simulating the vehicle end sensors. Thus, the problems in the prior art, such as high difficulty, high cost, complex installation, difficult maintenance and the like of data collection and processing, and the difficulty in meeting the increasing demand for vehicle end sensor data, are solved.

Description

Technical Field

[0001] This application relates to the field of vehicle control technology, and in particular to a method and apparatus for simulating vehicle-side sensor data based on roadside sensors. Background Technology

[0002] Vehicle-mounted sensor data is a crucial source of information for autonomous driving systems to perceive their surroundings, make decisions, and execute actions. This data interacts with other modules of the autonomous driving system to jointly achieve autonomous driving functions. Currently, vehicle-mounted sensor data is primarily collected using sensors equipped on the vehicle itself, such as cameras, LiDAR, and ultrasonic sensors.

[0003] In related technologies, the vehicle-mounted unit can send a roadside data provision request to the cloud backend. When the roadside unit detects that a vehicle has entered its signal detection area, it will send the roadside data from the roadside unit to the vehicle-mounted unit. Alternatively, according to the vehicle-road cooperative system, the roadside base station can receive various information transmitted by the vehicle-mounted terminal and end-side equipment within the communication range. Through the comprehensive processing of the cloud control platform and the roadside base station, the final decision result of the vehicle-mounted terminal can be obtained.

[0004] However, the relevant technologies are difficult to acquire and process, costly, complex to install, and difficult to maintain, making it hard to meet the growing demand for vehicle-side sensor data and urgently needing improvement. Summary of the Invention

[0005] This application provides a method and apparatus for simulating vehicle-side sensor data based on roadside sensors, in order to solve the problems in related technologies such as high difficulty and cost of data acquisition and processing, complex installation, and difficult maintenance, which make it difficult to meet the growing demand for vehicle-side sensor data.

[0006] The first aspect of this application provides a method for simulating vehicle-side sensor data based on roadside sensors, comprising the following steps: generating roadside three-dimensional scene data of the roadside sensors based on the roadside sensor data; generating vehicle-side roadside three-dimensional scene data of the roadside three-dimensional scene data in the vehicle-side coordinate system based on the roadside three-dimensional scene data and the vehicle-side coordinate system of the vehicle-side sensors; and rendering the vehicle-side roadside three-dimensional scene data to obtain simulation data of the roadside sensors simulating the vehicle-side sensors.

[0007] Optionally, in one embodiment of this application, before generating the roadside three-dimensional scene data of the roadside sensor based on the roadside sensor data of the roadside sensor, the method further includes: acquiring initial roadside sensor data of the roadside sensor; acquiring initial vehicle-end sensor data of the vehicle-end sensor; and performing data preprocessing on the initial roadside sensor data and the initial vehicle-end sensor data to obtain roadside sensor data and vehicle-end sensor data that meet preset matching conditions.

[0008] Optionally, in one embodiment of this application, the step of generating roadside 3D scene data of the roadside sensor based on roadside sensor data includes: obtaining initial roadside point cloud data of the roadside sensor based on roadside radar data and / or roadside camera data in the roadside sensor data; generating initial roadside 3D scene data of the roadside sensor using the initial roadside point cloud data; and calculating the roadside 3D scene data of the roadside sensor based on the initial roadside 3D scene data.

[0009] Optionally, in one embodiment of this application, rendering the roadside three-dimensional scene data of the vehicle body to obtain the simulated data of the roadside sensor simulating the vehicle-end sensor includes: determining whether the simulated data and the vehicle-end sensor data meet preset evaluation conditions; if the simulated data does not meet the preset evaluation conditions, then re-acquiring the simulated data based on the roadside sensor and the vehicle-end sensor until the simulated data meets the preset evaluation conditions.

[0010] Optionally, in one embodiment of this application, the step of calculating the roadside three-dimensional scene data of the roadside sensor based on the initial roadside three-dimensional scene data includes: obtaining the true three-dimensional scene data of the roadside sensor based on the initial roadside three-dimensional scene data; obtaining the loss function of the roadside sensor based on the true three-dimensional scene data; and calculating the roadside three-dimensional scene data through the loss function.

[0011] A second aspect of this application provides an apparatus for simulating vehicle-side sensor data based on roadside sensors, comprising: a first generation module for generating roadside three-dimensional scene data of the roadside sensors based on the roadside sensor data; a second generation module for generating vehicle-side roadside three-dimensional scene data of the roadside three-dimensional scene data in the vehicle-side coordinate system based on the roadside three-dimensional scene data and the vehicle-side coordinate system of the vehicle-side sensors; and a first acquisition module for rendering the vehicle-side roadside three-dimensional scene data to acquire the simulation data of the roadside sensors simulating the vehicle-side sensors.

[0012] Optionally, in one embodiment of this application, it further includes: a second acquisition module, used to acquire initial roadside sensor data of the roadside sensor before generating roadside three-dimensional scene data of the roadside sensor based on the roadside sensor data of the roadside sensor; a third acquisition module, used to acquire initial vehicle-end sensor data of the vehicle-end sensor; and a preprocessing module, used to perform data preprocessing on the initial roadside sensor data and the initial vehicle-end sensor data to obtain roadside sensor data and vehicle-end sensor data that meet preset matching conditions.

[0013] Optionally, in one embodiment of this application, the first generation module includes: a first generation unit, configured to obtain initial roadside point cloud data of the roadside sensor based on roadside radar data and / or roadside camera data in the roadside sensor data; a second generation unit, configured to generate initial roadside three-dimensional scene data of the roadside sensor using the initial roadside point cloud data; and a calculation unit, configured to calculate the roadside three-dimensional scene data of the roadside sensor based on the initial roadside three-dimensional scene data.

[0014] Optionally, in one embodiment of this application, the first acquisition module includes: a first judgment unit, configured to judge whether the simulated data and the vehicle-side sensor data meet preset evaluation conditions; and an acquisition unit, configured to, when the simulated data does not meet the preset evaluation conditions, re-acquire the simulated data based on the roadside sensor and the vehicle-side sensor until the simulated data meets the preset evaluation conditions.

[0015] Optionally, in one embodiment of this application, the calculation unit includes: a first generation subunit, configured to obtain the real three-dimensional scene data of the roadside sensor based on the initial three-dimensional scene data of the roadside; a second generation subunit, configured to obtain the loss function of the roadside sensor based on the real three-dimensional scene data; and a calculation subunit, configured to calculate the roadside three-dimensional scene data using the loss function.

[0016] A third aspect of this application provides a vehicle, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for simulating vehicle-side sensor data based on roadside sensors as described in the above embodiments.

[0017] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for simulating vehicle-side sensor data based on roadside sensors.

[0018] A fifth aspect of this application provides a computer program product, including a computer program that, when executed, implements the above-described method for simulating vehicle-side sensor data based on roadside sensors.

[0019] This application embodiment can obtain roadside 3D scene data based on roadside sensor data, and then combine it with the vehicle coordinate system of the vehicle-mounted sensors to generate vehicle-mounted 3D scene data of the roadside 3D scene data in the vehicle coordinate system. The vehicle-mounted 3D scene data is then rendered to obtain simulated data from the roadside sensors simulating the vehicle-mounted sensors. Through 3D scene reconstruction and coordinate transformation, simulated data from the vehicle-mounted sensors can be effectively generated using roadside sensor data, providing richer and more reliable information for the perception module of the autonomous driving system. This has high practical value in real-world applications and fully utilizes relatively inexpensive and readily available roadside sensor data to generate abundant vehicle-mounted sensor data, reducing the cost required for training the autonomous driving system. Therefore, it solves the problems in related technologies, such as high difficulty and cost of data acquisition and processing, complex installation, and difficult maintenance, making it difficult to meet the ever-increasing demand for vehicle-mounted sensor data.

[0020] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0021] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0022] Figure 1 This is a flowchart illustrating a method for simulating vehicle-side sensor data based on roadside sensors, according to an embodiment of this application.

[0023] Figure 2 This is a flowchart illustrating the generation of roadside 3D scene data according to an embodiment of this application;

[0024] Figure 3 This is a flowchart illustrating the working principle of a method for simulating vehicle-side sensor data based on roadside sensors according to an embodiment of this application;

[0025] Figure 4 This is a block diagram of a device for simulating vehicle-side sensor data based on roadside sensors according to an embodiment of this application;

[0026] Figure 5 This is a structural schematic diagram of a vehicle provided according to an embodiment of this application. Detailed Implementation

[0027] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0028] The following describes a method and apparatus for simulating vehicle-side sensor data based on roadside sensors, according to embodiments of this application, with reference to the accompanying drawings. Addressing the problems mentioned in the background art, such as the difficulty and high cost of data acquisition and processing, complex installation, and difficult maintenance, which make it difficult to meet the ever-increasing demand for vehicle-side sensor data, this application provides a method for simulating vehicle-side sensor data based on roadside sensors. In this method, roadside 3D scene data can be obtained based on roadside sensor data from roadside sensors. Then, combined with the vehicle coordinate system of the vehicle-side sensors, vehicle-side 3D scene data in the vehicle coordinate system is generated. This vehicle-side 3D scene data is then rendered to obtain simulated data of the roadside sensors simulating the vehicle-side sensors. Through 3D scene reconstruction and coordinate transformation, simulated data of the vehicle-side sensors can be effectively generated using roadside sensor data, providing richer and more reliable information for the perception module of the autonomous driving system. This method has high practical value in real-world applications and fully utilizes the relatively inexpensive and readily available data from roadside sensors to generate rich vehicle-side sensor data, reducing the cost required for training autonomous driving systems. This solves the problems in related technologies, such as the difficulty and high cost of data acquisition and processing, complex installation, and difficult maintenance, which make it difficult to meet the growing demand for vehicle-side sensor data.

[0029] Specifically, Figure 1 This is a flowchart of a method for simulating vehicle-side sensor data based on roadside sensors, according to an embodiment of this application.

[0030] like Figure 1 As shown, the method for simulating vehicle-side sensor data based on roadside sensors includes the following steps:

[0031] In step S101, roadside three-dimensional scene data of the roadside sensor is generated based on the roadside sensor data of the roadside sensor.

[0032] It is understood that the roadside sensors in the embodiments of this application can be installed on both sides of the road, on bridges, or on the structure of traffic lights. They may include, but are not limited to, roadside radar sensors, roadside lidar sensors, and roadside camera sensors. The specific settings can be configured by those skilled in the art according to the actual situation, and this application does not impose any specific limitations.

[0033] Furthermore, in the embodiments of this application, the roadside three-dimensional scene data may include, but is not limited to, roadside radar data, roadside radar point cloud data, roadside lidar data, roadside lidar point cloud data, roadside two-dimensional image data, etc., and this application does not impose specific limitations.

[0034] As one possible approach, embodiments of this application can generate roadside 3D scene data based on roadside sensor data collected in real time by roadside sensors.

[0035] Optionally, in one embodiment of this application, generating roadside 3D scene data of roadside sensors based on roadside sensor data includes: obtaining initial roadside point cloud data of roadside sensors based on roadside radar data and / or roadside camera data in the roadside sensor data; generating initial roadside 3D scene data of roadside sensors using the initial roadside point cloud data; and calculating roadside 3D scene data of roadside sensors based on the initial roadside 3D scene data.

[0036] As one possible approach, in this embodiment of the application, for roadside sensors, if both radar (such as lidar) and cameras are present on the roadside, the lidar point clouds can be stitched together to automatically calibrate multi-sensor parameters. If only cameras are present on the roadside without lidar, a data acquisition vehicle can be used to collect lidar point cloud data in a specific scenario, and then the vehicle-side point cloud can be stitched together to reconstruct a 3D scene, thereby automatically calibrating sensor parameters. Simultaneously, during the generation of roadside 3D scene data, this embodiment of the application can select a specific vehicle, model the vehicle-side virtual sensors, and then generate roadside 3D scene data.

[0037] In some embodiments of this application, the process for generating roadside 3D scene data is as follows: Figure 2 As shown, the steps can be as follows:

[0038] Step S201: Obtain initial point cloud data of the roadside.

[0039] In this application embodiment, roadside radar data, such as lidar point cloud data, can be used as the initial roadside point cloud data. If lidar is not available, a data acquisition vehicle can be used to collect lidar point cloud data in a specific scenario, and then the point cloud data of the vehicle can be stitched together to reconstruct a three-dimensional scene, thereby automatically calibrating the sensor parameters and obtaining the initial roadside point cloud data. Other radar point cloud data can also be used. The specific settings can be made by those skilled in the art according to the actual situation, and this application does not impose any specific limitations.

[0040] Step S202: Generate initial 3D scene data for the roadside.

[0041] In this embodiment, the lidar point cloud data is first used as a basis. The lidar point cloud data is then registered with SfM (Structure from Motion) technology. A set of point clouds that can represent three-dimensional objects is obtained from the two-dimensional image data given in the roadside sensor data. The initial point cloud data is obtained with the position of each point cloud as the center. A three-dimensional Gaussian is constructed for each point in the initial point cloud data. Then, the three-dimensional Gaussian is projected onto the image plane with the help of camera extrinsic parameters, thereby generating the initial three-dimensional scene data of the roadside.

[0042] Step S203: Calculate the roadside 3D scene data.

[0043] In this embodiment, the initial three-dimensional scene data of the roadside can be rendered in a differential rasterization manner to obtain the three-dimensional scene data of the roadside.

[0044] Optionally, in one embodiment of this application, calculating the roadside three-dimensional scene data of the roadside sensor based on the initial roadside three-dimensional scene data includes: obtaining the true three-dimensional scene data of the roadside sensor based on the initial roadside three-dimensional scene data; obtaining the loss function of the roadside sensor based on the true three-dimensional scene data; and calculating the roadside three-dimensional scene data through the loss function.

[0045] In some embodiments, the present application embodiments can determine the real three-dimensional scene data of the roadside sensor based on the initial three-dimensional scene data of the roadside, and then compare the rendered image with the real three-dimensional scene data to obtain a loss function. Backpropagation is then performed based on the loss function to update the parameters in the three-dimensional Gaussian. The number of updated point clouds is controlled by adaptive density to achieve high-quality three-dimensional scene reconstruction and obtain roadside three-dimensional scene data.

[0046] Optionally, in one embodiment of this application, before generating roadside 3D scene data of the roadside sensor based on the roadside sensor data of the roadside sensor, the method further includes: acquiring initial roadside sensor data of the roadside sensor; acquiring initial vehicle-end sensor data of the vehicle-end sensor; and performing data preprocessing on the initial roadside sensor data and the initial vehicle-end sensor data to obtain roadside sensor data and vehicle-end sensor data that meet preset matching conditions.

[0047] As one possible implementation, embodiments of this application can perform data preprocessing on the initial roadside sensor data acquired by the roadside sensors and the initial vehicle-side sensor data acquired by the vehicle-side sensors to obtain roadside sensor data and vehicle-side sensor data that meet certain matching conditions. These matching conditions can be time synchronization conditions and / or location matching conditions, and can be specifically set by those skilled in the art according to actual circumstances; this application does not impose specific limitations.

[0048] For example, embodiments of this application can perform time synchronization and position matching on the initial roadside sensor data acquired by the roadside sensor and the initial vehicle-side sensor data acquired by the vehicle-side sensor, thereby obtaining the relative pose between the roadside sensor data and the vehicle-side sensor data.

[0049] In step S102, based on the roadside 3D scene data and the vehicle coordinate system of the vehicle-end sensors, the roadside 3D scene data in the vehicle coordinate system is generated as roadside 3D scene data of the roadside scene data.

[0050] As one possible implementation method, embodiments of this application can set virtual surround-view cameras on each vehicle with reference to Nuscenes parameters in the roadside 3D scene data, and transform the roadside 3D scene data in the vehicle camera coordinate system through the transformation of the global coordinate system, the vehicle coordinate system, and the camera coordinate system to obtain the vehicle roadside 3D scene data.

[0051] In step S103, the three-dimensional scene data of the roadside of the vehicle is rendered to obtain the simulated data of the roadside sensor simulating the vehicle-side sensor.

[0052] In actual implementation, the embodiments of this application can render the three-dimensional scene data of the roadside vehicle to obtain the simulated data of the roadside sensor simulating the vehicle-side sensor.

[0053] For example, in the embodiments of this application, NeRF (Neural Radiance Field) or 3DGS (3D Graphics Shader) methods can be used to render and generate three-dimensional scene data of the vehicle roadside. If the generated data is relatively sparse, a spread model is further adopted to ensure the generation of high-quality data, thereby obtaining simulation data.

[0054] Optionally, in one embodiment of this application, rendering the three-dimensional scene data of the roadside vehicle to obtain simulated data of the roadside sensor simulating the vehicle-end sensor includes: determining whether the simulated data and the vehicle-end sensor data meet preset evaluation conditions; if the simulated data does not meet the preset evaluation conditions, then re-acquiring the simulated data based on the roadside sensor and the vehicle-end sensor until the simulated data meets the preset evaluation conditions.

[0055] In some embodiments of this application, during the process of simulating vehicle-side sensor data using roadside sensors, it can determine whether the generated simulated data and the vehicle-side sensor data meet certain evaluation conditions, such as whether they meet indicators like PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity Index Measure). If the simulated data does not meet certain evaluation conditions, it can re-acquire the simulated data until the simulated data meets certain evaluation conditions.

[0056] Certain evaluation criteria may be set by those skilled in the art according to the actual situation, and this application does not impose specific restrictions.

[0057] The working principle of the method for simulating vehicle-side sensor data based on roadside sensors proposed in this application will be described in detail below with reference to a specific embodiment.

[0058] in, Figure 3 This is a flowchart illustrating the working principle of a method for simulating vehicle-side sensor data based on roadside sensors according to an embodiment of this application.

[0059] Step S301: Roadside sensor.

[0060] In this embodiment, roadside sensor data can be collected in real time using roadside sensors.

[0061] Step S302: Determine whether it is a lidar.

[0062] In this embodiment of the application, if there are radars, such as lidar and cameras, on the roadside, step S304 can be executed; otherwise, step S303 is executed.

[0063] Step S303: The data acquisition vehicle collects point cloud data from a specific scene using lidar.

[0064] Step S304: Roadside lidar point cloud stitching.

[0065] In this embodiment of the application, the point cloud of the LiDAR on the roadside is stitched together to execute step S306.

[0066] Step S305: Vehicle-side LiDAR point cloud stitching.

[0067] In this embodiment of the application, when there is only a camera on the roadside and no LiDAR, a data acquisition vehicle can be used to collect LiDAR point cloud data in a specific scenario, and then the point cloud data of the vehicle can be stitched together to reconstruct a three-dimensional scene, thereby executing step S306.

[0068] Step S306: Automatic calibration of roadside multi-sensor parameters.

[0069] In this embodiment, roadside sensor data can be used to calibrate multi-sensor parameters, thereby generating roadside three-dimensional scene data.

[0070] Step S307: Model the vehicle-side simulated sensor.

[0071] In this embodiment of the application, the roadside 3D scene data can be transformed in the vehicle body camera coordinate system through the transformation between the global coordinate system, the vehicle body coordinate system, and the camera coordinate system to obtain the vehicle body roadside 3D scene data.

[0072] Step S308: Simulate data.

[0073] In this embodiment, NeRF or 3DGS methods can be used to render and generate three-dimensional scene data of the vehicle roadside. If the generated data is relatively sparse, a spread model is further adopted to ensure the generation of high-quality data, thereby obtaining simulation data.

[0074] Furthermore, in this embodiment of the application, the simulated data obtained by rendering the three-dimensional scene data of the vehicle body roadside can be compared and evaluated with the vehicle-side sensor data to obtain simulated data that meets certain evaluation conditions.

[0075] The method for simulating vehicle-side sensor data based on roadside sensors, as proposed in this application, can obtain roadside 3D scene data based on roadside sensor data. Then, it combines the vehicle coordinate system of the vehicle-side sensors to generate vehicle-side 3D scene data in the vehicle coordinate system. This vehicle-side 3D scene data is then rendered to obtain simulated data from the roadside sensors simulating the vehicle-side sensors. Through 3D scene reconstruction and coordinate transformation, simulated data from the vehicle-side sensors can be effectively generated using roadside sensor data, providing richer and more reliable information for the perception module of the autonomous driving system. This method has high practical value in real-world applications and fully utilizes the relatively inexpensive and readily available data from roadside sensors to generate abundant vehicle-side sensor data, reducing the cost required for training autonomous driving systems. Therefore, it solves the problems in related technologies, such as high difficulty and cost of data acquisition and processing, complex installation, and difficult maintenance, which make it difficult to meet the ever-increasing demand for vehicle-side sensor data.

[0076] Next, with reference to the accompanying drawings, a device for simulating vehicle-side sensor data based on roadside sensors according to embodiments of this application is described.

[0077] Figure 4 This is a block diagram of a device for simulating vehicle-side sensor data based on roadside sensors, according to an embodiment of this application.

[0078] like Figure 4 As shown, the device 10, which simulates vehicle-side sensor data based on roadside sensors, includes: a first generation module 100, a second generation module 200, and a first acquisition module 300.

[0079] The first generation module 100 is used to generate roadside three-dimensional scene data of the roadside sensor based on the roadside sensor data of the roadside sensor.

[0080] The second generation module 200 is used to generate roadside 3D scene data of the vehicle body in the vehicle body coordinate system based on the roadside 3D scene data and the vehicle body coordinate system of the vehicle end sensor.

[0081] The first acquisition module 300 is used to render the three-dimensional scene data of the roadside vehicle to obtain the simulated data of the roadside sensor simulating the vehicle-side sensor.

[0082] Optionally, in one embodiment of this application, it further includes: a second acquisition module, a third acquisition module, and a preprocessing module.

[0083] The second acquisition module is used to acquire the initial roadside sensor data of the roadside sensor before generating the roadside three-dimensional scene data of the roadside sensor based on the roadside sensor data of the roadside sensor.

[0084] The third acquisition module is used to acquire the initial vehicle-side sensor data from the vehicle-side sensors.

[0085] The preprocessing module is used to preprocess the initial roadside sensor data and the initial vehicle-side sensor data to obtain roadside sensor data and vehicle-side sensor data that meet the preset matching conditions.

[0086] Optionally, in one embodiment of this application, the first generation module 100 includes: a first generation unit, a second generation unit, and a calculation unit.

[0087] The first generation unit is used to obtain the initial point cloud data of the roadside sensors based on the roadside radar data and / or roadside camera data in the roadside sensor data.

[0088] The second generation unit is used to generate initial 3D scene data of the roadside sensor using the initial point cloud data of the roadside.

[0089] The calculation unit is used to calculate the roadside three-dimensional scene data of the roadside sensor based on the initial roadside three-dimensional scene data.

[0090] Optionally, in one embodiment of this application, the first acquisition module 300 includes: a first judgment unit and an acquisition unit.

[0091] The first judgment unit is used to determine whether the simulated data and the vehicle-side sensor data meet the preset evaluation conditions.

[0092] The acquisition unit is used to reacquire the simulated data based on roadside sensors and vehicle-end sensors when the simulated data does not meet the preset evaluation conditions, until the simulated data meets the preset evaluation conditions.

[0093] Optionally, in one embodiment of this application, the computing unit includes: a first generation subunit, a second generation subunit, and a computing subunit.

[0094] The first generation sub-unit is used to obtain the real three-dimensional scene data of the roadside sensor based on the initial three-dimensional scene data of the roadside.

[0095] The second generation subunit is used to obtain the loss function of the roadside sensor based on real 3D scene data.

[0096] The computational subunit is used to calculate roadside 3D scene data using a loss function.

[0097] It should be noted that the foregoing explanation of the method embodiment for simulating vehicle-side sensor data based on roadside sensors also applies to the apparatus for simulating vehicle-side sensor data based on roadside sensors in this embodiment, and will not be repeated here.

[0098] The apparatus for simulating vehicle-side sensor data based on roadside sensor data, as proposed in this application, can obtain roadside 3D scene data based on roadside sensor data. Then, it combines this roadside 3D scene data with the vehicle coordinate system of the vehicle-side sensor to generate vehicle-side 3D scene data in the vehicle coordinate system. This vehicle-side 3D scene data is then rendered to obtain simulated data from the roadside sensor simulating the vehicle-side sensor data. Through 3D scene reconstruction and coordinate transformation, it can effectively utilize roadside sensor data to generate simulated data from the vehicle-side sensor, providing richer and more reliable information for the perception module of the autonomous driving system. This apparatus has high practical value in real-world applications and fully utilizes relatively inexpensive and readily available roadside sensor data to generate abundant vehicle-side sensor data, reducing the cost required for training autonomous driving systems. Therefore, it solves the problems in related technologies, such as high difficulty and cost of data acquisition and processing, complex installation, and difficult maintenance, making it difficult to meet the ever-increasing demand for vehicle-side sensor data.

[0099] Figure 5 This is a schematic diagram of the structure of a vehicle according to an embodiment of this application. The vehicle may include:

[0100] The memory 501, the processor 502, and the computer program stored on the memory 501 and capable of running on the processor 502.

[0101] When the processor 502 executes the program, it implements the method for simulating vehicle-side sensor data based on roadside sensors provided in the above embodiments.

[0102] Furthermore, the vehicle also includes:

[0103] Communication interface 503 is used for communication between memory 501 and processor 502.

[0104] The memory 501 is used to store computer programs that can run on the processor 502.

[0105] The memory 501 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0106] If the memory 501, processor 502, and communication interface 503 are implemented independently, then the communication interface 503, memory 501, and processor 502 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0107] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.

[0108] Processor 502 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0109] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described method for simulating vehicle-side sensor data based on roadside sensors.

[0110] This application also provides a computer program product, including a computer program that, when executed, implements the above-described method for simulating vehicle-side sensor data based on roadside sensors.

[0111] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0112] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0113] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0114] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0115] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or more of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0116] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0117] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0118] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A method for simulating vehicle-side sensor data based on roadside sensors, characterized in that, Includes the following steps: The roadside 3D scene data of the roadside sensor is generated based on the roadside sensor data; Based on the roadside 3D scene data and the vehicle coordinate system of the vehicle-end sensors, generate the roadside 3D scene data of the vehicle in the vehicle coordinate system; The roadside 3D scene data of the vehicle body is rendered to obtain the simulated data of the roadside sensors simulating the vehicle-side sensors; The generation of roadside 3D scene data based on roadside sensor data includes: The initial point cloud data of the roadside sensors is obtained based on the roadside radar data and / or roadside camera data in the roadside sensor data. The roadside initial 3D scene data of the roadside sensor is generated using the roadside initial point cloud data; The roadside three-dimensional scene data of the roadside sensor is calculated based on the initial roadside three-dimensional scene data. The process of rendering the roadside 3D scene data of the vehicle body to obtain the simulated data of the roadside sensors simulating the vehicle-side sensors includes: Determine whether the simulated data and the vehicle-side sensor data meet preset evaluation conditions; If the simulated data does not meet the preset evaluation conditions, the simulated data is reacquired based on the roadside sensor and the vehicle-end sensor until the simulated data meets the preset evaluation conditions. The step of calculating the roadside 3D scene data of the roadside sensor based on the initial roadside 3D scene data includes: The actual three-dimensional scene data of the roadside sensor is obtained based on the initial three-dimensional scene data of the roadside; The loss function of the roadside sensor is obtained based on the real 3D scene data; The roadside 3D scene data is calculated using the loss function. The step of calculating the roadside 3D scene data using the loss function includes: performing backpropagation based on the loss function to update the parameters in the 3D Gaussian cloud; and updating the number of point clouds through adaptive density control to achieve high-quality 3D scene reconstruction and obtain roadside 3D scene data.

2. The method according to claim 1, characterized in that, Before generating the roadside 3D scene data of the roadside sensor based on the roadside sensor data, the method further includes: Acquire the initial roadside sensor data of the roadside sensor; Acquire the initial vehicle-side sensor data of the vehicle-side sensor; The initial roadside sensor data and the initial vehicle-side sensor data are preprocessed to obtain roadside sensor data and vehicle-side sensor data that meet preset matching conditions.

3. A device for simulating vehicle-side sensor data based on roadside sensors, characterized in that, include: The first generation module is used to generate roadside three-dimensional scene data of the roadside sensor based on the roadside sensor data of the roadside sensor; The second generation module is used to generate the roadside three-dimensional scene data in the vehicle coordinate system based on the roadside three-dimensional scene data and the vehicle coordinate system of the vehicle-end sensor. The first acquisition module is used to render the three-dimensional scene data of the roadside of the vehicle body to obtain the simulated data of the roadside sensor simulating the vehicle-end sensor; The generation of roadside 3D scene data based on roadside sensor data includes: The initial point cloud data of the roadside sensors is obtained based on the roadside radar data and / or roadside camera data in the roadside sensor data. The roadside initial 3D scene data of the roadside sensor is generated using the roadside initial point cloud data; The roadside three-dimensional scene data of the roadside sensor is calculated based on the initial roadside three-dimensional scene data. The process of rendering the roadside 3D scene data of the vehicle body to obtain the simulated data of the roadside sensors simulating the vehicle-side sensors includes: Determine whether the simulated data and the vehicle-side sensor data meet preset evaluation conditions; If the simulated data does not meet the preset evaluation conditions, the simulated data is reacquired based on the roadside sensor and the vehicle-end sensor until the simulated data meets the preset evaluation conditions. The step of calculating the roadside 3D scene data of the roadside sensor based on the initial roadside 3D scene data includes: The actual three-dimensional scene data of the roadside sensor is obtained based on the initial three-dimensional scene data of the roadside; The loss function of the roadside sensor is obtained based on the real 3D scene data; The roadside 3D scene data is calculated using the loss function. The step of calculating the roadside 3D scene data using the loss function includes: performing backpropagation based on the loss function to update the parameters in the 3D Gaussian cloud; and updating the number of point clouds through adaptive density control to achieve high-quality 3D scene reconstruction and obtain roadside 3D scene data.

4. The apparatus according to claim 3, characterized in that, Also includes: The second acquisition module is used to acquire the initial roadside sensor data of the roadside sensor before generating the roadside three-dimensional scene data of the roadside sensor based on the roadside sensor data of the roadside sensor. The third acquisition module is used to acquire the initial vehicle-side sensor data of the vehicle-side sensor; The preprocessing module is used to preprocess the initial roadside sensor data and the initial vehicle-side sensor data to obtain roadside sensor data and vehicle-side sensor data that meet preset matching conditions.

5. A vehicle, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the method for simulating vehicle-side sensor data based on roadside sensors as described in any one of claims 1-2.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the method for simulating vehicle-side sensor data based on roadside sensors as described in any one of claims 1-2.

7. A computer program product, characterized in that, Includes a computer program, which, when executed, is used to implement the method for simulating vehicle-side sensor data based on roadside sensors as described in any one of claims 1-2.

Images