Method and device for simulating vehicle end sensor data based on roadside sensor

Abstract

The invention relates to the technical field of vehicle control, in particular to a method and device for simulating vehicle end sensor data based on a roadside sensor, and the method comprises the steps: generating roadside three-dimensional scene data of the roadside sensor based on the roadside sensor data of the roadside sensor; generating vehicle body roadside three-dimensional scene data of the roadside three-dimensional scene data in a vehicle body coordinate system based on the roadside three-dimensional scene data and the vehicle body coordinate system of the vehicle end sensor; and rendering the vehicle body roadside three-dimensional scene data to obtain simulation data of the roadside sensor simulating the vehicle end sensor. Therefore, the problems of high data acquisition and processing difficulty, high cost, complex installation, difficult maintenance, difficulty in meeting the increasing data requirements of the vehicle-end sensor and the like in the prior art are solved.

Description

Technical Field

[0001] This application relates to the technical field of vehicle control, and particularly relates to a method and device for simulating vehicle-end sensor data based on roadside sensors. Background Art

[0002] Vehicle-end sensor data is a key information source for an autonomous driving system to perceive the surrounding environment, make decisions, and execute actions. The vehicle-end sensor data interacts with other modules of the autonomous driving system to jointly achieve the autonomous driving function. Currently, vehicle-end sensor data is mainly collected by sensors equipped on the vehicle itself, such as cameras, lidar, ultrasonic sensors, etc.

[0003] In related technologies, a roadside data provision request can be sent from an in-vehicle unit to a cloud backend. Then, when a roadside unit detects that a vehicle enters the signal detection area it covers, the roadside data of the roadside unit is sent down to the in-vehicle unit. Alternatively, according to a vehicle-road collaborative system, various information transmitted by in-vehicle terminals and end-side devices within the communication range received by a roadside base station can be comprehensively processed by a cloud control platform and the roadside base station to obtain the final decision result of the in-vehicle terminal.

[0004] However, in related technologies, data acquisition and processing are difficult, costly, complex to install, and difficult to maintain, making it difficult to meet the growing demand for vehicle-end sensor data and urgently requiring improvement. Summary of the Invention

[0005] This application provides a method and device for simulating vehicle-end sensor data based on roadside sensors to solve the problems in related technologies, such as difficult data acquisition and processing, high cost, complex installation, difficult maintenance, and difficulty in meeting the growing demand for vehicle-end sensor data.

[0006] The first aspect of the embodiments of this application provides a method for simulating vehicle-end sensor data based on roadside sensors, including the following steps: generating roadside three-dimensional scene data of the roadside sensor based on the roadside sensor data of the roadside sensor; generating the roadside three-dimensional scene data in the vehicle body coordinate system of the vehicle-end sensor based on the roadside three-dimensional scene data and the vehicle body coordinate system of the vehicle-end sensor; rendering the roadside three-dimensional scene data in the vehicle body coordinate system to obtain simulation data of the roadside sensor simulating the vehicle-end sensor.

[0007] Optionally, in an embodiment of the present application, before generating the roadside three-dimensional scene data of the roadside sensor based on the roadside sensor data, the method further includes: obtaining the initial roadside sensor data of the roadside sensor; obtaining the initial vehicle-end sensor data of the vehicle-end sensor; and performing data preprocessing on the initial roadside sensor data and the initial vehicle-end sensor data to obtain the roadside sensor data and the vehicle-end sensor data that meet the preset matching conditions.

[0008] Optionally, in an embodiment of the present application, the generating the roadside three-dimensional scene data of the roadside sensor based on the roadside sensor data includes: obtaining the initial roadside point cloud data of the roadside sensor based on the roadside radar data and / or the roadside camera data in the roadside sensor data; generating the initial roadside three-dimensional scene data of the roadside sensor by using the initial roadside point cloud data; and calculating the roadside three-dimensional scene data of the roadside sensor according to the initial roadside three-dimensional scene data.

[0009] Optionally, in an embodiment of the present application, the rendering the roadside three-dimensional scene data of the vehicle body to obtain the simulation data of the roadside sensor simulating the vehicle-end sensor includes: determining whether the simulation data and the vehicle-end sensor data meet the preset evaluation conditions; and if the simulation data does not meet the preset evaluation conditions, re-obtaining the simulation data based on the roadside sensor and the vehicle-end sensor until the simulation data meets the preset evaluation conditions.

[0010] Optionally, in an embodiment of the present application, the calculating the roadside three-dimensional scene data of the roadside sensor according to the initial roadside three-dimensional scene data includes: obtaining the real three-dimensional scene data of the roadside sensor according to the initial roadside three-dimensional scene data; obtaining the loss function of the roadside sensor based on the real three-dimensional scene data; and calculating the roadside three-dimensional scene data through the loss function.

[0011] An embodiment of the second aspect of the present application provides a device for simulating vehicle-end sensor data based on a roadside sensor, including: a first generating module, configured to generate the roadside three-dimensional scene data of the roadside sensor based on the roadside sensor data of the roadside sensor; a second generating module, configured to generate the roadside three-dimensional scene data of the vehicle body in the vehicle body coordinate system based on the roadside three-dimensional scene data and the vehicle body coordinate system of the vehicle-end sensor; and a first obtaining module, configured to render the roadside three-dimensional scene data of the vehicle body to obtain the simulation data of the roadside sensor simulating the vehicle-end sensor.

[0012] Optionally, in an embodiment of the present application, it further includes: a second acquisition module, configured to acquire initial roadside sensor data of the roadside sensor before generating the roadside three-dimensional scene data of the roadside sensor based on the roadside sensor data of the roadside sensor; a third acquisition module, configured to acquire initial vehicle-end sensor data of the vehicle-end sensor; and a preprocessing module, configured to perform data preprocessing on the initial roadside sensor data and the initial vehicle-end sensor data to obtain roadside sensor data and vehicle-end sensor data that meet a preset matching condition.

[0013] Optionally, in an embodiment of the present application, the first generation module includes: a first generation unit, configured to obtain initial roadside point cloud data of the roadside sensor based on roadside radar data and / or roadside camera data in the roadside sensor data; a second generation unit, configured to generate initial roadside three-dimensional scene data of the roadside sensor by using the initial roadside point cloud data; and a calculation unit, configured to calculate the roadside three-dimensional scene data of the roadside sensor according to the initial roadside three-dimensional scene data.

[0014] Optionally, in an embodiment of the present application, the first acquisition module includes: a first determination unit, configured to determine whether the simulation data and the vehicle-end sensor data meet a preset evaluation condition; and an acquisition unit, configured to, when the simulation data does not meet the preset evaluation condition, re-acquire the simulation data based on the roadside sensor and the vehicle-end sensor until the simulation data meets the preset evaluation condition.

[0015] Optionally, in an embodiment of the present application, the calculation unit includes: a first generation sub-unit, configured to obtain real three-dimensional scene data of the roadside sensor according to the initial roadside three-dimensional scene data; a second generation sub-unit, configured to obtain a loss function of the roadside sensor based on the real three-dimensional scene data; and a calculation sub-unit, configured to calculate the roadside three-dimensional scene data through the loss function.

[0016] An embodiment of the third aspect of the present application provides a vehicle, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor executes the program to implement the method for simulating vehicle-end sensor data based on a roadside sensor as described in the above embodiment.

[0017] An embodiment of the fourth aspect of the present application provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the program is executed by a processor, the method for simulating vehicle-end sensor data based on a roadside sensor as described above is implemented.

[0018] The fifth aspect of the present application provides a computer program product, including a computer program which, when executed, implements the method for simulating in-vehicle sensor data based on roadside sensors as described above.

[0019] Embodiments of the present application can obtain roadside three-dimensional scene data based on roadside sensor data of roadside sensors, and then generate the in-vehicle roadside three-dimensional scene data of the roadside three-dimensional scene data in the vehicle body coordinate system in combination with the vehicle body coordinate system of in-vehicle sensors, and render the in-vehicle roadside three-dimensional scene data, and then obtain the simulated data of the roadside sensors simulating in-vehicle sensors. Through three-dimensional scene reconstruction and coordinate transformation, it is possible to effectively use the roadside sensor data to generate the simulated data of in-vehicle sensors, providing richer and more reliable information for the perception module of the autonomous driving system. It has high practical value in practical applications, and makes full use of the relatively cheap and easily accessible data of roadside sensors to generate rich in-vehicle sensor data, reducing the cost required for the training of the autonomous driving system. Thus, the problems in the related art, such as the great difficulty in data acquisition and processing, high cost, complex installation, difficult maintenance, and the difficulty in meeting the growing demand for in-vehicle sensor data, are solved.

[0020] Additional aspects and advantages of the present application will be given in part in the following description, become apparent in part from the following description, or be understood through the practice of the present application. Description of the Drawings

[0021] The above and / or additional aspects and advantages of the present application will become apparent and easy to understand from the following description of the embodiments in conjunction with the drawings, where:

[0022] Figure 1 is a flowchart of a method for simulating in-vehicle sensor data based on roadside sensors according to an embodiment of the present application;

[0023] Figure 2 is a flowchart of generating roadside three-dimensional scene data according to an embodiment of the present application;

[0024] Figure 3 is a flowchart of the working principle of a method for simulating in-vehicle sensor data based on roadside sensors according to an embodiment of the present application;

[0025] Figure 4 is a block diagram of a device for simulating in-vehicle sensor data based on roadside sensors according to an embodiment of the present application;

[0026] Figure 5 is a schematic structural diagram of a vehicle according to an embodiment of the present application. Detailed Embodiments

[0027] Embodiments of the present application will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present application, and should not be construed as limiting the present application.

[0028] The method and device for simulating vehicle-end sensor data based on roadside sensors in the embodiments of the present application will be described below with reference to the accompanying drawings. In view of the problems of high difficulty in data acquisition and processing, high cost, complex installation, and difficult maintenance in the above-mentioned background technology, which are difficult to meet the growing demand for vehicle-end sensor data, the present application provides a method for simulating vehicle-end sensor data based on roadside sensors. In this method, roadside three-dimensional scene data can be obtained based on the roadside sensor data of the roadside sensors, and then combined with the vehicle body coordinate system of the vehicle-end sensors to generate the vehicle body roadside three-dimensional scene data of the roadside three-dimensional scene data in the vehicle body coordinate system, and the vehicle body roadside three-dimensional scene data is rendered, and then the simulation data of the roadside sensors simulating the vehicle-end sensors is obtained. Through three-dimensional scene reconstruction and coordinate transformation, the roadside sensor data can be effectively used to generate the simulation data of the vehicle-end sensors, providing richer and more reliable information for the perception module of the autonomous driving system, having high practical value in practical applications, and making full use of the relatively cheap and easily obtained data of the roadside sensors to generate rich vehicle-end sensor data, reducing the cost required for the training of the autonomous driving system. Thus, the problems in the related technology, such as high difficulty in data acquisition and processing, high cost, complex installation, difficult maintenance, and difficulty in meeting the growing demand for vehicle-end sensor data, are solved.

[0029] Specifically, Figure 1 FIG. is a flowchart of a method for simulating vehicle-end sensor data based on roadside sensors according to an embodiment of the present application.

[0030] As Figure 1 shown, the method for simulating vehicle-end sensor data based on roadside sensors includes the following steps:

[0031] In step S101, roadside three-dimensional scene data of the roadside sensors is generated based on the roadside sensor data of the roadside sensors.

[0032] It can be understood that the roadside sensors in the embodiments of the present application can be installed on both sides of the road, on bridges, or on the frameworks of signal lights, and they can include, but are not limited to, roadside radar sensors, roadside lidar sensors, and roadside camera sensors. Specifically, those skilled in the art can set them according to the actual situation, and the present application does not make specific limitations.

[0033] Furthermore, in the embodiments of the present application, the roadside three-dimensional scene data may include, but is not limited to, roadside radar data, roadside radar point cloud data, roadside lidar data, roadside lidar point cloud data, roadside two-dimensional image data, etc. The present application does not make specific limitations.

[0034] As a possible implementation manner, the embodiments of the present application may generate roadside three-dimensional scene data according to the roadside sensor data collected by roadside sensors in real time.

[0035] Optionally, in an embodiment of the present application, generating the roadside three-dimensional scene data of a roadside sensor based on the roadside sensor data of the roadside sensor includes: obtaining the initial roadside point cloud data of the roadside sensor based on the roadside radar data and / or roadside camera data in the roadside sensor data; generating the initial roadside three-dimensional scene data of the roadside sensor by using the initial roadside point cloud data; calculating the roadside three-dimensional scene data of the roadside sensor according to the initial roadside three-dimensional scene data.

[0036] As a possible implementation manner, for the roadside sensor in the embodiments of the present application, if there are radars on the roadside at the same time, such as lidar and cameras, the lidar point clouds on the roadside can be stitched to automatically calibrate the multi-sensor parameters; if there is only a camera on the roadside without a lidar, a data acquisition vehicle can be used to collect lidar point cloud data in a specific scenario, and then the vehicle-end point clouds can be stitched to reconstruct the three-dimensional scene, so as to automatically calibrate the sensor parameters. At the same time, during the generation process of the roadside three-dimensional scene data, the embodiments of the present application can select a specified vehicle to model the vehicle-end virtual sensor, and then generate the roadside three-dimensional scene data.

[0037] In some embodiments, the process of generating the roadside three-dimensional scene data in the embodiments of the present application is as Figure 2 shown, and its steps may be:

[0038] Step S201: Obtain the initial roadside point cloud data.

[0039] Among them, in the embodiments of the present application, the roadside radar data, such as lidar point cloud data, may be used as the initial roadside point cloud data. If there is no lidar, a data acquisition vehicle can be used to collect lidar point cloud data in a specific scenario, and then the vehicle-end point clouds can be stitched to reconstruct the three-dimensional scene, so as to automatically calibrate the sensor parameters to obtain the initial roadside point cloud data. Other radar point cloud data may also be used, which can be specifically set by those skilled in the art according to the actual situation. The present application does not make specific limitations.

[0040] Step S202: Generate the initial roadside three-dimensional scene data.

[0041] Among them, in the embodiment of the present application, first based on the lidar point cloud data, registration is performed by combining the SfM (Structure from Motion) technology with the lidar point cloud data. A set of point clouds that can represent three-dimensional objects is obtained from the two-dimensional image data given in the roadside sensor data. With the position of each point cloud as the center, initial point cloud data is obtained. A three-dimensional Gaussian is constructed for each point in the initial point cloud data, and then the three-dimensional Gaussian is projected onto the image plane with the help of the external camera parameters, thereby generating the initial roadside three-dimensional scene data.

[0042] Step S203: Calculate the roadside three-dimensional scene data.

[0043] Among them, in the embodiment of the present application, differentiable rasterization rendering can be performed on the initial roadside three-dimensional scene data, thereby obtaining the roadside three-dimensional scene data.

[0044] Optionally, in an embodiment of the present application, calculating the roadside three-dimensional scene data of the roadside sensor according to the initial roadside three-dimensional scene data includes: obtaining the real three-dimensional scene data of the roadside sensor according to the initial roadside three-dimensional scene data; obtaining the loss function of the roadside sensor based on the real three-dimensional scene data; calculating the roadside three-dimensional scene data through the loss function.

[0045] In some embodiments, the embodiment of the present application can determine the real three-dimensional scene data of the roadside sensor according to the initial roadside three-dimensional scene data, and then compare the rendered image with the real three-dimensional scene data to obtain the loss function, and perform backpropagation based on the loss function to update the parameters in the three-dimensional Gaussian, and update the number of point clouds through adaptive density control to achieve high-quality three-dimensional scene reconstruction and obtain the roadside three-dimensional scene data.

[0046] Optionally, in an embodiment of the present application, before generating the roadside three-dimensional scene data of the roadside sensor based on the roadside sensor data of the roadside sensor, it further includes: obtaining the initial roadside sensor data of the roadside sensor; obtaining the initial vehicle-end sensor data of the vehicle-end sensor; performing data preprocessing on the initial roadside sensor data and the initial vehicle-end sensor data to obtain roadside sensor data and vehicle-end sensor data that meet the preset matching conditions.

[0047] As a possible implementation manner, the embodiment of the present application can perform data preprocessing on the initial roadside sensor data obtained by the roadside sensor and the initial vehicle-end sensor data obtained by the vehicle-end sensor, thereby obtaining roadside sensor data and vehicle-end sensor data that meet certain matching conditions. Among them, the certain matching conditions can be time synchronization conditions and / or position matching conditions, which can be specifically set by those skilled in the art according to the actual situation, and the present application does not make specific limitations.

[0048] Exemplarily, embodiments of the present application can perform time synchronization and position matching on the initial roadside sensor data obtained by roadside sensors and the initial vehicle-end sensor data obtained by vehicle-end sensors, and then obtain the relative pose between the roadside sensor data and the vehicle-end sensor data.

[0049] In step S102, based on the roadside three-dimensional scene data and the vehicle body coordinate system of the vehicle-end sensor, the roadside three-dimensional scene data in the vehicle body coordinate system is generated as the vehicle body roadside three-dimensional scene data.

[0050] As a possible implementation manner, embodiments of the present application can set virtual surround cameras on each vehicle in the roadside three-dimensional scene data. Through the conversion of the global coordinate system - vehicle body coordinate system - camera coordinate system, the roadside three-dimensional scene data is converted in the vehicle body camera coordinate system to obtain the vehicle body roadside three-dimensional scene data.

[0051] In step S103, the vehicle body roadside three-dimensional scene data is rendered to obtain the simulated data of the roadside sensor simulating the vehicle-end sensor.

[0052] In the actual execution process, embodiments of the present application can render the vehicle body roadside three-dimensional scene data to obtain the simulated data of the roadside sensor simulating the vehicle-end sensor.

[0053] Exemplarily, embodiments of the present application can first use methods such as NeRF (Neural Radiance Field) or 3DGS (3D Graphics Shader) to render the vehicle body roadside three-dimensional scene data. If the generated data is relatively sparse, a diffusion model is further used to ensure the generation of high-quality data, and then the simulated data is obtained.

[0054] Optionally, in an embodiment of the present application, rendering the vehicle body roadside three-dimensional scene data to obtain the simulated data of the roadside sensor simulating the vehicle-end sensor includes: determining whether the simulated data and the vehicle-end sensor data meet a preset evaluation condition; if the simulated data does not meet the preset evaluation condition, re-obtain the simulated data based on the roadside sensor and the vehicle-end sensor until the simulated data meets the preset evaluation condition.

[0055] In some embodiments, during the process of simulating vehicle - end sensor data through roadside sensors in the embodiments of the present application, it can be determined whether the generated simulated data meets certain evaluation conditions with the vehicle - end sensor data, such as whether it meets indicators such as PSNR (Peak Signal - to - Noise Ratio) and SSIM (Structural Similarity Index Measure). And when the certain evaluation conditions are not met, the simulated data is re - obtained until the simulated data meets the certain evaluation conditions.

[0056] Among them, the certain evaluation conditions can be set by those skilled in the art according to the actual situation, and the present application does not make specific limitations.

[0057] Next, a specific embodiment is used to introduce in detail the working principle of the method for simulating vehicle - end sensor data based on roadside sensors proposed in the embodiments of the present application.

[0058] Among them, Figure 3 is a flowchart of the working principle of the method for simulating vehicle - end sensor data based on roadside sensors according to an embodiment of the present application.

[0059] Step S301: Roadside sensor.

[0060] Among them, in the embodiments of the present application, roadside sensor data can be collected in real - time through roadside sensors.

[0061] Step S302: Determine whether it is a lidar.

[0062] Among them, in the embodiments of the present application, for roadside sensors, if there are radars on the roadside at the same time, such as in the case of lidars and cameras, step S304 can be executed; otherwise, step S303 is executed.

[0063] Step S303: The data acquisition vehicle collects lidar point cloud data of a specific scenario.

[0064] Step S304: Roadside lidar point cloud stitching.

[0065] Among them, in the embodiments of the present application, the lidar point clouds on the roadside are stitched, and then step S306 is executed.

[0066] Step S305: Vehicle - end lidar point cloud stitching.

[0067] Among them, in the embodiments of the present application, when there is only a camera on the roadside and no lidar, the data acquisition vehicle can be used to collect lidar point cloud data in a specific scenario, and then the vehicle - end point cloud is stitched to reconstruct a three - dimensional scene, and then step S306 is executed.

[0068] Step S306: Automatic calibration of roadside multi-sensor parameters.

[0069] Among them, in the embodiment of the present application, multi-sensor parameter calibration can be performed on roadside sensor data, and then roadside three-dimensional scene data can be generated.

[0070] Step S307: Modeling of vehicle-end analog sensors.

[0071] Among them, in the embodiment of the present application, through the conversion of the global coordinate system - vehicle body coordinate system - camera coordinate system, the roadside three-dimensional scene data can be converted in the vehicle body camera coordinate system to obtain the roadside three-dimensional scene data of the vehicle body.

[0072] Step S308: Simulated data.

[0073] Among them, in the embodiment of the present application, the NeRF or 3DGS method can be used to render the roadside three-dimensional scene data of the vehicle body first. If the generated data is relatively sparse, a diffusion model can be further adopted to ensure the generation of high-quality data, and then simulated data can be obtained.

[0074] Furthermore, in the embodiment of the present application, the simulated data obtained by rendering the roadside three-dimensional scene data of the vehicle body can be compared and evaluated with the vehicle-end sensor data, and then simulated data that meets certain evaluation conditions can be obtained.

[0075] According to the method for simulating vehicle-end sensor data based on roadside sensors proposed in the embodiment of the present application, roadside three-dimensional scene data can be obtained based on the roadside sensor data of the roadside sensors, and then combined with the vehicle body coordinate system of the vehicle-end sensors to generate the roadside three-dimensional scene data of the vehicle body in the vehicle body coordinate system, and the roadside three-dimensional scene data of the vehicle body can be rendered to obtain the simulated data of the roadside sensors simulating the vehicle-end sensors. Through three-dimensional scene reconstruction and coordinate transformation, the roadside sensor data can be effectively used to generate the simulated data of the vehicle-end sensors, providing richer and more reliable information for the perception module of the autonomous driving system. It has high practical value in actual applications, and makes full use of the relatively cheap and easily obtainable data of the roadside sensors to generate rich vehicle-end sensor data, reducing the cost required for the training of the autonomous driving system. Thus, it solves the problems in the related technology, such as the difficulty in data acquisition and processing, high cost, complex installation, difficult maintenance, and the difficulty in meeting the growing demand for vehicle-end sensor data.

[0076] Secondly, a device for simulating vehicle-end sensor data based on roadside sensors according to an embodiment of the present application will be described with reference to the accompanying drawings.

[0077] Figure 4 It is a block diagram of a device for simulating vehicle-end sensor data based on roadside sensors provided according to an embodiment of the present application.

[0078] As shown Figure 4 in FIG. 1, the device 10 for simulating vehicle-end sensor data based on roadside sensors includes: a first generation module 100, a second generation module 200, and a first acquisition module 300.

[0079] Among them, the first generation module 100 is configured to generate roadside three-dimensional scene data of the roadside sensors based on the roadside sensor data of the roadside sensors.

[0080] The second generation module 200 is configured to generate vehicle-body roadside three-dimensional scene data of the roadside three-dimensional scene data in the vehicle-body coordinate system based on the roadside three-dimensional scene data and the vehicle-body coordinate system of the vehicle-end sensors.

[0081] The first acquisition module 300 is configured to render the vehicle-body roadside three-dimensional scene data to obtain simulation data of the roadside sensors simulating the vehicle-end sensors.

[0082] Optionally, in an embodiment of the present application, it further includes: a second acquisition module, a third acquisition module, and a preprocessing module.

[0083] Among them, the second acquisition module is configured to obtain initial roadside sensor data of the roadside sensors before generating roadside three-dimensional scene data of the roadside sensors based on the roadside sensor data of the roadside sensors.

[0084] The third acquisition module is configured to obtain initial vehicle-end sensor data of the vehicle-end sensors.

[0085] The preprocessing module is configured to perform data preprocessing on the initial roadside sensor data and the initial vehicle-end sensor data to obtain roadside sensor data and vehicle-end sensor data that meet preset matching conditions.

[0086] Optionally, in an embodiment of the present application, the first generation module 100 includes: a first generation unit, a second generation unit, and a calculation unit.

[0087] Among them, the first generation unit is configured to obtain initial roadside point cloud data of the roadside sensors based on the roadside radar data and / or roadside camera data in the roadside sensor data.

[0088] The second generation unit is configured to generate initial roadside three-dimensional scene data of the roadside sensors by using the initial roadside point cloud data.

[0089] The calculation unit is configured to calculate the roadside three-dimensional scene data of the roadside sensors according to the initial roadside three-dimensional scene data.

[0090] Optionally, in an embodiment of the present application, the first acquisition module 300 includes: a first judgment unit and an acquisition unit.

[0091] Among them, the first judgment unit is used to judge whether the analog data and the vehicle-end sensor data meet the preset evaluation conditions.

[0092] The acquisition unit is used to, when the analog data does not meet the preset evaluation conditions, re-acquire the analog data based on the roadside sensor and the vehicle-end sensor until the analog data meets the preset evaluation conditions.

[0093] Optionally, in an embodiment of the present application, the calculation unit includes: a first generation subunit, a second generation subunit, and a calculation subunit.

[0094] Among them, the first generation subunit is used to obtain the real three-dimensional scene data of the roadside sensor according to the initial roadside three-dimensional scene data.

[0095] The second generation subunit is used to obtain the loss function of the roadside sensor based on the real three-dimensional scene data.

[0096] The calculation subunit is used to calculate the roadside three-dimensional scene data through the loss function.

[0097] It should be noted that the foregoing explanation of the method embodiment for simulating vehicle-end sensor data based on roadside sensors also applies to the device for simulating vehicle-end sensor data based on roadside sensors in this embodiment, and will not be elaborated here.

[0098] The device for simulating vehicle-end sensor data based on roadside sensors proposed according to the embodiments of the present application can obtain roadside three-dimensional scene data based on the roadside sensor data of the roadside sensor, and then combine the vehicle body coordinate system of the vehicle-end sensor to generate the vehicle body roadside three-dimensional scene data of the roadside three-dimensional scene data in the vehicle body coordinate system, and render the vehicle body roadside three-dimensional scene data, and then obtain the analog data of the roadside sensor simulating the vehicle-end sensor. Through three-dimensional scene reconstruction and coordinate transformation, it can effectively use the roadside sensor data to generate the analog data of the vehicle-end sensor, provide richer and more reliable information for the perception module of the autonomous driving system, and has high practical value in actual applications. Moreover, it makes full use of the relatively cheap and easily obtainable roadside sensor data to generate rich vehicle-end sensor data, reducing the cost required for the training of the autonomous driving system. Thus, it solves the problems in the related technologies, such as the great difficulty in data acquisition and processing, high cost, complex installation, difficult maintenance, and the difficulty in meeting the growing demand for vehicle-end sensor data.

[0099] Figure 5 It is a schematic structural diagram of a vehicle provided according to an embodiment of the present application. The vehicle may include:

[0100] A memory 501, a processor 502, and a computer program stored on the memory 501 and executable on the processor 502.

[0101] When the processor 502 executes the program, it implements the method for simulating vehicle - end sensor data based on roadside sensors provided in the above - mentioned embodiments.

[0102] Furthermore, the vehicle further includes:

[0103] A communication interface 503, which is used for communication between the memory 501 and the processor 502.

[0104] A memory 501, which is used to store computer programs that can run on the processor 502.

[0105] The memory 501 may include high - speed RAM memory, and may also include non - volatile memory, such as at least one disk memory.

[0106] If the memory 501, the processor 502, and the communication interface 503 are implemented independently, the communication interface 503, the memory 501, and the processor 502 can be interconnected through a bus and complete communication with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of simplicity of representation, Figure 5 only a thick line is used to represent it in the figure, but it does not mean that there is only one bus or one type of bus.

[0107] Optionally, in a specific implementation, if the memory 501, the processor 502, and the communication interface 503 are integrated on a chip, the memory 501, the processor 502, and the communication interface 503 can complete communication with each other through an internal interface.

[0108] The processor 502 may be a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0109] The embodiments of the present application also provide a computer - readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the above - mentioned method for simulating vehicle - end sensor data based on roadside sensors.

[0110] An embodiment of the present application further provides a computer program product, including a computer program, which when executed implements the method for simulating in-vehicle sensor data based on roadside sensors as described above.

[0111] In the description of this specification, the description with reference to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0112] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of these features. In the description of the present application, the meaning of "N" is at least two, such as two, three, etc., unless otherwise specifically defined.

[0113] Any process or method description shown in the flowchart or described in other ways herein can be understood as representing a module, segment, or part of code including one or N executable instructions for implementing a customized logic function or process, and the scope of the preferred embodiments of the present application includes additional implementations, where the functions may be executed in a substantially simultaneous manner or in a reverse order according to the involved functions, rather than in the order shown or discussed, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0114] The logic and / or steps represented in the flowchart or otherwise described herein can, for example, be considered a definitional sequence list of executable instructions for implementing logical functions, which can be embodied specifically in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or used in conjunction with these instruction execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection portion (electronic device) having one or N wirings, a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other media, followed by editing, interpretation, or otherwise processing as appropriate, and then stored in a computer memory.

[0115] It should be understood that the various parts of the present application can be implemented using hardware, software, firmware, or combinations thereof. In the above-described embodiments, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits having suitable combinational logic gate circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), and the like.

[0116] Those of ordinary skill in the art of this technology can understand that all or part of the steps carried by the method of implementing the above embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0117] In addition, each functional unit in various embodiments of the present application may be integrated into a processing module, may exist physically alone for each unit, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0118] The above-mentioned storage medium may be a read-only memory, a magnetic disk, an optical disc, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A method for simulating vehicle-side sensor data based on roadside sensors, characterized in that: The following steps are involved: Generate roadside three-dimensional scene data of the roadside sensor based on the roadside sensor data of the roadside sensor; Based on the roadside three-dimensional scene data and the vehicle body coordinate system of the vehicle-side sensor, generating vehicle body roadside three-dimensional scene data of the roadside three-dimensional scene data in the vehicle body coordinate system; The three-dimensional scene data of the vehicle body roadside is rendered to obtain simulation data of the roadside sensor simulating the vehicle-end sensor.

2. The method according to claim 1, characterized in that Before generating the roadside three-dimensional scene data of the roadside sensor based on the roadside sensor data of the roadside sensor, the method further includes: Acquiring initial roadside sensor data of the roadside sensor; Acquiring initial vehicle-end sensor data of the vehicle-end sensor; The initial roadside sensor data and the initial vehicle-side sensor data are preprocessed to obtain the roadside sensor data and the vehicle-side sensor data that meet preset matching conditions.

3. The method according to claim 1, characterized in that The generating of the roadside three-dimensional scene data of the roadside sensor based on the roadside sensor data includes: Obtaining roadside initial point cloud data of the roadside sensor based on roadside radar data and / or roadside camera data in the roadside sensor data; Generating the roadside initial three-dimensional scene data of the roadside sensor using the roadside initial point cloud data; The roadside three-dimensional scene data of the roadside sensor is calculated according to the roadside initial three-dimensional scene data.

4. The method according to claim 2, characterized in that: The rendering of the vehicle body roadside three-dimensional scene data to obtain simulation data of the roadside sensor simulating the vehicle end sensor includes: Determining whether the simulation data and the vehicle-side sensor data meet preset evaluation conditions; If the simulation data does not satisfy the preset evaluation condition, the simulation data is reacquired based on the roadside sensor and the vehicle-side sensor until the simulation data satisfies the preset evaluation condition.

5. The method according to claim 3, characterized in that: The calculating the roadside three-dimensional scene data of the roadside sensor according to the roadside initial three-dimensional scene data comprises: Obtaining the real three-dimensional scene data of the roadside sensor according to the roadside initial three-dimensional scene data; Obtaining a loss function of the roadside sensor based on the real three-dimensional scene data; The roadside three-dimensional scene data is calculated using the loss function.

6. A device for simulating vehicle-side sensor data based on roadside sensors, characterized in that: include: A first generating module, configured to generate roadside three-dimensional scene data of the roadside sensor based on roadside sensor data of the roadside sensor; A second generating module is used to generate vehicle body roadside three-dimensional scene data of the roadside three-dimensional scene data in the vehicle body coordinate system based on the roadside three-dimensional scene data and the vehicle body coordinate system of the vehicle-side sensor; The first acquisition module is used to render the three-dimensional scene data of the vehicle body roadside to obtain the simulation data of the roadside sensor simulating the vehicle end sensor.

7. The device according to claim 6, characterized in that Also includes: A second acquisition module, configured to acquire initial roadside sensor data of the roadside sensor before generating roadside three-dimensional scene data of the roadside sensor based on the roadside sensor data of the roadside sensor; A third acquisition module, used to acquire initial vehicle-end sensor data of the vehicle-end sensor; The preprocessing module is used to perform data preprocessing on the initial roadside sensor data and the initial vehicle-side sensor data to obtain roadside sensor data and vehicle-side sensor data that meet preset matching conditions.

8. A vehicle, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for simulating vehicle-side sensor data based on roadside sensors as described in any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the method for simulating vehicle-side sensor data based on a roadside sensor as described in any one of claims 1 to 5.

10. A computer program product, characterized in that It includes a computer program, which, when executed, is used to implement the method of simulating vehicle-side sensor data based on roadside sensors as described in any one of claims 1 to 5.

Images