3D Reconstruction Method, Device, Vehicle, and Storage Medium

Abstract

An embodiment of the present application discloses a three-dimensional reconstruction method, device, vehicle, and storage medium. The method includes: collecting road data of a target area through a vehicle sensor disposed on a target vehicle; performing three-dimensional reconstruction on the target area based on the road data to obtain a three-dimensional model of the target area; and constructing a virtual three-dimensional scene corresponding to the preset scene elements based on the three-dimensional model and the preset scene elements. Through the above method, by combining the obtained three-dimensional model of the target scene with the preset scene elements, a virtual three-dimensional scene that meets various usage scenarios can be provided, taking into account both the efficiency of simulation and the requirements of high-fidelity simulation.

Description

Technical Field

[0001] This application belongs to the technical field of autonomous driving virtual simulation, and particularly relates to a three-dimensional reconstruction method, device, vehicle, and storage medium. Background Art

[0002] In the field of autonomous driving virtual simulation, three-dimensional modeling of virtual simulation scenarios is an essential part. Currently, there are many ways to build three-dimensional models, and the differences in the authenticity of virtual simulation of the environment are also very large. In the in-loop simulation of running decision-making and planning algorithms, establishing a simple scenario model can meet the requirements of virtual in-loop simulation. However, in the in-loop simulation of vehicle perception algorithms, in order to make the operation of the simulation model consistent with the operation and perception characteristics of the actual vehicle model, a more realistic three-dimensional scenario model needs to be established to meet the requirements of virtual in-loop simulation. Summary of the Invention

[0003] In view of the above problems, this application proposes a three-dimensional reconstruction method, device, vehicle, and storage medium to improve the above problems.

[0004] In a first aspect, an embodiment of this application provides a three-dimensional reconstruction method, which includes: collecting road data of a target area through vehicle sensors disposed on a target vehicle; performing three-dimensional reconstruction on the target area based on the road data to obtain a three-dimensional model of the target area; and constructing a virtual three-dimensional scene corresponding to the preset scene elements based on the three-dimensional model and the preset scene elements.

[0005] In a second aspect, an embodiment of this application provides a three-dimensional reconstruction device, which includes: a data acquisition unit for collecting road data of a target area through vehicle sensors disposed on a target vehicle; a reconstruction unit for performing three-dimensional reconstruction on the target area based on the road data to obtain a three-dimensional model of the target area; and a construction unit for constructing a virtual three-dimensional scene corresponding to the preset scene elements based on the three-dimensional model and the preset scene elements.

[0006] In a third aspect, an embodiment of this application provides a vehicle, which includes one or more processors and a memory; one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the above method.

[0007] In a fourth aspect, an embodiment of this application provides a computer-readable storage medium, in which program code is stored, and when the program code runs, it executes the above method.

[0008] Embodiments of the present application provide a three-dimensional reconstruction method, apparatus, vehicle, and storage medium. First, road data of a target area is collected by vehicle sensors disposed on a target vehicle, then the target area is three-dimensionally reconstructed based on the road data to obtain a three-dimensional model of the target area, and finally, a virtual three-dimensional scene corresponding to preset scene elements is constructed based on the three-dimensional model and the preset scene elements. By the above method, the three-dimensional model of the target scene obtained and the preset scene elements are combined, so that a virtual three-dimensional scene satisfying various usage scenarios can be provided, taking into account both the efficiency of simulation and the requirements of high-fidelity simulation. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings based on these drawings without creative efforts.

[0010] Figure 1 The flowchart of a three-dimensional reconstruction method proposed in an embodiment of the present application is shown;

[0011] Figure 2 The flowchart of another three-dimensional reconstruction method proposed in an embodiment of the present application is shown;

[0012] Figure 3 The structural block diagram of a three-dimensional reconstruction apparatus proposed in an embodiment of the present application is shown;

[0013] Figure 4 The structural block diagram of a three-dimensional reconstruction apparatus proposed in an embodiment of the present application is shown;

[0014] Figure 5 The structural block diagram of a vehicle for executing the three-dimensional reconstruction method according to the embodiment of the present application in the embodiment of the present application is shown;

[0015] Figure 6 The storage unit for storing or carrying the program code for implementing the three-dimensional reconstruction method according to the embodiment of the present application in the embodiment of the present application is shown. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0016] The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the scope of protection of the present application.

[0017] In an embodiment of the present application, the inventor proposes a three-dimensional reconstruction method, device, vehicle, and storage medium. First, road data of a target area is collected by vehicle sensors disposed on a target vehicle, then the target area is three-dimensionally reconstructed based on the road data to obtain a three-dimensional model of the target area, and finally, a virtual three-dimensional scene corresponding to a preset scene element is constructed based on the three-dimensional model and the preset scene element. By the above method, the three-dimensional model of the obtained target scene is combined with the preset scene element, so that a virtual three-dimensional scene satisfying various usage scenarios can be provided, taking into account the efficiency of simulation and the requirements of high-fidelity simulation.

[0018] The following will specifically describe each embodiment of the present application in conjunction with the accompanying drawings.

[0019] Please refer to Figure 1 , a three-dimensional reconstruction method provided by an embodiment of the present application, the method includes:

[0020] Step S110: Collect road data of a target area by vehicle sensors disposed on a target vehicle.

[0021] In an embodiment of the present application, the target vehicle is a vehicle preset for virtual simulation. Among them, the target vehicle can be a vehicle of any model, and no specific limitation is made here. The vehicle sensor can be preset at a preset position of the target vehicle and is a sensor for collecting road data of the target area. Among them, the vehicle sensor can be an image acquisition device or a lidar, etc., and no specific limitation is made here.

[0022] The target area is an area preset for three-dimensional reconstruction, and the target area is the running area of the target vehicle. Among them, the target area can be limited to a small area, that is, within the running lane. For example, the target area can be set as a park and a closed road site. Limiting the target area to a small area can make there be no movement offset after reconstruction longitudinally, and the lateral movement offset is only limited to the inside of the lane.

[0023] The road data is data of the running lane, environmental data around the target vehicle, and attitude data of the target vehicle, etc. collected by the vehicle sensor disposed in the target vehicle when the target vehicle runs in the target area. In an embodiment of the present application, the road data may include image data or video data, and no specific limitation is made here.

[0024] Among them, when the road data is image data, the road data may include multiple frames of images collected in chronological order. The attitude data of the target vehicle also includes multiple attitude data in chronological order. In an embodiment of the present application, the attitude data of the target vehicle is used to establish a world coordinate system with the target vehicle as the origin, so that the coordinates of the road data can be determined based on this world coordinate system.

[0025] As a way, in response to a data acquisition instruction, road data of a target area is collected by vehicle sensors provided in a target vehicle.

[0026] Among them, the data acquisition instruction can be an instruction sent by other vehicles or electronic devices that have established a communication connection with the target vehicle. Optionally, the data acquisition instruction can also be an instruction triggered after detecting that the target vehicle has performed a specified operation.

[0027] In the case where the data acquisition instruction is an instruction sent by other vehicles or electronic devices that have established a communication connection with the target vehicle, the other vehicles or electronic devices can periodically send data acquisition instructions to the target vehicle. When the target vehicle receives the data acquisition instruction sent by the other vehicles or electronic devices, in response to the data acquisition instruction, it starts to collect road data of the target area through vehicle sensors provided in the target vehicle. Among them, the electronic device can be a smart phone, a tablet computer, a computer, a smart wearable device, etc., which is not specifically limited here.

[0028] In the case where the data acquisition instruction is an instruction triggered after detecting that the target vehicle has performed a specified operation, it can be detected in real time whether the target vehicle has performed the specified operation. Once it is detected that the target vehicle has performed the specified operation, the target vehicle has triggered the data acquisition instruction. Furthermore, the target vehicle can, in response to the data acquisition instruction, start to collect road data of the target area through vehicle sensors provided in the target vehicle. Among them, the specified operation is an operation that can trigger the data acquisition instruction set in advance. For example, the specified operation can be set as a clock synchronization operation or a start operation, etc.

[0029] Furthermore, a series of data collection points can be set in advance in the target area. When the target vehicle runs to a data collection point, road data of the data collection point can be collected through vehicle sensors provided in the target vehicle. At this time, the road data corresponding to the target area is the road data corresponding to this series of data collection points.

[0030] In the embodiment of the present application, there can be multiple vehicle sensors provided in the target vehicle. When there are multiple vehicle sensors, the multiple vehicle sensors can be provided at different positions of the target vehicle.

[0031] Step S120: Perform three-dimensional reconstruction on the target area based on the road data to obtain a three-dimensional model of the target area.

[0032] In an embodiment of the present application, after obtaining the road data of the target area, the target area can be three-dimensionally reconstructed based on the collected road data of the target area. Among them, three-dimensional reconstruction refers to establishing a mathematical model suitable for computer representation and processing of three-dimensional objects. Three-dimensional reconstruction is the basis for processing, operating, and analyzing its properties in a computer environment, and is also a key technology for establishing a virtual reality expressing the objective world in a computer.

[0033] The three-dimensional model is the three-dimensional scene information corresponding to the target area, that is, the information obtained after three-dimensionally reconstructing the target area through three-dimensional reconstruction technology. The three-dimensional model refers to using mathematical equations or geometric figures in three-dimensional space to describe the shape, size, position, attitude, and other attributes of the objects in the target area.

[0034] As a method, the steps of three-dimensional reconstruction can include: image acquisition, camera calibration, feature extraction, stereo matching, and three-dimensional reconstruction, etc., five steps.

[0035] (1) Image acquisition: Before image processing, a camera (in the embodiment of the present application, it is a vehicle-mounted sensor) is used to acquire two-dimensional images of three-dimensional objects.

[0036] (2) Camera calibration: Through camera (in the embodiment of the present application, it is a vehicle-mounted sensor) calibration, an effective imaging model is established, and the internal and external parameters of the camera (in the embodiment of the present application, it is a vehicle-mounted sensor) are solved, so as to obtain the coordinates of three-dimensional points in space in combination with the matching results of the images, thereby achieving the purpose of three-dimensional reconstruction.

[0037] (3) Feature extraction: Features mainly include feature points, feature lines, and regions. In most cases, feature points are used as matching primitives, and the form of feature point extraction is closely related to the matching strategy.

[0038] Feature extraction algorithms can be summarized as: methods based on directional derivatives, methods based on image brightness contrast, and methods based on mathematical morphology.

[0039] (4) Stereo matching: Stereo matching refers to establishing a correspondence relationship between image pairs based on the extracted features, that is, a one-to-one correspondence relationship between the imaging points of the same physical space point in two different images.

[0040] (5) Three-dimensional reconstruction: With more accurate matching results, combined with the internal and external parameters calibrated by the camera (in the embodiment of the present application, it is a vehicle-mounted sensor), the three-dimensional scene information can be restored (obtaining the three-dimensional model in the embodiment of the present application). Because the accuracy of three-dimensional reconstruction is affected by factors such as matching accuracy and errors in the internal and external parameters of the camera.

[0041] That is, as can be known from the above steps, when performing 3D reconstruction on the target area, it is necessary to first calibrate the vehicle sensors, establish an effective imaging model, solve the internal and external parameters of the vehicle sensors, and then match the acquired road data through the effective imaging model to obtain the coordinates of the spatial 3D points corresponding to the road data; after obtaining the coordinates of the spatial 3D points corresponding to the road data, feature extraction can be performed on the spatial 3D points corresponding to the road data, so that the corresponding relationship between image pairs can be established based on the extracted features, and then based on the corresponding relationship between image pairs and the internal and external parameters calibrated by the vehicle-mounted sensors, the 3D model corresponding to the target area can be obtained.

[0042] Step S130: Based on the 3D model and preset scene elements, construct a virtual 3D scene corresponding to the preset scene elements.

[0043] In the embodiment of the present application, the preset scene elements are scene elements preset for enriching the 3D model. Optionally, the preset scene elements can also be understood as the things that need to be added to the 3D model when building a preset virtual 3D scene (for example, a continuous traffic flow scene in autonomous driving), such as the preset scene elements can be at least one of vehicles, pedestrians, traffic lights, sidewalks, vehicle running trajectories, etc. The virtual 3D scene is a virtual simulation scene built to be the same as the scene where the target vehicle actually runs on the road.

[0044] As a way, the preset scene elements can be determined according to the virtual 3D scene to be constructed. For example, if the virtual 3D scene to be constructed is a continuous virtual traffic flow scene, then the determined preset scene elements may include vehicles, pedestrians, traffic lights, sidewalks, etc.; if the virtual 3D scene to be constructed is a functional scene, then the determined preset scene elements may include traffic lights, pedestrians, etc. Among them, the functional scene is a scene built when simulating a certain state in autonomous driving. For example, the functional scene can be a scene built when simulating the traffic light state in autonomous driving; the continuous virtual traffic flow scene refers to a scene built when simulating the traffic conditions within a preset time period in autonomous driving. In the embodiment of the present application, the continuous virtual traffic flow scene may include at least one functional scene, which is not specifically limited herein.

[0045] Optionally, in the embodiment of the present application, after obtaining the 3D model corresponding to the target area, a virtual 3D scene corresponding to the preset scene elements can be constructed by combining the 3D model corresponding to the target area and the preset scene elements. Among them, when combining the 3D model corresponding to the target area and the preset scene elements to construct a virtual 3D scene corresponding to the preset scene elements, the preset scene elements can be set at the preset positions in the 3D model corresponding to the target area, so that the virtual 3D scene corresponding to the preset scene elements can be obtained.

[0046] A 3D reconstruction method provided by this application first collects road data of a target area through vehicle sensors disposed on a target vehicle, then performs 3D reconstruction on the target area based on the road data to obtain a 3D model of the target area, and finally constructs a virtual 3D scene corresponding to preset scene elements based on the 3D model and the preset scene elements. Through the above method, by combining the obtained 3D model of the target scene and the preset scene elements, a virtual 3D scene that meets various usage scenarios can be provided, taking into account the efficiency of simulation and the requirements of high-fidelity simulation.

[0047] Please refer to Figure 2 , a 3D reconstruction method provided by an embodiment of this application, the method includes:

[0048] Step S210: Collect road data of a target area through vehicle sensors disposed on a target vehicle.

[0049] As a way, the vehicle sensors include multiple front-view cameras and multiple surround-view cameras; the collecting road data of the target area through the vehicle sensors disposed on the target vehicle includes: obtaining first road data of the target vehicle through the multiple front-view cameras, where the first road data is the road data in front of the target vehicle and the attitude data of the target vehicle when the target vehicle is in the target area; obtaining second road data of the target vehicle through the multiple surround-view cameras, where the second road data is the road data around the target vehicle and the attitude data of the target vehicle when the target vehicle is in the target area; and taking the first road data and the second road data as the road data of the target area.

[0050] In the embodiment of this application, the multiple front-view cameras refer to front-view cameras that can capture multiple pictures simultaneously. For example, the multiple front-view cameras can be 2 high-definition front-view cameras; similarly, the multiple surround-view cameras refer to surround-view cameras that can capture multiple pictures simultaneously. For example, the multiple surround-view cameras can be 4 surround-view cameras. Optionally, the multiple front-view cameras can also be understood as multiple front-view cameras, that is, 2 high-definition front-view cameras can be understood as having two high-definition front-view cameras; similarly, the multiple surround-view cameras can be understood as multiple surround-view cameras, that is, 4 surround-view cameras can be understood as 4 surround-view cameras. Among them, multiple can be understood as more than 1 way, such as 2-way, 4-way, 6-way, 8-way, 16-way, 32-way, etc. The front-view camera refers to a camera installed in front of the vehicle, and the surround-view camera or panoramic image monitoring system can stitch the bird's-eye view pictures in all directions on the top of the car and dynamically display them on the liquid crystal screen in the car. Among them, the viewing angle of the surround-view camera is much larger than that of the front-view camera. For example, the viewing angle of the front-view camera can be 45 。, the viewing angle of the surround camera can be 180 。 .

[0051] As a way, multiple forward-looking cameras can be set on the front windshield of the target vehicle, and multiple surround cameras can be installed near the logo of the target vehicle, on the left and right rearview mirrors of the target vehicle, on the front windshield of the target vehicle, etc. The installation positions of the multiple forward-looking cameras and the multiple surround cameras are not specifically limited here, as long as the surrounding images of the target vehicle can be completely obtained.

[0052] The first road data refers to the image data or video data in the front area of the target vehicle during the operation of the target vehicle in the target area obtained by the multiple forward-looking cameras. The second road data refers to the image data or video data in the front area, rear area, left area, and right area of the target vehicle during the operation of the target vehicle in the target area obtained by the multiple surround cameras.

[0053] In the embodiments of the present application, before the multiple forward-looking cameras and the multiple surround cameras collect road data, the clock synchronization of the multiple forward-looking cameras and the multiple surround cameras can be ensured first, so that the multiple forward-looking cameras and the multiple surround cameras can collect road data at the same moment.

[0054] As a way, before obtaining the first road data of the target vehicle through the multiple forward-looking cameras, it further includes: sending a clock synchronization trigger signal to the multiple forward-looking cameras and the multiple surround cameras, so that the multiple forward-looking cameras and the multiple surround cameras maintain clock synchronization based on the clock synchronization trigger signal.

[0055] Among them, the clock synchronization trigger signal can be a control signal sent by the controller of the target vehicle to the multiple forward-looking cameras and the multiple surround cameras. This clock synchronization trigger signal is used to control the multiple forward-looking cameras and the multiple surround cameras to perform clock synchronization, that is, the multiple forward-looking cameras and the multiple surround cameras maintain clock synchronization.

[0056] As another way, send a first clock synchronization trigger signal to the multiple forward-looking cameras to make the multiple forward-looking cameras maintain clock synchronization based on the first clock synchronization trigger signal; send a second clock synchronization trigger signal to the multiple surround cameras to make the multiple forward-looking cameras maintain clock synchronization based on the second clock synchronization trigger signal.

[0057] That is, the clock synchronization of the multiple forward-looking cameras and the multiple surround cameras can be performed separately, so that only the clock synchronization of the road data collected by the multiple forward-looking cameras and the clock synchronization of the road data collected by the multiple surround cameras need to be ensured.

[0058] In the embodiments of the present application, the transmission times of the first clock synchronization trigger signal and the second clock synchronization trigger signal can be different, that is, the multi-channel forward-looking cameras and the multi-channel surround-view cameras can perform clock synchronization at different times.

[0059] Optionally, after obtaining the first road data and the second road data of the target area, clock synchronization check can be performed on the first road data and the second road data to at least ensure clock synchronization of the multi-channel forward-looking cameras and clock synchronization of the multi-channel surround-view cameras.

[0060] For the first road data and the second road data, the road data that fails to maintain clock synchronization can be eliminated or clock synchronization can be performed again.

[0061] Step S220: Perform three-dimensional reconstruction on the target area based on the road data to obtain a three-dimensional model of the target area.

[0062] As a method, the performing three-dimensional reconstruction on the target area based on the road data to obtain a three-dimensional model of the target area includes: performing three-dimensional reconstruction on the road area in front of the target vehicle in the target area based on the first road data to obtain a first three-dimensional model of the target area; performing three-dimensional reconstruction on the road area around the target vehicle in the target area based on the second road data to obtain a second three-dimensional model corresponding to the target area; determining the three-dimensional model of the target area based on the first three-dimensional model and the second three-dimensional model.

[0063] In the embodiments of the present application, when performing three-dimensional reconstruction on the target area, three-dimensional reconstruction can be performed on different areas of the target area based on the road data collected by the multi-channel forward-looking cameras and the multi-channel surround-view cameras respectively. Specifically, the first road data collected by the multi-channel forward-looking cameras can be used to perform three-dimensional reconstruction on the front area of the target vehicle to obtain a first three-dimensional model; the second road data collected by the multi-channel surround-view cameras can be used to perform three-dimensional reconstruction on the front area, rear area, left area, and right area of the target vehicle to obtain a second three-dimensional model, so that the three-dimensional model corresponding to the target area can be obtained based on the first three-dimensional model and the second three-dimensional model.

[0064] When obtaining the three-dimensional model corresponding to the target area based on the first three-dimensional model and the second three-dimensional model, the three-dimensional model corresponding to the target area can be obtained simply by combining the first three-dimensional model and the second three-dimensional model.

[0065] Optionally, when obtaining the 3D model corresponding to the target area based on the first 3D model and the second 3D model, the first 3D model and the second 3D model can also be fused according to a preset fusion rule to obtain the 3D model corresponding to the target area. The preset fusion rule can be a rule for fusing 3D models set in advance. For example, the preset fusion rule can be a grid vertex pre-fusion algorithm.

[0066] Step S230: Based on the 3D model and preset scene elements, construct a virtual 3D scene corresponding to the preset scene elements.

[0067] In the embodiments of the present application, the preset scene elements can be manually set through scene editing software or generated by a pre-trained model. When manually setting the preset scene elements through scene editing software, elements such as vehicles and pedestrians can be manually added through the scene editing software, and the movement trajectories of the corresponding scene elements can be set.

[0068] As a way, before constructing the virtual 3D scene corresponding to the preset scene elements based on the 3D model and the preset scene elements, it further includes: obtaining preset scene parameters, where the preset scene parameters are used to limit the preset scene elements; inputting the preset scene parameters and the 3D model into a pre-trained scene setting model, and obtaining the preset scene elements corresponding to the preset scene parameters output by the scene setting model.

[0069] In the embodiments of the present application, the preset scene parameters refer to the parameters for limiting the preset scene elements. For example, if the preset scene elements include vehicles, at this time, the preset scene parameters can be vehicle density, traffic flow volume, etc., which are not specifically limited herein. Optionally, the preset scene parameters can include traffic flow volume, traffic flow density, pedestrian density, pedestrian volume, number of traffic lights, etc., which are not specifically limited herein.

[0070] The scene setting model can be a pre-trained continuous traffic flow model for generating real traffic flows on large-scale road networks. The continuous traffic flow model can be one of the LWR (Lighthill, Whitham and Richards) model, Payne model, Kuhne model, and Wu Zheng model, which are not specifically limited herein.

[0071] As a way, the corresponding relationship between the scene parameters, preset scene elements, and preset scenes can be set in advance. Then, after determining the preset scene that needs to be 3D simulated, the preset scene parameters and preset scene elements corresponding to the preset scene can be determined through the corresponding relationship between the scene parameters and the preset scene.

[0072] Step S240: Render the virtual three-dimensional scene to obtain a rendered virtual three-dimensional scene.

[0073] In the embodiment of the present application, rendering the virtual three-dimensional scene refers to the process of setting the scene, assigning object materials, textures, lights and other elements. After the virtual three-dimensional scene is established, elements such as skin and texture can be added to the virtual three-dimensional scene through rendering for material mapping. Lights and auxiliary lines will also be added, and 3D modeling and rendering software is used to create an illusion of light, shadow, color, etc. in people's vision, so that the virtual three-dimensional scene presents an image with a high sense of two-dimensional reality from the three-dimensional model grid. Rendering can complement the deficiencies of modeling and surfaceize complex modeling.

[0074] Step S250: Identify the rendered virtual three-dimensional scene to obtain an identification result corresponding to the virtual three-dimensional scene.

[0075] In the embodiment of the present application, identifying the rendered virtual three-dimensional scene refers to outputting recognizable image and video information in the rendered virtual three-dimensional scene through a perception algorithm. Among them, the perception algorithm refers to the perception algorithm in autonomous driving.

[0076] As a way, the identifying the rendered virtual three-dimensional scene to obtain an identification result corresponding to the virtual three-dimensional scene includes: inputting the rendered virtual three-dimensional scene into a pre-trained perception model, and obtaining the virtualized image and video information corresponding to the virtual three-dimensional scene output by the perception model.

[0077] After obtaining the identification result corresponding to the virtual three-dimensional scene, the identification result of the virtual three-dimensional scene can be compared with the true identification result of the virtual three-dimensional scene, so that the parameters of the perception model can be adjusted based on the comparison result.

[0078] A three-dimensional reconstruction method provided by the present application can complete the reconstruction of the virtual three-dimensional scene by using the existing perception devices of the target vehicle, and then the reconstructed virtual three-dimensional scene can be directly applied to the simulation test of the perception algorithm in the corresponding area, which can ensure a high-fidelity effect and improve the operation efficiency of the simulation engine.

[0079] Please refer to Figure 3 , a three-dimensional reconstruction device 300 provided by the embodiment of the present application, the device 300 includes:

[0080] A data acquisition unit 310, configured to collect road data of a target area through vehicle sensors disposed on a target vehicle.

[0081] As a way, the vehicle sensor includes a multi-channel front view camera and a multi-channel surround view camera; the data acquisition unit 310 is specifically configured to obtain first road data of the target vehicle through the multi-channel front view camera, where the first road data is the road data in front of the target vehicle and the attitude data of the target vehicle when the target vehicle is in the target area; obtain second road data of the target vehicle through the multi-channel surround view camera, where the second road data is the road data around the target vehicle and the attitude data of the target vehicle when the target vehicle is in the target area; and use the first road data and the second road data as the road data of the target area.

[0082] The reconstruction unit 320 is configured to perform three-dimensional reconstruction on the target area based on the road data to obtain a three-dimensional model of the target area.

[0083] As a way, the reconstruction unit 320 is specifically configured to perform three-dimensional reconstruction on the road area in front of the target vehicle in the target area based on the first road data to obtain a first three-dimensional model of the target area; perform three-dimensional reconstruction on the road area around the target vehicle in the target area based on the second road data to obtain a second three-dimensional model corresponding to the target area; and determine the three-dimensional model of the target area based on the first three-dimensional model and the second three-dimensional model.

[0084] The construction unit 330 is configured to construct a virtual three-dimensional scene corresponding to the preset scene elements based on the three-dimensional model and the preset scene elements.

[0085] Please refer to Figure 4 , the device 300 further includes:

[0086] The clock synchronization unit 340 is configured to send a clock synchronization trigger signal to the multi-channel front view camera and the multi-channel surround view camera, so that the multi-channel front view camera and the multi-channel surround view camera maintain clock synchronization based on the clock synchronization trigger signal.

[0087] The rendering unit 350 is configured to render the virtual three-dimensional scene to obtain a rendered virtual three-dimensional scene; and identify the rendered virtual three-dimensional scene to obtain an identification result corresponding to the virtual three-dimensional scene.

[0088] As a way, the rendering unit 350 is specifically configured to input the rendered virtual three-dimensional scene into a pre-trained perception model to obtain a virtualized image and video information corresponding to the virtual three-dimensional scene output by the perception model.

[0089] An element acquisition unit 360 is configured to acquire preset scenario parameters, where the preset scenario parameters are used to restrict preset scenario elements; input the preset scenario parameters and the three-dimensional model into a pre-trained scenario setting model, and acquire the preset scenario elements corresponding to the preset scenario parameters output by the scenario setting model.

[0090] It should be noted that the device embodiments in this application correspond to the foregoing method embodiments. For the specific principles in the device embodiments, reference can be made to the content in the foregoing method embodiments, which will not be elaborated here.

[0091] The following will be combined with Figure 5 to describe a vehicle provided by this application.

[0092] Please refer to Figure 5 , based on the foregoing three-dimensional reconstruction method and device, another vehicle 800 capable of executing the foregoing three-dimensional reconstruction method is further provided in an embodiment of this application. The vehicle 800 includes one or more (only one is shown in the figure) processors 802, a memory 804, an image processor 806, a camera 807, and a network module 808 that are coupled to each other. Among them, a program that can execute the content in the foregoing embodiments is stored in the memory 804, and the processor 802 can execute the program stored in the memory 804.

[0093] Among them, the processor 802 may include one or more processing cores. The processor 802 connects various parts within the entire vehicle 800 using various interfaces and lines, and executes various functions of the vehicle 800 and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 804, and by calling data stored in the memory 804. Optionally, the processor 802 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). The processor 802 may integrate one or a combination of several of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, application programs, etc.; the GPU is responsible for rendering and drawing display content; the modem is used to process wireless communication. It can be understood that the foregoing modem may not be integrated into the processor 802 and may be implemented separately through a communication chip.

[0094] The memory 804 may include a Random Access Memory (RAM), or may also include a Read-Only Memory. The memory 804 can be used to store instructions, programs, codes, code sets, or instruction sets. The memory 804 may include a program storage area and a data storage area. Among them, the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data created during the use of the vehicle 800 (such as a phone book, audio and video data, chat record data), etc.

[0095] The camera 807 is used to collect images and transmit the collected images to the image processor 806 for processing. Among them, the camera 807 may include a multi-channel front view camera and a multi-channel surround view camera.

[0096] The image processor 806 is used to perform three-dimensional reconstruction on the target area based on the road parameters of the target area collected by the camera 807.

[0097] The network module 808 is used to receive and send electromagnetic waves, realize the mutual conversion between electromagnetic waves and electrical signals, so as to communicate with a communication network or other devices, such as communicating with a vehicle. The network module 808 may include various existing circuit elements for performing these functions. For example, an antenna, a radio frequency transceiver, a digital signal processor, an encryption / decryption chip, a Subscriber Identity Module (SIM) card, a memory, etc. The network module 808 can communicate with various networks such as the Internet, an enterprise intranet, a wireless network, or communicate with other devices through a wireless network. The above wireless network may include a cellular phone network, a wireless local area network, or a metropolitan area network. For example, the network module 808 can interact with a base station for information.

[0098] Please refer to Figure 6 , which shows a structural block diagram of a computer-readable storage medium provided by an embodiment of the present application. Program code is stored in the computer-readable storage medium 900, and the program code can be called by a processor to execute the method described in the above method embodiments.

[0099] The computer-readable storage medium 900 can be an electronic memory such as a flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, a hard disk, or a ROM. Optionally, the computer-readable storage medium 900 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 900 has a storage space for program code 910 that executes any of the method steps in the above-described method. These program codes can be read from or written to one or more computer program products. The program code 910 can be compressed in a suitable form, for example.

[0100] A three-dimensional reconstruction method, apparatus, vehicle, and storage medium provided by this application first collect road data of a target area through vehicle sensors disposed on a target vehicle, then perform three-dimensional reconstruction on the target area based on the road data to obtain a three-dimensional model of the target area, and finally construct a virtual three-dimensional scene corresponding to a preset scene element based on the three-dimensional model and the preset scene element. Through the above method, by combining the obtained three-dimensional model of the target scene and the preset scene element, a virtual three-dimensional scene that meets various usage scenarios can be provided, taking into account the efficiency of simulation and the requirements of high-fidelity simulation.

[0101] The embodiments of the present invention have been described above in conjunction with the accompanying drawings. However, the present invention is not limited to the above specific embodiments. The above specific embodiments are merely illustrative and not restrictive. Under the inspiration of the present invention, those of ordinary skill in the art can also make many forms without departing from the spirit and scope protected by the present invention and the claims, and all of them belong to the protection scope of the present invention.

Claims

1. A three-dimensional reconstruction method, characterized in that, The method includes: Collecting road data of a target area through vehicle sensors disposed on a target vehicle. The road data includes data of a running lane collected by the vehicle sensors disposed in the target vehicle when the target vehicle runs in the target area, environmental data around the target vehicle, and attitude data of the target vehicle. The vehicle sensors include multiple forward-looking cameras and multiple surround-view cameras. The surround-view cameras can splice bird's-eye view images in all directions on the top of the vehicle and dynamically display them on a liquid crystal display screen in the vehicle; Performing three-dimensional reconstruction on the target area based on the road data to obtain a three-dimensional model of the target area; Constructing a virtual three-dimensional scene corresponding to the preset scene elements based on the three-dimensional model and the preset scene elements. The preset scene elements are determined based on the virtual three-dimensional scene to be constructed. The virtual three-dimensional scene is a functional scene, and the functional scene is a scene built when simulating a certain state in autonomous driving; Rendering the virtual three-dimensional scene to obtain a rendered virtual three-dimensional scene; Inputting the rendered virtual three-dimensional scene into a pre-trained perception model to obtain a virtualized image and video information corresponding to the virtual three-dimensional scene output by the perception model.

2. The method according to claim 1, wherein The vehicle sensors include multiple forward-looking cameras and multiple surround-view cameras. The collecting road data of the target area through the vehicle sensors disposed on the target vehicle includes: Obtaining first road data of the target vehicle through the multiple forward-looking cameras. The first road data is road data in front of the target vehicle and attitude data of the target vehicle when the target vehicle is in the target area; Obtaining second road data of the target vehicle through the multiple surround-view cameras. The second road data is road data around the target vehicle and attitude data of the target vehicle when the target vehicle is in the target area; Using the first road data and the second road data as the road data of the target area.

3. The method according to claim 2, characterized in that, The performing three-dimensional reconstruction on the target area based on the road data to obtain a three-dimensional model of the target area includes: Performing three-dimensional reconstruction on the road area in front of the target vehicle in the target area based on the first road data to obtain a first three-dimensional model of the target area; Performing three-dimensional reconstruction on the road area around the target vehicle in the target area based on the second road data to obtain a second three-dimensional model corresponding to the target area; Determining the three-dimensional model of the target area based on the first three-dimensional model and the second three-dimensional model.

4. The method according to claim 2, wherein Before obtaining the first road data of the target vehicle through the multiple forward-looking cameras, it further includes: Sending a clock synchronization trigger signal to the multiple forward-looking cameras and the multiple surround-view cameras so that the multiple forward-looking cameras and the multiple surround-view cameras maintain clock synchronization based on the clock synchronization trigger signal.

5. The method according to claim 1, characterized in that, Before constructing a virtual three-dimensional scene corresponding to the preset scene elements based on the three-dimensional model and the preset scene elements, the following steps are also included: Obtain preset scene parameters, where the preset scene parameters are used to restrict the preset scene elements; Input the preset scene parameters and the three-dimensional model into a pre-trained scene setting model, and obtain the preset scene elements corresponding to the preset scene parameters output by the scene setting model.

6. A three-dimensional reconstruction device, characterized in that, The device includes: A data acquisition unit, configured to collect road data of a target area through vehicle sensors disposed on a target vehicle. The road data includes data of a running lane collected by the vehicle sensors disposed on the target vehicle when the target vehicle runs in the target area, environmental data around the target vehicle, and attitude data of the target vehicle. The vehicle sensors include a plurality of forward-looking cameras and a plurality of surround-view cameras. The surround-view cameras can splice bird's-eye view images in all directions on the top of the vehicle and dynamically display them on a liquid crystal screen in the vehicle; A reconstruction unit, configured to perform three-dimensional reconstruction on the target area based on the road data to obtain a three-dimensional model of the target area; A construction unit, configured to construct a virtual three-dimensional scene corresponding to the preset scene elements based on the three-dimensional model and the preset scene elements. The preset scene elements are determined based on the virtual three-dimensional scene to be constructed. The virtual three-dimensional scene is a functional scene, and the functional scene is a scene built when simulating a certain state in autonomous driving; A rendering unit, configured to render the virtual three-dimensional scene to obtain a rendered virtual three-dimensional scene; input the rendered virtual three-dimensional scene into a pre-trained perception model, and obtain a virtualized image and video information corresponding to the virtual three-dimensional scene output by the perception model.

7. A vehicle, characterized in that, Comprising one or more processors; one or more programs are stored in a memory and configured to be executed by the one or more processors to perform the method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, Program code is stored in the computer-readable storage medium, where the method according to any one of claims 1-5 is executed when the program code is run by a processor.

Images