A single-image vehicle reconstruction method and system based on 3DGS

Abstract

The present invention discloses a single-image vehicle reconstruction method and system based on 3DGS, belonging to the field of machine vision technology. The method comprises: obtaining an autonomous driving dataset and preprocessing the data in the autonomous driving dataset; wherein the data in the autonomous driving dataset is multi-view RGB image data obtained by multiple RGB cameras installed on the autonomous driving vehicle capturing the surrounding environment at different perspectives; constructing a vehicle reconstruction model based on 3DGS, wherein the input of the vehicle reconstruction model is a single RGB image and the output is a reconstructed vehicle image; training the vehicle reconstruction model using the preprocessed autonomous driving dataset; and using the trained vehicle reconstruction model to achieve vehicle reconstruction based on the single RGB image. The technical solution provided by the present invention significantly reduces the cost of sensors and data acquisition and is particularly suitable for dynamic driving scenarios.

Description

Technical Field

[0001] The present invention relates to the field of machine vision technology, and in particular to a single-image vehicle reconstruction method and system based on 3DGS. Background Art

[0002] Autonomous driving scene simulation is one of the key links in realizing autonomous driving technology. It aims to reconstruct the three-dimensional scene around the vehicle through sensors, providing the autonomous driving vehicle with a comprehensive understanding of the surrounding environment, thereby supporting its safe and effective decision-making and navigation.

[0003] With the rapid development of computer vision and artificial intelligence technologies, 3D reconstruction has become a crucial component in many application areas. In particular, in areas such as autonomous driving, intelligent transportation, virtual reality, and augmented reality, accurate 3D vehicle model reconstruction is fundamental to understanding and perceiving complex scenes. Currently, common 3D reconstruction methods are primarily based on multi-view stereo, structured light, and laser scanning. These technologies enable the coordinated operation of multiple sensors from multiple angles to reconstruct the 3D structure of a scene.

[0004] Traditional 3D reconstruction methods typically rely on multi-view data or multimodal priors (such as lidar point clouds), but these methods have limitations, such as high sensor costs, poor real-time performance, and difficulty aligning multiple views. In addition, dynamic objects in complex urban environments (such as other vehicles and pedestrians) also increase the difficulty of scene reconstruction. Depth information is limited in monocular reconstruction. Monocular depth information cannot reflect the true depth of the invisible parts of the vehicle. For example, given the perspective of the rear of the vehicle, the depth of the front of the vehicle is obscured. During reconstruction, the network still needs to imagine to derive this part of the point cloud.

[0005] Existing deep learning-based sparse-view 3D reconstruction efforts can be categorized into three technical approaches. The first involves manually or inferring the target object in a single image to directly obtain its 3D shape. Texture generation models are then used to color and manipulate the lighting. This entire process requires phased training, resulting in increased training and testing time. This approach requires a large amount of input data and a large number of different data types, and the resulting 3D shapes are very similar and lack uniqueness.

[0006] The second approach is to directly use a given priori template for posture learning, fine-tune the template geometry, and simultaneously perform sampling to learn color features in order to achieve grid-based 3D model reconstruction. Specifically, rendering is performed using a differentiable renderer, followed by post-processing to generate a projection of a certain 3D shape onto a 2D shape; segmentation maps, keypoint coordinates, RGB pixels, etc. are generated based on the projection, and then the segmentation and color are optimized. The disadvantage of this approach is that there is a priori assumption that the training object is a symmetrical object. Training requires initializing the template based on the prior shape of the synthetic object. For objects with complex structures and asymmetric shapes, the learning difficulty is significantly increased. There is still some work that uses generative adversarial networks to improve the quality of geometric textures. However, due to the use of self-supervised learning, there is no clear definition of the true value, and it is easy to converge to a suboptimal state or fail to converge. In addition, due to the limitations of the volume and complexity of the object, the learning effect for complex objects is not good.

[0007] The third method is to use Neural Radiance Fields (NeRF) and volume rendering to learn three-dimensional structures, which can directly learn the three-dimensional shape and color of objects. Currently, using this method for three-dimensional reconstruction is the mainstream in academia, and most of the related papers in recent years have focused on this technology. Compared with other methods, NeRF-based methods can generate higher-definition virtual images, but because such algorithms are based on implicit three-dimensional representations, they will reduce the explicit geometric quality and multi-view three-dimensional consistency, and require longer training time and higher computational overhead.

[0008] Recovering three-dimensional information from a single two-dimensional image requires overcoming multiple limitations such as perspective, lighting, and occlusion. Existing single-image reconstruction work mainly relies on deep learning methods, using CNN or Transformer to extract geometric information such as depth and normals from the image, thereby generating corresponding three-dimensional representations such as point clouds, meshes, voxels, or implicit functions. Three-dimensional reconstruction of autonomous driving scenes in natural environments can reduce the difficulty of data acquisition, does not require complex calibration processes, and does not rely on multi-view or multi-modal priors. Compared with multi-view or multi-sensor data fusion, single-image reconstruction reduces dependence on multi-source data and helps improve the real-time performance of the system.

[0009] However, using only a single 2D RGB image as input means that some parts of the target vehicle are necessarily invisible when extracting features. Without using timestamps and obj_ids, the target vehicle cannot be tracked, and multiple viewpoints cannot be matched. The sparse viewing angles in autonomous driving datasets also lead to an uneven distribution of vehicle poses. For the ego car, the visible viewpoints of the vehicle are unevenly distributed, generally focusing on the rear and front. If there are more front-facing cameras than rear-facing cameras, the rear viewpoint will be more than the front viewpoint. Furthermore, autonomous driving datasets are limited by the ego car's capture method, resulting in a limited view of the driving scene. The ego car is confined to driving lanes and must obey traffic rules, not turning arbitrarily. Therefore, although autonomous driving datasets often feature RGB cameras from multiple angles—for example, the ego cars in PandaSet and KITTI360 both have six cameras—these ego car-centric capture angles are effectively six different single viewpoints, which are not helpful for constructing target objects from multiple perspectives. When the ego car is driving straight along the road, none of the ego car's cameras can see a partially occluded object, such as a parked car, facing away from the road. Furthermore, autonomous driving datasets contain a large number of dynamic foreground objects, some of which are moving too fast relative to the ego car, resulting in these foreground objects appearing in only a few video frames, making reconstruction of these foreground objects very difficult.

[0010] In summary, existing 3D reconstruction technologies usually require multiple input images or a large amount of computing resources, which makes them difficult to adapt to application scenarios with limited resources or scarce image information. Summary of the Invention

[0011] The present invention provides a single-image vehicle reconstruction method and system based on 3DGS to solve the technical problem that the current existing three-dimensional reconstruction technology usually requires multiple input images or a large amount of computing resources, which is difficult to adapt to application scenarios with limited resources or scarce image information.

[0012] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0013] In one aspect, the present invention provides a single-image vehicle reconstruction method based on 3DGS, comprising:

[0014] Obtaining an autonomous driving dataset and preprocessing data in the autonomous driving dataset; wherein the data in the autonomous driving dataset is multi-view RGB image data obtained by capturing the surrounding environment from different perspectives by a camera installed on the autonomous driving vehicle;

[0015] Constructing a vehicle reconstruction model based on 3DGS; wherein the input of the vehicle reconstruction model is a single RGB image, and the output is a reconstructed vehicle image;

[0016] Use the preprocessed autonomous driving dataset to train the vehicle reconstruction model;

[0017] Using the trained vehicle reconstruction model, the vehicle is reconstructed based on a single RGB image.

[0018] Furthermore, preprocessing includes: label generation, data enhancement, data storage and data quality control.

[0019] Furthermore, the tag generation includes:

[0020] Use pre-trained segmentation models and object detection models to generate vehicle mask data from the original RGB image and add category labels, location information, and pose angle data to the vehicle.

[0021] Furthermore, the data enhancement includes any one or more combinations of random rotation, cropping, scaling, and adding noise.

[0022] Furthermore, the data storage includes:

[0023] Based on the data usage and scenarios, the data is divided into training set, validation set and test set in preset proportions, and the data is stored in structured folders. When storing, RGB images are stored in PNG format, the generated point cloud data is stored in PLY format, and the label data is stored in json format. The data in the training set contains a variety of vehicle types, angles and environments, which are used for model training. The data in the validation set contains some scenes and angles that have not been seen by the model, which are used to evaluate model performance. The test set is used for the final performance test of the model.

[0024] Furthermore, the data quality control includes:

[0025] Perform quality checks on the raw data to remove data that is ambiguous, incomplete, does not meet the size requirements (too small), or has incorrect labels.

[0026] Furthermore, the process of processing the input RGB image by the vehicle reconstruction model includes:

[0027] Generate a mask for the vehicle using a pre-trained mask generator; and complete the vehicle mask and texture by fine-tuning the Lama model on the pre-processed autonomous driving dataset;

[0028] A preset pose feature extractor is used to learn the vehicle's pose features, which are then fused with the completed vehicle texture and mask. The features are then fed into a pre-trained encoder to generate latent features. The input RGB image and mask are loaded into the GPU memory and stored as tensors. The latent features and the fused feature tensor are stored in a dedicated buffer in the GPU memory as continuous floating-point arrays.

[0029] Using the sphere template and the Fibonacci sampling method, a uniformly distributed point cloud is generated on the surface of the sphere.

[0030] Convert the 3D point cloud coordinates in the spherical template from Cartesian coordinates to polar coordinates, convert the updated polar coordinates back to rectangular coordinates, and return the deformed 3DGS point cloud attributes to obtain the vehicle point cloud;

[0031] The vehicle point cloud is used for rendering and optimization to obtain the RGB image of the vehicle.

[0032] Furthermore, when converting the 3D point cloud coordinates in the spherical template from Cartesian coordinates to polar coordinates, the extracted features are used to predict the deformation parameters of each point through a dual-plane decoder, and the deformation parameters are used to calculate the collapse factor and the updated polar coordinates.

[0033] Furthermore, the rendering and optimization using the vehicle point cloud includes:

[0034] Create a primitive camera object using the object-camera rotation matrix, camera position, image height, width, and intrinsic parameters provided by the dataset, and use this primitive camera object to render the image from the original perspective.

[0035] Create a camera object symmetrical to the original camera object, create camera pairs at a certain angle, render the normalized vehicle onto a 2D plane, and generate a rotation map of the vehicle;

[0036] By calculating the loss between the rendered image and the original image, back propagation is performed, and the model parameters are optimized so that the reconstruction result is closer to the input image, thereby gradually approaching the three-dimensional structure of the real vehicle.

[0037] On the other hand, the present invention also provides a single-image vehicle reconstruction system based on 3DGS, comprising:

[0038] a data processing module, configured to obtain an autonomous driving dataset and preprocess the data in the autonomous driving dataset; wherein the data in the autonomous driving dataset is multi-view RGB image data obtained by capturing the surrounding environment from different perspectives by a camera installed on the autonomous driving vehicle;

[0039] A model building module, configured to build a 3DGS-based vehicle reconstruction model; wherein the input of the vehicle reconstruction model is a single RGB image, and the output is a reconstructed vehicle image;

[0040] A model training module is used to train the vehicle reconstruction model using the preprocessed autonomous driving dataset;

[0041] The application module is used to use the trained vehicle reconstruction model to reconstruct the vehicle based on a single RGB image.

[0042] On the other hand, the present invention further provides an electronic device, comprising a processor and a memory; wherein the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the above method.

[0043] In yet another aspect, the present invention further provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, and the instruction is loaded and executed by a processor to implement the above method.

[0044] The beneficial effects brought about by the technical solution provided by the present invention include at least:

[0045] The present invention proposes a vehicle three-dimensional model reconstruction method based on three-dimensional Gaussian splattering (3DGS) that relies only on a single two-dimensional image data. It uses only a single 2D RGB picture and does not rely on camera parameters to embed the network, and can learn a normalized distributed 3D point cloud of the vehicle. By using a simple network structure, by aligning the mask shape of the vehicle with the vehicle posture, and utilizing the spherical template and the geometric symmetry of the vehicle, high-quality point cloud reconstruction of the vehicle is achieved. Unlike existing methods such as AutoRF, the present invention explicitly learns the three-dimensional point cloud of the vehicle and well combines the data representation form of 3DGS, which can achieve real-time rendering and solve the discontinuity problem of traditional point-based rendering. It significantly reduces the cost of sensors and data acquisition, and is particularly suitable for dynamic driving scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0047] Figure 1 3DGS-based single-image vehicle reconstruction method provided by an embodiment of the present invention;

[0048] Figure 2 is an architectural diagram of a vehicle reconstruction model provided by an embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram of the generation effect provided by an embodiment of the present invention;

[0050] Figure 4 This is a system block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0052] First, it should be noted that in the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as an "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "exemplarily" is intended to present concepts in a concrete manner. In addition, in the embodiments of the present invention, the meaning of "and / or" can be both or either of the two.

[0053] First embodiment

[0054] This embodiment provides a single-image vehicle reconstruction method based on 3DGS. The method can be implemented by an electronic device. The execution process of the method is as follows: Figure 1 As shown, the following steps are included:

[0055] S1, obtaining an autonomous driving dataset and preprocessing data in the autonomous driving dataset; wherein the data in the autonomous driving dataset is multi-view RGB image data obtained by capturing the surrounding environment from different perspectives by a camera installed on the autonomous driving vehicle;

[0056] It should be noted that the raw data required by the model of the present invention mainly includes RGB images and related label data. Data preprocessing includes label generation, data enhancement, data storage, data quality control, etc.

[0057] RGB images: We use multi-view on-board camera data from autonomous driving datasets (such as Waymo and KITTI). These cameras are installed on autonomous vehicles and capture the surrounding environment from different perspectives. The images cover a variety of scenarios, including urban roads, highways, and rural roads.

[0058] Label generation: Use pre-trained segmentation models (such as Segment-Anything) and object detection models (such as GroundingDINO) to generate vehicle mask data from the original RGB image and add data such as category labels, location information (such as bounding boxes), and pose angles to the vehicle.

[0059] Data augmentation: Data augmentation operations such as random rotation, cropping, scaling, and adding noise are performed during preprocessing to improve the robustness and generalization ability of the model.

[0060] Data Storage:

[0061] Storage format: Preprocessed RGB images are stored in PNG format, and generated point cloud data is stored in PLY format. All labels are stored in JSON format.

[0062] Tiered storage: Data is divided into training, validation, and test sets based on data usage and scenarios, and stored in structured folders in a ratio of 8:1:1 for easy subsequent access and use.

[0063] Training set: mainly contains diverse vehicle types, angles and environments.

[0064] Validation set: Contains some unseen scenes and angles, used to evaluate model performance.

[0065] Test set: Independent of the training and validation sets, it is specifically used for final performance testing.

[0066] In addition, this embodiment also controls data quality, performs automatic quality checks on the collected raw data, and eliminates ambiguous, incomplete, or mislabeled data.

[0067] S2, constructing a vehicle reconstruction model based on 3DGS; wherein the input of the vehicle reconstruction model is a single RGB image, and the output is a reconstructed vehicle image;

[0068] Specifically, the architecture of the model of the present invention is as follows Figure 2 As shown, the data processing process is as follows:

[0069] 1. Mask generation and completion

[0070] Before training, a mask generator based on the pre-trained language segment-anything model is first used to generate a mask for the vehicle by inputting car as a prompt word. Language Segment-Anything is based on the segment-anything segmentation model and the Grounding DINO detection model, and can generate a mask for a specific object in the image. The RGB image of the vehicle is cropped, deformed, and normalized to a resolution of 256×256 to provide a unified input for the network. Since the Waymo dataset lacks object occlusion and truncation rate labels, lower-quality data needs to be repaired. By fine-tuning the lama (Large Mask Inpainting) model on the self-made vehicle dataset, the mask and texture of the vehicle are completed.

[0071] 2. Feature extraction and fusion

[0072] Using an EgoNet-based pose feature extractor, the vehicle's pitch and yaw angles, and other pose features are learned. These features are then fused with the completed vehicle texture and mask features. These features are then fed into a pre-trained FeatUp ResNet50 encoder to generate a latent code. The input RGB image and mask are loaded into the GPU memory and stored as tensors using the CUDA version of torch.Tensor. On the GPU side, tensors are allocated to a shared memory area for direct access by subsequent deep learning networks. The latent code features and the fused feature tensors are stored in a dedicated buffer in the GPU memory in a contiguous floating-point array format (float32) to reduce memory access overhead.

[0073] 3. Gaussian point cloud deformation

[0074] A spherical template is proposed to generate an initialized point cloud. The initial spherical point cloud uses the Fibonacci sampling method to generate a uniformly distributed point cloud on the surface of the sphere.

[0075]

[0076] Where i = 1, 2, ..., N i Represents the index of the point, φ is the golden ratio, and N is the total number of points.

[0077] When deforming the template point cloud, the 3D point cloud coordinates are converted from Cartesian coordinates to polar coordinates.

[0078]

[0079] Among them, the polar coordinate parameter r i ,θ i ,φ iRespectively represent radial distance, polar angle, azimuth, x i ,y i ,z i They are the x, y, and z coordinate values ​​of the Cartesian coordinate system respectively.

[0080] The extracted features are passed through a biplane decoder to predict the deformation parameters of each point, which are used to calculate the collapse factor and the updated polar coordinates.

[0081]

[0082] Among them, δr i ,δθ i , is the deformation parameter, F feat is the extracted feature, D deform is the decoder, p i Represents a three-dimensional coordinate point.

[0083] The features are decoded into two planes, θ-Φ and Φ-θ, to increase the number of parameters. After the deformation is completed, the updated polar coordinates are converted back to rectangular coordinates, and the deformed 3DGS point cloud attributes (coordinates, scaling factor, rotation factor, opacity and color) are returned.

[0084] 4. Rendering and optimization

[0085] First, a primitive camera object C0 is created using the object-camera rotation matrix, camera position, image height, width, and intrinsic parameters provided by the dataset. This camera is then used to render an image from the original perspective. Next, symmetrical camera objects are created, and camera pairs are created at regular intervals. The normalized vehicle is rendered onto a 2D plane to generate a rotational image of the vehicle. By calculating the loss between the original image and the template shape and performing backpropagation, the template shape variables and Gaussian point cloud parameters are adjusted to make the generated 2D image more similar to the input image, thereby gradually approximating the 3D structure of the real vehicle.

[0086] S3, trains the vehicle reconstruction model using the preprocessed autonomous driving dataset;

[0087] S4 uses the trained vehicle reconstruction model to reconstruct the vehicle based on a single RGB image.

[0088] The vehicle image generated by the model of the present invention is as follows Figure 3 shown.

[0089] In summary, this embodiment proposes a method for reconstructing a three-dimensional model of a vehicle based on a three-dimensional Gaussian representation that relies only on a single two-dimensional image data. It uses only a single 2D RGB picture and does not rely on camera parameters to embed the network, and can learn a normalized distributed 3D point cloud of the vehicle. By using a simple network structure, by aligning the mask shape of the vehicle with the vehicle posture, and utilizing the spherical template and the geometric symmetry of the vehicle, high-quality point cloud reconstruction of the vehicle is achieved. Unlike existing methods such as AutoRF, the present invention explicitly learns the three-dimensional point cloud of the vehicle and well combines the data representation form of 3DGS, can achieve real-time rendering, and solves the discontinuity problem of traditional point-based rendering. It significantly reduces the cost of sensors and data acquisition, and is particularly suitable for dynamic driving scenarios.

[0090] Second embodiment

[0091] This embodiment provides a single-image vehicle reconstruction system based on 3DGS, which includes the following modules:

[0092] a data processing module, configured to obtain an autonomous driving dataset and preprocess the data in the autonomous driving dataset; wherein the data in the autonomous driving dataset is multi-view RGB image data obtained by capturing the surrounding environment from different perspectives by a camera installed on the autonomous driving vehicle;

[0093] A model building module, configured to build a 3DGS-based vehicle reconstruction model; wherein the input of the vehicle reconstruction model is a single RGB image, and the output is a reconstructed vehicle image;

[0094] A model training module is used to train the vehicle reconstruction model using the preprocessed autonomous driving dataset;

[0095] The application module is used to use the trained vehicle reconstruction model to reconstruct the vehicle based on a single RGB image.

[0096] It should be noted that the 3DGS-based single-image vehicle reconstruction system of this embodiment corresponds to the 3DGS-based single-image vehicle reconstruction method of the above-mentioned first embodiment; the functions implemented by each functional module in the 3DGS-based single-image vehicle reconstruction system of this embodiment correspond one-to-one to each process step in the 3DGS-based single-image vehicle reconstruction method of the above-mentioned first embodiment; therefore, they will not be repeated here.

[0097] Third embodiment

[0098] This embodiment provides an electronic device, such as Figure 4As shown, the electronic device includes: a central processing unit, a graphics processing unit, and a memory; wherein the processor and the memory can be connected via a communication bus; the memory stores at least one instruction, which is loaded and executed by the processor to implement the method of the first embodiment described above. In addition, the electronic device may also include a transceiver; the processor and the transceiver can be connected via a communication bus, and the transceiver is used to communicate with other devices.

[0099] Next, combine Figure 4 A detailed introduction to the various components of the electronic device is given below:

[0100] The processor is the control center of the electronic device, which may include multiple central processing units (CPUs) and multiple graphics processing units (GPUs). Each of these CPUs can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor can perform various functions of the electronic device by running or executing software programs stored in the memory and calling data stored in the memory.

[0101] In a specific implementation, as an embodiment, the processor may include one or more CPUs, such as Figure 4 The CPU0 and CPU1 shown in FIG are, of course, only exemplary.

[0102] The memory is used to store the software program for executing the solution of the present invention, and the execution is controlled by the processor. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0103] Optionally, the memory may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may be integrated with the processor or exist independently and accessed through the interface circuit ( Figure 4 (not shown) is coupled to the processor, which is not specifically limited in this embodiment of the present invention.

[0104] The transceiver may include a receiver and a transmitter ( Figure 4 The receiver is used to implement the receiving function, and the transmitter is used to implement the sending function. The transceiver can be integrated with the processor or exist independently and communicate with the electronic device through the interface circuit ( Figure 4 (not shown) is coupled to the processor, which is not specifically limited in this embodiment of the present invention.

[0105] In addition, it should be noted that Figure 4 The structure of the electronic device shown in the figure does not constitute a limitation on the device. The actual device may include more or fewer components than shown, or may combine certain components, or arrange the components differently. In addition, the technical effects achieved by the electronic device when executing the method of the first embodiment can refer to the technical effects described in the first embodiment above, and therefore will not be repeated here.

[0106] Fourth embodiment

[0107] This embodiment provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the method of the first embodiment described above. The computer-readable storage medium may be a ROM, random access memory, CD-ROM, magnetic tape, floppy disk, or optical data storage device. The instructions stored therein can be loaded by a processor in a terminal to execute the method described above.

[0108] Furthermore, it should be noted that the present invention may be provided as a method, apparatus, or computer program product. Therefore, embodiments of the present invention may take the form of a fully or partially hardware embodiment, a fully or partially software embodiment, or an embodiment combining software and hardware aspects. Furthermore, when implemented using software, embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code. The computer program product comprises one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are fully or partially generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired connection (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium accessible by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium. The semiconductor medium may be a solid state drive.

[0109] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the process in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0110] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device, so that a series of operation steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for implementing the process in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0111] It should also be noted that, in this document, relational terms such as first and second are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any actual relationship or order between these entities or operations. The terms "include," "comprises," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or terminal device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. Without further limitation, an element defined by the phrase "comprising a..." does not preclude the presence of other identical elements in the process, method, article, or terminal device comprising the element. In addition, the term "and / or" is merely a description of an associative relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: the presence of A alone, the presence of A and B simultaneously, or the presence of B alone, where A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding. "At least one" means one or more, and "more" means two or more. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0112] In addition, it can be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0113] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0114] In the several embodiments provided herein, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of functional modules / units is merely a logical functional division. In actual implementation, other division methods may be used, such as multiple units or components being combined or integrated into another device, or some features being ignored or not implemented. Furthermore, the coupling or direct coupling or communication connection shown or discussed between each other may be through some interface, indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms. Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, i.e., they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of the present embodiment according to actual needs. In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit.

[0115] If the method is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0116] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention. It should be noted that, although preferred embodiments of the present invention have been described, those skilled in the art, once understanding the basic inventive concepts of the present invention, may make various improvements and modifications without departing from the principles of the present invention. Such improvements and modifications should also be considered within the scope of protection of the present invention. Therefore, the appended claims are intended to be interpreted as covering the preferred embodiments and all variations and modifications that fall within the scope of the embodiments of the present invention.

Claims

1. A single-image vehicle reconstruction method based on 3DGS, characterized in that: include: Obtaining an autonomous driving dataset and preprocessing data in the autonomous driving dataset; wherein the data in the autonomous driving dataset is multi-view RGB image data obtained by capturing the surrounding environment from different perspectives by a camera installed on the autonomous driving vehicle; Constructing a vehicle reconstruction model based on 3DGS; wherein the input of the vehicle reconstruction model is a single RGB image, and the output is a reconstructed vehicle image; Use the preprocessed autonomous driving dataset to train the vehicle reconstruction model; Using the trained vehicle reconstruction model, the vehicle is reconstructed based on a single RGB image. The process of the vehicle reconstruction model processing the input RGB image includes: Generate a mask for the vehicle using a pre-trained mask generator; and complete the vehicle mask and texture by fine-tuning the Lama model on the pre-processed autonomous driving dataset; A preset pose feature extractor is used to learn the vehicle's pose features, which are then fused with the completed vehicle texture and mask. The features are then fed into a pre-trained encoder to generate latent features. The input RGB image and mask are loaded into the GPU memory and stored as tensors. The latent features and the fused feature tensor are stored in a dedicated buffer in the GPU memory as continuous floating-point arrays. Using the sphere template and the Fibonacci sampling method, a uniformly distributed point cloud is generated on the surface of the sphere. Convert the 3D point cloud coordinates in the spherical template from Cartesian coordinates to polar coordinates, convert the updated polar coordinates back to rectangular coordinates, and return the deformed 3DGS point cloud attributes to obtain the vehicle point cloud; The vehicle point cloud is used for rendering and optimization to obtain the RGB image of the vehicle.

2. The single-image vehicle reconstruction method based on 3DGS according to claim 1, characterized in that: The preprocessing includes: label generation, data enhancement, data storage and data quality control.

3. The single-image vehicle reconstruction method based on 3DGS according to claim 2, characterized in that: The label generation includes: Use pre-trained segmentation models and object detection models to generate vehicle mask data from the original RGB image and add category labels, location information, and pose angle data to the vehicle.

4. The single-image vehicle reconstruction method based on 3DGS according to claim 2, characterized in that: The data enhancement includes any one or more combinations of random rotation, cropping, scaling, and adding noise.

5. The single-image vehicle reconstruction method based on 3DGS according to claim 2, characterized in that: The data storage includes: Based on the data usage and scenarios, the data is divided into training set, validation set and test set in preset proportions, and the data is stored in structured folders. When storing, RGB images are stored in PNG format, the generated point cloud data is stored in PLY format, and the label data is stored in json format. The data in the training set contains a variety of vehicle types, angles and environments, which are used for model training. The data in the validation set contains some scenes and angles that have not been seen by the model, which are used to evaluate model performance. The test set is used for the final performance test of the model.

6. The single-image vehicle reconstruction method based on 3DGS according to claim 2, characterized in that: The data quality control includes: Perform quality checks on the raw data to remove data that is ambiguous, incomplete, does not meet the size requirements, or has incorrect labels.

7. The single-image vehicle reconstruction method based on 3DGS according to claim 1, characterized in that: When converting the 3D point cloud coordinates in the spherical template from Cartesian coordinates to polar coordinates, the extracted features are used to predict the deformation parameters of each point through a dual-plane decoder, and the deformation parameters are used to calculate the collapse factor and the updated polar coordinates.

8. The single-image vehicle reconstruction method based on 3DGS according to claim 1, characterized in that: The rendering and optimization using the vehicle point cloud includes: Create a primitive camera object using the object-camera rotation matrix, camera position, image height, width, and intrinsic parameters provided by the dataset, and use this primitive camera object to render the image from the original perspective. Create a camera object symmetrical to the original camera object, create camera pairs at a certain angle, render the normalized vehicle onto a 2D plane, and generate a rotation map of the vehicle; By calculating the loss between the rendered image and the original image, back propagation is performed, and the model parameters are optimized so that the reconstruction result is closer to the input image, thereby gradually approaching the three-dimensional structure of the real vehicle.

9. A single-image vehicle reconstruction system based on 3DGS, characterized in that: include: a data processing module, configured to obtain an autonomous driving dataset and preprocess the data in the autonomous driving dataset; wherein the data in the autonomous driving dataset is multi-view RGB image data obtained by capturing the surrounding environment from different perspectives by a camera installed on the autonomous driving vehicle; A model building module, configured to build a 3DGS-based vehicle reconstruction model; wherein the input of the vehicle reconstruction model is a single RGB image, and the output is a reconstructed vehicle image; A model training module is used to train the vehicle reconstruction model using the preprocessed autonomous driving dataset; The application module is used to use the trained vehicle reconstruction model to reconstruct the vehicle based on a single RGB image; The process of the vehicle reconstruction model processing the input RGB image includes: Generate a mask for the vehicle using a pre-trained mask generator; and complete the vehicle mask and texture by fine-tuning the Lama model on the pre-processed autonomous driving dataset; A preset pose feature extractor is used to learn the vehicle's pose features, which are then fused with the completed vehicle texture and mask. The features are then fed into a pre-trained encoder to generate latent features. The input RGB image and mask are loaded into the GPU memory and stored as tensors. The latent features and the fused feature tensor are stored in a dedicated buffer in the GPU memory as continuous floating-point arrays. Using the sphere template and the Fibonacci sampling method, a uniformly distributed point cloud is generated on the surface of the sphere. Convert the 3D point cloud coordinates in the spherical template from Cartesian coordinates to polar coordinates, convert the updated polar coordinates back to rectangular coordinates, and return the deformed 3DGS point cloud attributes to obtain the vehicle point cloud; The vehicle point cloud is used for rendering and optimization to obtain the RGB image of the vehicle.

Images