A method, apparatus, electronic device and medium for reconstructing vehicle-mounted panoramic images

Abstract

This application provides a method, apparatus, electronic device, and medium for reconstructing vehicle-mounted panoramic images. The method includes: acquiring a set of static image frames by obtaining multiple frames of images corresponding to the target scene captured by the vehicle's native fisheye camera and the vehicle's posture data at the time of acquisition; inputting the set of static image frames and the vehicle posture data into a feedforward neural network to obtain three-dimensional Gaussian point cloud parameters; mapping the panoramic texture obtained by stitching together the multiple frames of images to the three-dimensional Gaussian point cloud corresponding to the three-dimensional Gaussian point cloud parameters to obtain a three-dimensional representation of the target scene; aligning the three-dimensional representation of the target scene with the vehicle's real-time posture data, and rendering an interactive screen based on the alignment result. This application achieves second-level three-dimensional panoramic reconstruction and real-time pose alignment interaction of a target scene by reusing the vehicle's native fisheye camera without requiring additional hardware.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to a method, apparatus, electronic device, and medium for reconstructing panoramic images of vehicles. Background Technology

[0002] When vehicles enter narrow parking spaces or private garages, drivers increasingly need to obtain three-dimensional spatial information about the parking space and accurately perceive the distance to obstacles. Existing in-vehicle panoramic imaging solutions mostly rely on the vehicle's built-in fisheye camera to generate a two-dimensional surround view image through image stitching to assist parking.

[0003] However, two-dimensional surround view images only provide planar visual references and lack three-dimensional spatial depth information, making it impossible to support accurate prediction of parking space dimensions and obstacle distances. Some studies have attempted to introduce 3D reconstruction technology into in-vehicle scenarios, but related solutions generally employ iterative Gaussian parameter optimization processes, resulting in reconstruction times of up to minutes, and requiring motion recovery structures for camera pose initialization. This iterative reconstruction mode cannot meet the requirements for real-time reconstruction and application of vehicle data after acquisition, severely restricting its deployment and practical application in mass-produced vehicles. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a method, apparatus, electronic device and medium for reconstructing in-vehicle panoramic images, in order to overcome at least one of the above-mentioned defects.

[0005] Firstly, this application provides a method for reconstructing a vehicle-mounted panoramic image, including: Acquire multiple frames of images of the target scene captured by the vehicle's native fisheye camera and the vehicle's body posture data at the time of acquisition. Perform distortion correction and dynamic frame removal on the multiple frames of images to obtain a set of static image frames. The static image frame set and the vehicle body posture data are input into a feedforward neural network to obtain three-dimensional Gaussian point cloud parameters. The panoramic texture obtained by stitching together the multi-frame images is then mapped to the three-dimensional Gaussian point cloud corresponding to the three-dimensional Gaussian point cloud parameters to obtain a three-dimensional representation of the target scene. Based on the real-time attitude data of the vehicle, the 3D representation of the target scene is aligned with the real-time attitude of the vehicle, and an interactive screen is rendered based on the alignment result to be displayed on the vehicle's in-vehicle display screen.

[0006] In one possible implementation, distortion correction is performed on the multi-frame images in the following manner: Determine the pre-calibrated radial and tangential distortion coefficients of the native fisheye camera; For each pixel in the image frame, the distorted pixel coordinates of the pixel are mapped to the corrected pixel coordinates according to the radial distortion coefficient and the tangential distortion coefficient, thus obtaining the corrected image frame.

[0007] In one possible implementation, the set of static image frames consists of corrected image frames retained after dynamic frame removal; The corrected image frames are dynamically culled using the following method: Calculate the pixel motion vector between every two adjacent frames of each corrected image frame; Regions where all pixel motion vectors exceed a preset threshold are marked as dynamic regions, and corrected image frames containing the dynamic regions are removed to obtain the set of static image frames.

[0008] In one possible implementation, after obtaining the set of still image frames, the method further includes: Each corrected image frame in the static image frame set is enhanced in low-texture regions to obtain an enhanced static image frame set. The low-texture areas are enhanced in the following ways: The local contrast of the corrected image frame is improved by histogram equalization, and then a pre-trained generative adversarial network is used to generate texture details in low-texture areas to obtain the enhanced corrected image frame. The set of enhanced static image frames corresponding to the enhanced corrected image frame is then input into the feedforward neural network.

[0009] In one possible implementation, the three-dimensional Gaussian point cloud parameters include the position, scale, rotation, color, and transparency of each Gaussian point.

[0010] In one possible implementation, the panoramic texture is obtained in the following manner: The corrected image frame is projected onto a spherical coordinate system; In the spherical coordinate system, a weighted fusion algorithm is used to fuse the overlapping areas of adjacent corrected image frames to obtain the panoramic texture. The weights of the weighted fusion algorithm are positively correlated with pixel sharpness and overlap.

[0011] In one possible implementation, it also includes: After obtaining the three-dimensional representation of the target scene for the first time, the three-dimensional representation of the target scene is quantized, compressed, and stored in the vehicle storage. When the vehicle re-enters the target scene, a 3D representation of the target scene is loaded from the vehicle storage, and the loaded 3D representation of the target scene is aligned with the real-time attitude data of the vehicle and the interactive screen is rendered.

[0012] Secondly, this application provides an in-vehicle panoramic image reconstruction device, comprising: The acquisition module is used to acquire multiple frames of images corresponding to the target scene captured by the vehicle's native fisheye camera and the vehicle's body posture data at the time of acquisition. The multiple frames of images are then subjected to distortion correction and dynamic frame removal to obtain a set of static image frames. The mapping module is used to input the set of static image frames and the vehicle body posture data into a feedforward neural network to obtain three-dimensional Gaussian point cloud parameters, and to map the panoramic texture obtained by stitching together the multi-frame images to the three-dimensional Gaussian point cloud corresponding to the three-dimensional Gaussian point cloud parameters to obtain a three-dimensional representation of the target scene. The reconstruction module is used to align the 3D representation of the target scene with the real-time posture of the vehicle based on the real-time posture data of the vehicle, and render the interactive screen based on the alignment result so as to display the interactive screen on the vehicle's in-vehicle display screen.

[0013] Thirdly, this application also provides an electronic device, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, the steps of the method described above are performed.

[0014] Fourthly, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the method described above.

[0015] This application provides a method, apparatus, electronic device, and medium for reconstructing a vehicle-mounted panoramic image. The method includes: acquiring multiple frames of images of the target scene captured by the vehicle's native fisheye camera and the vehicle's posture data at the time of acquisition; performing distortion correction and dynamic frame removal on the multiple frames to obtain a set of static image frames; inputting the set of static image frames and the vehicle posture data into a feedforward neural network to obtain three-dimensional Gaussian point cloud parameters; mapping the panoramic texture obtained by stitching together the multiple frames to the three-dimensional Gaussian point cloud corresponding to the three-dimensional Gaussian point cloud parameters to obtain a three-dimensional representation of the target scene; aligning the three-dimensional representation of the target scene with the vehicle's real-time posture data; and rendering an interactive screen based on the alignment result to display the interactive screen on the vehicle's in-vehicle display. This application achieves second-level three-dimensional panoramic reconstruction and real-time pose alignment interaction of the target scene while reusing the vehicle's native fisheye camera without requiring additional hardware. It solves the problems of existing solutions relying on dedicated panoramic cameras, time-consuming iterative reconstruction, and inability to meet the real-time closed-loop requirements of in-vehicle systems.

[0016] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart of a vehicle-mounted panoramic image reconstruction method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of the vehicle-mounted panoramic image reconstruction device provided in the embodiments of this application; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.

[0020] First, the applicable scenarios for this application will be introduced. This application can be applied to the field of image processing technology.

[0021] Research has revealed that while the integration of panoramic imaging and 3D reconstruction technologies is progressing, and patent applications for solutions combining 3D Gaussian sputtering have emerged, there are significant gaps in adaptability to in-vehicle scenarios. Existing panoramic imaging technologies are mainly divided into four categories: The first category is pure 2D in-vehicle panoramic imaging technology, relying on a vehicle fisheye camera array to generate 360° 2D surround-view images through image stitching. This only provides planar visual reference, lacking 3D spatial depth information and failing to accurately perceive parking space dimensions and obstacle distances. The second category is 3D panoramic reconstruction technology based on incremental motion recovery structures. This technology relies on incremental motion recovery structures and multi-view stereo matching processes. It generates a 3D model from sparse point clouds and then maps panoramic textures, iteratively estimating camera pose frame by frame. This approach is extremely inefficient and lacks sufficient adaptation to fisheye distortion. The third category... The iterative 3D Gaussian splashing panoramic reconstruction technology uses 3D Gaussian splashing for panoramic reconstruction. Some technologies initialize point clouds by restoring structures through panoramic motion and optimize reconstruction accuracy for motion blur. Others use equidistant cylindrical projection to encode panoramic images to achieve 3D scene reconstruction. All of these technologies use an iterative 3D Gaussian splashing process and rely on dedicated panoramic cameras, and are not adapted to native fisheye hardware in vehicles. The fourth category consists of other 3D Gaussian splashing derivative technologies. Some of these technologies use 3D Gaussian splashing to enhance the rendering of oblique photography scenes, while others optimize the human eye perception effect of 3D Gaussian splashing new perspective synthesis. However, none of these technologies involve panoramic image fusion or in-vehicle scene applications.

[0022] The existing technologies have five main problems: First, they lack compatibility with automotive hardware. Existing 3D Gaussian splicing panoramic solutions all rely on dedicated panoramic cameras, which are not adapted to the large field of view and radial and tangential distortion characteristics of native fisheye cameras in vehicles. This requires additional equipment, increasing implementation costs and hindering large-scale deployment in mass-produced vehicles. Second, iterative processes restrict real-time automotive requirements. Existing 3D Gaussian splicing panoramic solutions all use iterative Gaussian parameter optimization, resulting in reconstruction times in the minute range. Furthermore, they require panoramic motion recovery structures for initialization, failing to meet the rapid closed-loop requirements of "acquisition-reconstruction-application" in vehicles. Third, static scene reconstruction... The existing solutions lack a mechanism for reusing static scenes such as private parking spaces, requiring repeated data collection and reconstruction for each use, wasting onboard computing resources and time, and failing to meet the convenience requirements. Fourth, there is insufficient adaptation to low-texture and complex onboard scenes. The existing 3D Gaussian splash panorama solution is not optimized for low-texture areas such as parking space markings and garage walls, and it does not incorporate a mechanism to eliminate dynamic interference from the vehicle, resulting in a high void rate in reconstruction and easy model drift. Fifth, there is insufficient integration with onboard interaction. The existing solutions only focus on reconstruction itself and do not build a pose alignment mechanism with the vehicle's IMU and GPS, which cannot directly support core onboard interaction needs such as precise parking and parking space navigation.

[0023] Based on this, embodiments of this application provide a method, apparatus, electronic device, and medium for reconstructing in-vehicle panoramic images, aiming to solve the problems of existing 3D Gaussian splash panoramic solutions that rely on dedicated panoramic cameras, have low efficiency in iterative processes, lack reuse mechanisms for static scenes, are not well adapted to low-texture and dynamic interference scenes, and lack an in-vehicle pose alignment and interaction mechanism. The solution achieves the effects of adapting to the vehicle's native fisheye hardware, second-level reconstruction, permanent reuse of one-time modeling, high-precision reconstruction of complex in-vehicle scenes, and real-time pose linkage with the vehicle to support precise parking navigation.

[0024] Please see Figure 1 , Figure 1 This is a main flowchart of a vehicle-mounted panoramic image reconstruction method provided in an embodiment of this application. Figure 1 As shown in the embodiments of this application, the vehicle panoramic image reconstruction method includes: Step S101: Obtain multiple frames of images of the target scene captured by the vehicle's native fisheye camera and the vehicle's body posture data at the time of acquisition. Perform distortion correction and dynamic frame removal on the multiple frames to obtain a set of static image frames.

[0025] The target scene refers to the physical space in which the vehicle will undergo panoramic 3D reconstruction, such as a private parking space and its surrounding area. The native fisheye camera refers to the standard fisheye camera that comes standard with the vehicle, with a single lens having a field of view greater than or equal to 180°. It typically includes four cameras (front, rear, left, and right), providing 360° coverage without the need for additional dedicated panoramic cameras. Vehicle attitude data refers to the data output by the onboard IMU at the moment of acquisition, used to record the vehicle's attitude and motion state at the instant of acquisition, providing auxiliary data for subsequent camera pose initialization. Vehicle attitude data includes three-axis acceleration values, three-axis angular velocity values, and three-axis attitude angles.

[0026] During the acquisition process, the vehicle's native fisheye camera is activated to simultaneously capture multiple frames of images of the target scene, with the timestamp error of the four images controlled within 1 millisecond. For a single private parking space scene, only 10 to 15 frames are needed to meet the 360° field-of-view coverage requirement, with an image overlap rate greater than or equal to 80%. During the acquisition process, the vehicle's IMU simultaneously records the vehicle's attitude data.

[0027] After obtaining multiple image frames, each frame undergoes two preprocessing operations: distortion correction and dynamic frame removal. Distortion correction eliminates the inherent radial and tangential distortions of the fisheye camera, correcting the curved fisheye image to an image with normal geometric relationships. Dynamic frame removal detects and removes image frames containing moving targets such as passing vehicles, pedestrians, and falling debris, retaining only the core valid images of the static scene. After these two operations, a set of static image frames is obtained, consisting of clean and geometrically correct image frames, providing high-quality input for subsequent 3D reconstruction.

[0028] The specific implementation methods of distortion correction and dynamic frame removal are explained in detail below.

[0029] In this embodiment, distortion correction of multiple frames of images is performed in the following manner: the radial distortion coefficient and tangential distortion coefficient pre-calibrated by the native fisheye camera are determined; for each pixel in the image, the distorted pixel coordinates of the pixel are mapped to the corrected pixel coordinates according to the radial distortion coefficient and tangential distortion coefficient, so as to obtain the corrected image frame.

[0030] Among them, radial distortion coefficients k1, k2, and k3 describe the degree of distortion along the radial direction, while tangential distortion coefficients p1 and p2 describe the tangential offset caused by the lens not being perfectly parallel to the imaging plane. These distortion coefficients are obtained in advance using camera intrinsic parameter calibration methods. Distortion correction employs a polynomial distortion correction algorithm, and the correction formula is:

[0031]

[0032] in, , and For the corrected pixel coordinates, and These are the coordinates of the distorted pixels.

[0033] This correction algorithm reduces the geometric error of fisheye images to ±1 pixel. Simultaneously, illumination equalization and Gaussian denoising are applied to the corrected images to eliminate interference from complex vehicle lighting conditions such as backlighting, shadows, and sensor noise. This provides high-precision input to the feedforward neural network and addresses the insufficient adaptation to fisheye distortion in existing 3D Gaussian splash panorama solutions.

[0034] The projection model of a fisheye camera describes the mapping relationship from points in three-dimensional space to the fisheye pixel plane, laying the foundation for distortion correction. The projection model formula is:

[0035]

[0036] Where x, y, and z are the coordinates of a point in three-dimensional space, and f is the focal length of the fisheye camera. , Here are the coordinates of the camera's principal point, and u and v are the pixel coordinates.

[0037] Based on the corrected image frames, dynamic frame removal is performed. Specifically, dynamic frame removal is performed on the corrected image frames in the following way: the pixel motion vector between each two adjacent frames of each corrected image frame is calculated; all regions where the pixel motion vector exceeds a preset threshold are marked as dynamic regions, and corrected image frames containing dynamic regions are removed to obtain a set of static image frames. The set of static image frames consists of the corrected image frames retained after dynamic frame removal.

[0038] Dynamic frame removal is implemented using optical flow, employing the RAFT optical flow algorithm to calculate the pixel motion vector between every two adjacent frames of each corrected image frame. The pixel motion vector reflects the displacement of the same pixel position between adjacent frames. Regions in each frame where the pixel motion vector exceeds a preset threshold of 0.5 pixels are marked as dynamic regions. This threshold is set to distinguish between real object motion and minute pixel displacements caused by vehicle vibrations or sensor noise; displacements below the threshold are not considered dynamic interference.

[0039] After marking the dynamic region, the corrected image frame containing the dynamic region is discarded, rather than discarding two adjacent frames in pairs. The principle is that motion detection is performed between two frames, but the moving object itself only exists in one of those frames. For example, when the first frame is a static scene and the second frame contains a moving object, the dynamic region only appears in the second frame. In this case, only the second frame is discarded, and the first frame is retained as a static frame. The retained corrected image frames constitute the set of static image frames.

[0040] In a preferred implementation of this application, after obtaining the set of static image frames, a low-texture region enhancement step is further included. Specifically, low-texture region enhancement is performed on each corrected image frame in the set of static image frames to obtain an enhanced set of static image frames. The low-texture region enhancement is performed as follows: local contrast of the corrected image frames is improved through histogram equalization, and texture details are generated in the low-texture regions using a pre-trained generative adversarial network to obtain enhanced corrected image frames. The enhanced set of static image frames corresponding to the enhanced corrected image frames is then input into a feedforward neural network.

[0041] Low-texture areas refer to regions with sparse texture features, such as parking space markings, garage walls, and concrete floors, where feature points are difficult to extract. The enhancement scheme employs a fusion strategy of lightweight generative adversarial networks and traditional algorithms as a fallback, balancing enhancement effectiveness with onboard computing power requirements.

[0042] Specifically, a lightweight variant of StyleGAN-Tiny was used as the pre-trained generative adversarial network, with the number of parameters compressed to less than 20M. After INT8 quantization optimization, it was adapted for automotive GPUs. First, histogram equalization preprocessing was used to improve local contrast. Then, the pre-trained StyleGAN-Tiny model was used for local texture enhancement. The original structure was anchored by scene geometric feature constraints, and details were generated only in low-texture areas, avoiding the resource consumption of full-frame rendering. Real-world testing on an automotive GPU showed that the feature point density in low-texture areas was increased by 60% compared to the original image, effectively compensating for the feature matching shortcomings of feedforward neural networks in weak-texture scenes.

[0043] Step S102: Input the set of static image frames and vehicle body posture data into the feedforward neural network to obtain the three-dimensional Gaussian point cloud parameters, and map the panoramic texture obtained by stitching together multiple images to the three-dimensional Gaussian point cloud corresponding to the three-dimensional Gaussian point cloud parameters to obtain the three-dimensional representation of the target scene.

[0044] This step replaces the existing iterative 3D Gaussian splashing process and panoramic motion recovery structure initialization with a feedforward 3D Gaussian splashing, achieving second-level dense 3D reconstruction.

[0045] In this context, a feedforward neural network (FNN) refers to a pre-trained neural network model that outputs a result after a single-step inference given an input. Unlike iterative methods that require repeated parameter optimization during inference, the FNN learns the mapping from 2D images to 3D Gaussian point parameters from massive amounts of data during the training phase. In practical use, only a single forward propagation is needed to output the reconstruction result. This feedforward neural network uses a Transformer architecture, taking a pre-processed set of static image frames and vehicle posture data as input, and outputting 3D Gaussian point cloud parameters. The 3D Gaussian point cloud parameters include the position, scale, rotation, color, and transparency of each Gaussian point. Specifically, the position... Describe the three-dimensional spatial coordinates and scale of a Gaussian point. Describe the extent of the Gaussian point in each direction, including rotation. Describe the orientation and color of the Gaussian point. Describe the appearance, color, and transparency of the Gaussian point. Describes the opacity of Gaussian points. The prediction process of a feedforward neural network can be represented as:

[0046] Wherein, FFN stands for Feedforward Neural Network. Given an image sequence from a set of input static image frames, where N is the number of image frames. This network predicts 3D Gaussian point parameters directly from the image sequence end-to-end without requiring frame-by-frame iterative optimization.

[0047] When inputting vehicle posture data into the feedforward neural network, this step initializes the camera pose based on the vehicle posture data. Specifically, based on vehicle IMU data and sparse feature points in the images, the PnP algorithm is used to quickly initialize the camera pose. The camera pose initialization result is then input into the feedforward neural network along with the set of static image frames, providing a reference for the camera's position and orientation at the time of acquisition, thus helping to improve the prediction accuracy of Gaussian point parameters. This process eliminates the need for the complex process of panoramic motion recovery, significantly reducing computational load. Simultaneously, a temporal consistency constraint for the vehicle scene is introduced to smooth the Gaussian point parameters, avoiding model distortion caused by minor camera shake.

[0048] After obtaining the 3D Gaussian point cloud parameters, panoramic texture mapping is performed. The panoramic texture is a 360° panoramic image obtained by stitching together multiple frames of images. It is used as a texture material and mapped onto the 3D Gaussian point cloud, giving the 3D Gaussian point cloud, which lacks native texture information, a realistic visual appearance. The reason for performing panoramic texture mapping is to solve two major pain points in the automotive scene: "full field of view coverage" and "visual realism of the 3D model". Although the four fisheye cameras each have a field of view of 180° or greater, there are blind spots and differences between multiple image sources. Panoramic stitching can generate a 360° global texture source without blind spots, avoiding texture fragmentation caused by single-channel images. The 3D Gaussian point cloud only describes the geometric structure through parameters. Fusion can map the panoramic texture to each Gaussian point, solving the problem of the disconnect between the geometric model and the visual texture, and adapting to the visualization requirements of automotive AR navigation.

[0049] The panoramic texture is obtained by projecting the corrected image frames onto a spherical coordinate system; in the spherical coordinate system, a weighted fusion algorithm is used to fuse the overlapping areas of adjacent corrected image frames and stitch them together to obtain the panoramic texture. The weights of the weighted fusion algorithm are positively correlated with pixel sharpness and overlap.

[0050] It's important to note that the corrected image frames used here for stitching panoramic textures refer to image frames that have undergone distortion correction but haven't undergone low-texture region enhancement. The enhanced corrected image frames obtained from the low-texture region enhancement step are specifically used as input to the feedforward neural network to improve feature point density and reconstruction accuracy; while the panoramic texture stitching uses the corrected image frames before enhancement to maintain the consistency of realistic lighting and color in the texture, avoiding visual inconsistencies at the stitching seams caused by generative details introduced by the generative adversarial network. Each plays a distinct role in the process: the enhanced image serves 3D geometric reconstruction, while the original corrected image serves visual texture representation.

[0051] Choosing a spherical coordinate system is a natural fit between the characteristics of fisheye imaging and the needs of panoramic stitching: fisheye cameras use hemispherical imaging, and light is projected approximately along the spherical path. The spherical coordinate system can naturally restore spatial geometric relationships and avoid edge stretching distortion when stitching with planar coordinate systems. At the same time, the sphere can achieve 360° borderless continuous stitching. When weighted fusion of overlapping areas, it can maintain the consistency of pixel distance, eliminate stitching seams, and simplify the mapping process between textures and Gaussian point clouds, reducing the computing load on the vehicle's GPU.

[0052] After obtaining the panoramic texture, it is mapped onto the corresponding 3D Gaussian point cloud parameters. The mapping achieves sub-pixel level alignment through Gaussian kernel smooth interpolation; the core formula is as follows:

[0053] in, Let be the covariance matrix of the Gaussian points. Here, x represents the Gaussian point color parameter, x is the spatial point coordinate, and C(x) is the final color value at spatial point x after texture mapping. This sub-pixel-level fusion eliminates texture misalignment between the image and the model, improving visual realism by 40%. The direct mapping method avoids the time-consuming and distorted nature of traditional texture baking, making it suitable for real-time automotive applications.

[0054] After the above steps, the 3D Gaussian point cloud parameters output by the feedforward neural network and the panoramic texture mapping results together constitute the 3D representation of the target scene, that is, the textured 3D digital scene model corresponding to the target scene. The density of the dense Gaussian point cloud can reach 800 points per square centimeter, and the void rate of the low texture area is reduced to less than 5%. The accuracy and efficiency far exceed the existing iterative 3D Gaussian splash panoramic solution.

[0055] Step S103: Based on the vehicle's real-time attitude data, align the 3D representation of the target scene with the vehicle's real-time attitude, and render the interactive screen based on the alignment result to display the interactive screen on the vehicle's in-vehicle display screen.

[0056] Real-time attitude data refers to the vehicle's real-time pose information output by onboard sensors such as IMU and GPS, including the vehicle's position coordinates and orientation angle, used to determine the spatial relationship between the vehicle and the target scene. Aligning the 3D representation of the target scene with the vehicle's real-time attitude involves rotating and translating the 3D representation of the target scene in a spatial coordinate system, using the vehicle's real-time attitude as a reference, to ensure the digital scene perfectly matches the real parking space in spatial relationship, thus determining the vehicle's precise relative position within the scene. After alignment, an interactive screen containing navigation paths, obstacle distance prompts, and other information can be rendered based on the relative position of the vehicle and the scene, and displayed on the vehicle's onboard display screen, supporting applications such as precise parking and parking space navigation.

[0057] The rendering process is based on the native real-time rendering capability of 3D Gaussian splashing technology. By using view frustum culling and LOD level of detail simplification technology, invisible areas and distant objects in the scene are clipped and simplified, ensuring that the rendering frame rate under the vehicle GPU is stable at more than 30fps and the navigation accuracy can reach ±5 cm.

[0058] This application constructs a lightweight rendering and pose alignment framework specifically for vehicles, enabling real-time linkage between 3D scene representation and vehicle motion state, thus bridging the gap between existing 3D Gaussian splashing solutions and the vehicle ecosystem.

[0059] In a preferred implementation of this application, a model storage reuse and local update mechanism is also included.

[0060] Specifically, after obtaining the 3D representation of the target scene for the first time, the 3D representation of the target scene is quantized, compressed, and stored in the vehicle's storage. When the vehicle enters the target scene again, the 3D representation of the target scene is loaded from the vehicle's storage, and the loaded 3D representation of the target scene is aligned with the vehicle's real-time attitude data and the interactive screen is rendered.

[0061] Quantization compression refers to converting Gaussian point parameters in the 3D representation from high-precision floating-point numbers to low-bit floating-point numbers, reducing the model file size. Specifically, 16-bit floating-point quantization of Gaussian point parameters is used to eliminate redundant information and adapt to the limited storage resources of vehicles. The compressed model is stored in the vehicle's local storage, such as an SSD, and supports cloud backup. When the vehicle re-enters the same target scene, the stored 3D representation is loaded directly from the vehicle's storage, with a loading time of less than 1 second, eliminating the need to repeat the entire process from image acquisition to 3D representation generation. This mechanism enables "one-time reconstruction, permanent reuse" of private parking spaces, solving the pain point of existing 3D Gaussian splash panorama solutions that lack reusability design for static scenes and require repeated acquisition and reconstruction each time, reducing long-term usage costs by 90%.

[0062] When the target scene undergoes structural changes, such as the renovation of parking space markings or the addition of obstacles, there is no need for full scene reconstruction. Only a local update mechanism needs to be activated: take 2 to 3 frames of images of the changed area, and use a feedforward neural network to locally update the parameters of the Gaussian points involved. The update time is controlled within 3 seconds to complete the model iteration.

[0063] The technical solution of this application will be fully described below with a specific embodiment. This embodiment takes the three-dimensional panoramic reconstruction and permanent reuse of private parking spaces as the core vehicle scenario.

[0064] During the data acquisition phase, the vehicle's four native fisheye cameras are activated to simultaneously capture images of the parking space and the surrounding area within a 5-meter radius, totaling 12 frames with an overlap rate of 82%. The acquisition time is controlled within 1 second, and the vehicle's posture data is recorded simultaneously, eliminating the need for a dedicated panoramic camera.

[0065] In the preprocessing stage, a polynomial distortion correction algorithm is used to process the image. The pre-calibrated distortion coefficients are substituted, and the geometric error after correction is ±1 pixel. Low-texture areas such as walls and parking space markings are enhanced. Two dynamic frames containing passing vehicles are removed by optical flow method, and a set of 10 static image frames is retained.

[0066] In the feedforward 3D Gaussian splash reconstruction stage, the camera pose is initialized based on the vehicle body posture data and sparse feature points using the PnP algorithm. There is no need for a panoramic motion recovery structure process. The pre-processed set of 10 static image frames is input into the feedforward neural network, and a dense Gaussian point cloud is output within 5 seconds with a density of 800 points per square centimeter. There are no obvious holes or distortions, and the efficiency far exceeds that of existing iterative solutions.

[0067] In the panoramic fusion stage, spherical projection stitches together a 360° panoramic image, and Gaussian kernel interpolation is used to achieve sub-pixel-level texture fusion, resulting in a three-dimensional representation of the target scene with texture.

[0068] During the storage and reuse phase, the quantized and compressed model is stored on the vehicle's SSD. Subsequently, each time the vehicle is parked, the model is loaded within 1 second, and the navigation path is output by combining it with the vehicle's real-time attitude alignment. This eliminates the need for repeated data collection and reconstruction, thus solving the problem of the lack of reuse capability in existing solutions.

[0069] During the partial update phase, the parking space markings were renovated six months later. Three frames of images of the marking area were reshot, and the Gaussian point texture parameters were locally updated. The update time was less than 3 seconds, and the model iteration was completed without the need for full scene reconstruction.

[0070] In this embodiment, the reconstruction efficiency, vehicle hardware compatibility, and static reuse capability are all superior to the existing 3D Gaussian splash panorama solution, and it is fully adapted to the needs of vehicle-mounted private parking space scenarios.

[0071] Compared with existing technologies, this application has the following advantages: Regarding vehicle hardware compatibility, this application directly reuses the vehicle's native fisheye camera and eliminates radial and tangential distortion of the fisheye image through distortion correction. There is no need to install an additional dedicated panoramic camera, which solves the problem of existing 3D Gaussian splash panoramic solutions relying on dedicated panoramic cameras and lacking hardware compatibility. This reduces the implementation cost and makes it easy to scale up and promote to mass-produced models.

[0072] In terms of reconstruction efficiency, this application replaces the existing iterative Gaussian parameter optimization process and panoramic motion recovery structure initialization with feedforward 3D Gaussian splashing, compressing the reconstruction time of a single parking space scene from minutes to seconds, breaking through the efficiency bottleneck of real-time vehicle scenes, and realizing "collection and reconstruction".

[0073] Regarding the reuse of static scenes, this application innovatively designs a "one-time reconstruction, permanent reuse" mechanism and a local update technology for static parking spaces. The 3D representation of the target scene is quantized and compressed and then stored in the vehicle storage. Subsequent use can directly load and reuse the scene without repeated acquisition and reconstruction. When the scene changes locally, only the image of the changed area needs to be retaken to complete the local update without full scene reconstruction. This fills the gap in the existing 3D Gaussian splash panoramic solution for static scene reuse design and reduces long-term usage costs by 90%.

[0074] Regarding adaptability to complex scenes, this application enhances low-texture areas such as parking space markings and garage walls to improve the robustness of low-texture area reconstruction; it combines vehicle-mounted dynamic interference design with a dynamic frame culling mechanism to efficiently filter out dynamic interference sources; and it improves visual consistency through sub-pixel-level texture fusion, which can stably adapt to typical vehicle-mounted scenes such as garages and open-air parking spaces, and is superior to existing general-purpose 3D Gaussian splash panorama solutions.

[0075] In terms of in-vehicle interaction, this application constructs a lightweight rendering and pose alignment framework specifically for vehicles, aligns the 3D representation of the target scene with the real-time posture of the vehicle and renders the interactive screen, realizing real-time linkage between the 3D panoramic model and the vehicle's motion state, supporting precise parking and navigation interaction, and making up for the gap in the connection of existing solutions that only focus on reconstruction itself and do not build a pose alignment mechanism.

[0076] Based on the same inventive concept, this application also provides an in-vehicle panoramic image reconstruction device corresponding to the in-vehicle panoramic image reconstruction method. Since the principle of the device in this application is similar to the in-vehicle panoramic image reconstruction method described above in this application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0077] Please see Figure 2 , Figure 2 This is a schematic diagram of the vehicle-mounted panoramic image reconstruction device provided in an embodiment of this application. Figure 2 As shown, the vehicle-mounted panoramic image reconstruction device 200 includes: The acquisition module 201 is used to acquire multiple frames of images corresponding to the target scene captured by the vehicle's native fisheye camera and the vehicle's body posture data at the time of acquisition, and to perform distortion correction and dynamic frame removal on the multiple frames of images to obtain a set of static image frames.

[0078] The mapping module 202 is used to input the set of static image frames and the vehicle body posture data into a feedforward neural network to obtain three-dimensional Gaussian point cloud parameters, and to map the panoramic texture obtained by stitching together the multi-frame images to the three-dimensional Gaussian point cloud corresponding to the three-dimensional Gaussian point cloud parameters to obtain a three-dimensional representation of the target scene.

[0079] The reconstruction module 203 is used to align the three-dimensional representation of the target scene with the real-time posture of the vehicle based on the real-time posture data of the vehicle, and render the interactive screen according to the alignment result so as to display the interactive screen on the vehicle's in-vehicle display screen.

[0080] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 3 As shown, the electronic device 300 includes a processor 310, a memory 320, and a bus 330.

[0081] The memory 320 stores machine-readable instructions that can be executed by the processor 310. When the electronic device 300 is running, the processor 310 and the memory 320 communicate via the bus 330. When the machine-readable instructions are executed by the processor 310, the steps of the method described above can be performed. For specific implementation details, please refer to the method embodiment, which will not be repeated here.

[0082] This application also provides a computer-readable storage medium storing a computer program. When the computer program is run by a processor, it can execute the steps of the method described above. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0083] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0084] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0085] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0086] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0087] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0088] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for reconstructing a vehicle-mounted panoramic image, characterized in that, include: The system acquires multiple frames of images corresponding to the target scene captured by the vehicle's native fisheye camera, as well as the vehicle's body posture data at the time of acquisition. It then performs distortion correction and dynamic frame removal on the multiple frames of images to obtain a set of static image frames. The static image frame set and the vehicle body posture data are input into a feedforward neural network to obtain three-dimensional Gaussian point cloud parameters. The panoramic texture obtained by stitching together the multi-frame images is then mapped onto the three-dimensional Gaussian point cloud corresponding to the three-dimensional Gaussian point cloud parameters to obtain a three-dimensional representation of the target scene. Based on the real-time attitude data of the vehicle, the 3D representation of the target scene is aligned with the real-time attitude of the vehicle, and an interactive screen is rendered based on the alignment result to be displayed on the vehicle's in-vehicle display screen.

2. The method according to claim 1, characterized in that, Distortion correction is performed on the multi-frame images using the following method: Determine the pre-calibrated radial and tangential distortion coefficients of the native fisheye camera; For each pixel in the image frame, the distorted pixel coordinates of the pixel are mapped to the corrected pixel coordinates according to the radial distortion coefficient and the tangential distortion coefficient, thus obtaining the corrected image frame.

3. The method according to claim 2, characterized in that, The set of static image frames consists of corrected image frames retained after dynamic frame removal; The corrected image frames are dynamically culled using the following method: Calculate the pixel motion vector between two adjacent corrected image frames; Regions exceeding a preset threshold in all pixel motion vectors are marked as dynamic regions, and corrected image frames containing the dynamic regions are removed to obtain the set of static image frames.

4. The method according to claim 3, characterized in that, After obtaining the set of still image frames, the process also includes: Each corrected image frame in the static image frame set is enhanced in low-texture regions to obtain an enhanced static image frame set. The low-texture areas are enhanced in the following ways: The local contrast of the corrected image frame is enhanced by histogram equalization, and then a pre-trained generative adversarial network is used to generate texture details in low-texture areas to obtain the enhanced corrected image frame. The set of static image frames corresponding to the enhanced corrected image frame is then input into the feedforward neural network.

5. The method according to claim 1, characterized in that, The parameters of the three-dimensional Gaussian point cloud include the position, scale, rotation, color, and transparency of each Gaussian point.

6. The method according to claim 2, characterized in that, The panoramic texture is obtained through the following method: The corrected image frame is projected onto a spherical coordinate system; In the spherical coordinate system, a weighted fusion algorithm is used to fuse the overlapping areas of adjacent corrected image frames to obtain the panoramic texture. The weights of the weighted fusion algorithm are positively correlated with pixel sharpness and overlap.

7. The method according to claim 1, characterized in that, Also includes: After obtaining the three-dimensional representation of the target scene for the first time, the three-dimensional representation of the target scene is quantized, compressed, and stored in the vehicle storage. When the vehicle re-enters the target scene, a 3D representation of the target scene is loaded from the vehicle storage, and the loaded 3D representation of the target scene is aligned with the real-time attitude data of the vehicle and the interactive screen is rendered.

8. A vehicle-mounted panoramic image reconstruction device, characterized in that, include: The acquisition module acquires multiple frames of images corresponding to the target scene captured by the vehicle's native fisheye camera, as well as the vehicle's body posture data at the acquisition time, and performs distortion correction and dynamic frame removal on the multiple frames of images to obtain a set of static image frames. The mapping module is used to input the set of static image frames and the vehicle body posture data into a feedforward neural network to obtain three-dimensional Gaussian point cloud parameters, and to map the panoramic texture obtained by stitching together the multi-frame images onto the three-dimensional Gaussian point cloud corresponding to the three-dimensional Gaussian point cloud parameters to obtain a three-dimensional representation of the target scene. The reconstruction module is used to align the 3D representation of the target scene with the real-time posture of the vehicle based on the real-time posture data of the vehicle, and render the interactive screen based on the alignment result so as to display the interactive screen on the vehicle's in-vehicle display screen.

9. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is in operation, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the method as described in any one of claims 1 to 7.

Images